duck: fix build for Linux

[danielnachun]

• Have you followed the guidelines for contributing?
• Have you ensured that your commits follow the commit style guide?
• Have you checked that there aren't other open pull requests for the same formula update/change?
• Have you built your formula locally with brew install --build-from-source <formula>, where <formula> is the name of the formula you're submitting?
• Is your test running fine brew test <formula>, where <formula> is the name of the formula you're submitting?
• Does your build pass brew audit --strict <formula> (after doing brew install --build-from-source <formula>)? If this is a new formula, does it pass brew audit --new <formula>?

---

[danielnachun]

This formula had depend_on :macos in linuxbrew-core, but the CLI component supports Linux, and upstream even provides RPM and DEB packages. I had to hack the build.xml file so that it would build an app-image target which just provides the binaries and libraries in a directory we can install to libexec.

I will open an issue on the upstream Trac for Cyberduck (https://trac.cyberduck.io/query) once the CI passes here asking for guidance on how to make a PR to properly add app-image support to the build.xml. Merely adding app-image as a build type is not quite enough for it to work, as their call to jpackage includes 3 flags which are only valid for RPM or DEB targets. I circumvented this for now by simply deleting those lines from the build.xml (we don't care about breaking RPM or DEB in our setup), but I want to get guidance from upstream about how to add app-image support without breaking the other targets.

[danielnachun]

Ironically this now builds fine on Linux but fails on macOS because of universal/non-native binaries.

It seems this does not actually support native ARM yet: https://trac.cyberduck.io/ticket/11101, which is blocked by iterate-ch/rococoa#20. So I think this needs depends_on :x86_64.

As for the universal binaries, I think I have seen elsewhere that these are actually from macOS itself? I'm not sure how we've handled this before.

[carlocab]

JavaNativeFoundation.framework is likely copied from an Xcode SDK, which is why it's universal. Not sure about libjnidispatch.dylib. If we can figure out where duck is getting these from it'd be a lot easier to figure out what to do about them. Will try to poke at their build system a bit.

Agreed that this needs depends_on arch: :x86_64.

[cho-m]

There is a new bump PR for this formula #83146 which will probably fail for same reason on macOS.

Can either combine into single PR, or will need to fix to one PR and then rebase the other.

[danielnachun]

I can confirm that the libjnidispatch.dylib is indeed being downloaded as a prebuilt binary:

Downloading from maven.cyberduck.io-release: https://repo.maven.cyberduck.io.s3.amazonaws.com/releases/net/java/dev/jna/libjnidispatch/5.7.0/libjnidispatch-5.7.0-x86_64.dylib
Downloaded from maven.cyberduck.io-release: https://repo.maven.cyberduck.io.s3.amazonaws.com/releases/net/java/dev/jna/libjnidispatch/5.7.0/libjnidispatch-5.7.0-x86_64.dylib (291 kB at 293 kB/s)

I think I've figured how to build libjnidispatch.dylib as a resource from upstream: https://github.com/java-native-access/jna, and now I'm just cleaning up the changes before trying them here.

[danielnachun]

I've now pushed some major changes that should build libjnidispatch from source on macOS and Linux. The build system for java-native-access took quite a while to understand, and I'll explain below:

1. A call is made to ant to generate some platform-specific native JNA C header files from the JAR files.
2. A bash build script (build.sh) is generated for building the native library with a Makefile.
3. The JNA header files and build script are packaged with rest of the native library C source (which is not platform-specific) into a zip. Unfortunately the original files are deleted after the zip is made.
4. The end user is supposed to extract the zip file, then run build.sh to actually build the final shared library. The shared library needs libffi, which is supposed to be found by pkg-config (which works on Linux but not on macOS). On Linux, the library ends in .so, but on macOS it ends in .jnilib and is later renamed to .dylib.

Initially I just extracted the zip archive after it was made and then ran the build script inside of the folder. However, I realized that the naming of the zip file is rather inconvient to work with programmatically - it changes based on platform, and based on the JNI version, which cannot easily be extracted from anywhere. I also found it cumbersome to have to unzip a file like this. My workaround for this was to just delete the line that packages the header files into the zip file. Along with a small change in the build.sh, this was sufficient to make that script work as desired. The other fixes I made were for some bugs on macOS.

@carlocab I'm pretty sure after checking the logs that maven completely ignores the host JDK and Framework installations altogether:

Downloading from maven.cyberduck.io-release: https://repo.maven.cyberduck.io.s3.amazonaws.com/releases/ch/iterate/JavaNativeFoundation/81/JavaNativeFoundation-81.zip
Downloading from maven.cyberduck.io-release: https://repo.maven.cyberduck.io.s3.amazonaws.com/releases/net/adoptopenjdk/jre/15.0.1/jre-15.0.1.zip

We happened to catch libjnidispatch.dylib and JavaNativeFoundation becuase they were built as universal binaries, but technically the JRE this is using is also not the brewed one, but a copy downloaded from AdoptOpenJDK.

There is a Github repo for JavaNativeFoundation from none other than Apple themselves: https://github.com/apple/openjdk/tree/xcodejdk14-release/apple/JavaNativeFoundation. That is probably what is being used to build the universal JavaNativeFoundation artifact. I think we can also build this from source as another resource.

For now I think we should focus on getting the universal native binaries built from source so that this passes our audit. However, it would be great for formulae like this to make sure we are building all the native binaries from source.

[danielnachun]

I did it! JavaNativeFoundation was quite trivial to build from source, and after replacing that in the Frameworks folder, this passes the audit! To me this gives us hope that we'll be able to fix these issues in other formulae using the same approach. As I mentioned before we can consider when it makes sense to create formulae for native dependencies, and when it should just be a resource.

[SMillerDev]

It's a huge change and I hope those two resources will become formula, but for now this should be enough.

[carlocab]

Can we add a comment explaining how to update these resources? This can be in a follow-up syntax-only PR.

[danielnachun]

I am actually thinking we should make these resources into formulae. I am going to update this to the newest version and then I'd like to make formulae out of the resources and remove those parts of this formulae. All we'd have to do if the resources are formulae is link the necessary files into libexec after deleting the pre-packaged ones. The resources themselves both use GitHub so we can use livecheck to keep them up to date if they are formulae.

[carlocab]

Sounds good to me. Thanks for working on this!

[danielnachun]

Could you approve again? I had to dismiss the previous approval from @SMillerDev when I updated the version.

[danielnachun]

It's a huge change and I hope those two resources will become formula, but for now this should be enough.

For jna making a formula should be pretty easy now that I figured out its build system. It is cross-platform and used in many packages so I would not be surprised if we end up using it elsewhere. For JavaNativeFoundation, the only snag here is that the repo it is part of doesn't quite meet our notability requirements. Perhaps we can make an exception?

[SMillerDev]

I'm fine with an exception if other tools need it.

[BrewTestBot]

@danielnachun has triggered a merge.

[dkocher]

This will break when we update the JNA dependency upstream and this formulae will take the older version instead.

[danielnachun]

Thanks for following up on this. I have 2 thoughts about this:

1. These resources will be made into formulae soon, which will be updated automatically with livecheck and brew bump.
2. The only thing we're actually taking from JNA is the dylib/so, which is built with -compatibility_version 5 such that it should maintain ABI compatibility as long as the major version remains at 5. If and when a version 6 is released which breaks ABI compatibility, we'd either rebuild duck against version 6 (if it has been updated), or if it takes a while for upstream to update, create a jna@5 formula to use in the interim.

I'm not intimately familiar with the details of how JNA is implemented, but it's generally rare for native dynamic libraries to break ABI compatibility between minor versions, let alone point releases. I'm certainly open to being proven wrong on this but we'll have to see how the next update goes.

I should also add here that we are pushing harder to try to avoid downloading pre-built native binaries because doing so has both security and compatibility implications. However we're hoping to do so without huge inconvenience to upstream projects. In this particular case, no changes are needed in how the package is built upstream - we simply delete the binaries downloaded by Maven and replace them with ones that will eventually come from formulae.