Consider distributing prebuilt static libs

[cretz]

Would need CI infrastructure, don't want to do this for all 3 platforms myself.

[ciehanski]

I would be willing to help on this front. What are you looking for exactly? Travis, CircleCI, etc...which 3 platforms?

[cretz]

Could be any CI. I'd like to have the build.go file run successfully for 64bit versions of linux, windows, and macos and have the artifacts reachable. In the future, I'd like those static artifacts to be bundled into a download archive by themselves (but same dir structure). I'd be happy just putting that bundle in the releases area when I tag. I have considered having the script also fetch last dependencies from their respective repos, but there's human intervention required anyways so submodules are fine.

Oh, and probably just a quick sanity test for each platform maybe using Bine just to confirm successful compile, but that's future too.

[ciehanski]

Sounds good, let me lay out a basic structure and run it by you. I'm more familiar and prefer travis so that is what I'll be using, but that can be changed later if necessary. I can set up the automated GitHub releases as well. At first, I'll set draft: true so that it will not publish so you can review.

Regarding confirming if the artifacts are reachable, I may ping you regarding commands you use now for testing, etc.

[cretz]

Thanks! Yeah, I don't really test much beyond a couple of examples in https://github.com/cretz/bine/tree/master/examples and https://github.com/cretz/bine/tree/master/process/embedded/tor-0.3.5/embeddedtest. I think an integration test that does what https://github.com/cretz/bine/blob/master/tests/tor_dialer_test.go does but with embedded would be ideal.

The way I planned on doing releases henceforth was a branch for a Tor version, a PR, and then a merge-master, then a tagged release.

[ciehanski]

I understand now. I could write something simple to test the service after the fact. Do you mind if I play with docker and docker-compose for building as well or would you like to exclude docker? Would be more for internal testing.

We can fine tune the branch names and releases later on once we have the builds somewhat working.

Also, feel free to peek at what I am working on: https://github.com/ciehanski/tor-static/blob/integrate-ci/.travis.yml

[cretz]

Ideally docker wouldn't be needed, but I'm open to whatever. The build is pretty much straightforward, if the prereqs are followed, this repo is recursively git clone'd, and go run build.go -verbose build-all is run it should compile what's necessary on each platform. As for bundling the artifacts in the future, we might have to plug in to show-libs to see what should be bundled (you can see what libs are used in https://github.com/cretz/bine/blob/master/process/embedded/tor-0.3.5/process.go).

For anything like bundling those artifacts in the future, I would like to put the code as a command in build.go instead of in travis's yml file or shell scripts.

That yml looks quite perfect/clean so far.

[ciehanski]

Makes sense, I only mentioned it because I'm on macos and use go modules mostly so my gopath was funky when trying to build locally, no worries though.

As per showing the libs, I think your way of thinking is valid and I like that method.

So far, everything is building fine in osx and linux(xenial). Still working on the msys2 shell/retries for windows builds.

Where can I find the exact files you would like pushed to releases?

[cretz]

Awesome. Yeah, the Windows shell reloads are annoying, maybe there are better ways or it's not needed but the tutorials I've seen suggest to keep re-running until no longer asked.

I want exactly the libs needed to satisfy a static build. So literally what is linked as a result of show-libs. All the files listed at https://github.com/cretz/bine/blob/master/process/embedded/tor-0.3.5/process.go, but we want them in their same directory structure (so the difference between using this artifact-only archive and building from scratch has the same layout). That's why I mentioned the show-libs part, because you can see in the build.go code where I have them as a multidimensional array of dir name then array of file names. Ideally in build.go some of that show-lib code wuld be factored out to just return that lib set, then it would have a new "package-artifacts" or some such command that instead of just dumping it to screen instead built both a zip and tarball out of it. But all of that can be a future step.

[ciehanski]

Sounds good thanks for explaining again, this is a new area for my so my apologies. I'll see what I can come up with.

[ciehanski]

Just a heads up, I believe my first PR will not include windows, unfortunately. The windows build fails too frequently to rely upon, from travis' side of things considering Windows support is still experimental. Hopefully this will be acceptable at first and we can add it in later.

[cretz]

Sure. Also, I am quite adept at Windows, so I would be willing to help you work through bugs there. You'll note at https://github.com/torproject/tor/blob/master/.travis.yml they don't even include Windows which is one of the reasons I think this project is so valuable.

So yes, feel free to PR without it or let me know your issues and maybe we can work through em. I did note you have an after_success script that does a bunch of mv of a bunch of paths. Would you mind if I wrote a "package-libs" command in build.go that did this? I want to dump a tarball and a zip based on "show-libs".

[cretz]

Ok, see the referenced commit above (edit: and below) this comment. There is a "package-libs" command for build.go now that will create libs.tar.gz and libs.zip for the required libs. You can merge master into your branch and use it. Unfamiliar w/ Travis, but on release I'd take the libs.zip from the Windows build and libz.tar.gz from the Linux and Darwin builds and rename them to something like tor-static-0.3.5.7-win64.zip, tor-static-0.3.5.7-linux64.tar.gz, and tor-static-0.3.5.7-mac64.tar.gz respectively but all that can come later.

[ciehanski]

That looks fantastic! I have implemented the package-libs in my new travis configuration. I see you did add in a copy for tor_api.h but I am still getting the following error in travis when running tests:

$ go test -v build_test.go -tor.verbose
# github.com/cretz/bine/process/embedded/tor-0.3.5
../../cretz/bine/process/embedded/tor-0.3.5/process.go:35:21: fatal error: tor_api.h: No such file or directory
compilation terminated.
FAIL	command-line-arguments [build failed]
The command "go test -v build_test.go -tor.verbose" exited with 2.

You can view my tests here.

[cretz]

Where is tor-static checked out and built in relation to bine? You can see in Bine that the folder it's expecting tor_api.h to be at is relative to its current location: https://github.com/cretz/bine/blob/f5c65d3eb01d457b539c71b0113ffeaf0b200f24/process/embedded/tor-0.3.5/process.go#L16

[cretz]

With #17 merged, once #18 is done and we have a new version to tag, we'll make sure the release gets its binaries and then update the docs accordingly on how to use them instead of compiling yourself.

[igrb]

I see you have issues with Travis CI for building on Windows. I have a bitbucket.org academic plan account, maybe I can write a configuration for bitbucket pipelines which builds inside docker images, the hard part would be to write the configuration for pipelines. I can make a mirror of this repo and if someone can help me with the configuration of the pipelines we can produce and serve those artifacts on my mirror.

I discovered bine a couple of days ago and it's great! I want to build a tor hidden service which will run on a raspberry pi on my home's local network and will expose an authenticated interface for switching on my PC with WOL (Wake On LAN) packets, so I can then connect with ssh through tor to my home's PC.

[cretz]

I see you have issues with Travis CI for building on Windows

I ended up getting it working with AppVeyor. I am a bit busy, but I will be testing this issue (static-lib distribution) hopefully soon.

[m4573rh4ck3r]

In my case the problem was, that I was using go modules.
I used a replace directive to replace github.com/cretz/bine with ./github.com/cretz/bine and ran go mod init inside its root directory.
I placed my Project directly inside my $GOPATH so that ./ for the replace directive was the same as $GOPATH and used docker so that this has no side effects.
Compilation now works!

[mibmo]

Any update on this?

GH actions exist now as well, so building wouldn't even require much in the "infrastructure" department anymore.

[cretz]

Agreed, GH actions would be just fine for this and the workflow yaml is fairly easy to write. I am afraid I don't really work on this library much anymore so I don't know if/when I'd be able to get to this.

[mibmo]

I'll give it a go then! :)

Going for Linux builds first. Is there a specific Go version you want to build at?

[cretz]

Is there a specific Go version you want to build at?

Nope. Probably oldest non-EOL which I think is 1.17.