Support Vagrant for non-Docker Builds

[supermarin]

Hi! Thanks for open sourcing this, it's a great thing.

I'm opening this issue to discuss a potential decoupling from Docker.
It's noticeable that Drone was built for cloud services, allowing clients to run under a specific container (e.g. just Ruby2.0).

What world needs is a good Jenkins replacement. It's something people would install with 1 click on any platform (I'm looking at you OSX), and be able to run the job on a host machine.
Something people would update with just go get or git pull and go install.

It's a big set of users, as currently there's no sane self-hosted CI for building iOS apps.

[bradrydzewski]

Thanks for getting this discussion started. Since our goal is to replace Jenkins, you bring up an important use case where our implementation falls short. Hopefully we can hash through the design in this thread (please nobody start coding this just yet)

Maybe the markup could look something like this:

image: win32
image: win64
image: osx

These values would instruct the system to run the build on the host machine, external to a Docker container. Or maybe something like this:

image: host

We would have to work through some potential issues with the yaml file. Certain deployment options will download an executable or modify global variables on the host machine (see the S3 plugin). The services section might also be problematic.

services:
  - mysql
  - redis

[davelab6]

Actually I see this as a regression. Apple allows osx to run virtualized,
so why not run it through docker?
On 26 Feb 2014 18:04, "Brad Rydzewski" notifications@github.com wrote:

Thanks for getting this discussion started. Since our goal is to replace
Jenkins, you bring up an important use case where our implementation falls
short. Hopefully we can hash through the design in this thread (please
nobody start coding this just yet)

Maybe the markup could look something like this:

image: win32
image: win64
image: osx

These values would instruct the system to run the build on the host
machine, external to a Docker container. Or maybe something like this:

image: host

We would have to work through some potential issues with the yaml file.
Certain deployment options will download an executable or modify global
variables on the host machine (see the S3 plugin). The services section
might also be problematic.

services:

• mysql
• redis

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/135#issuecomment-36156059
.

[supermarin]

Because docker works with LXC (Linux containers), which is not available on OSX.
BSD has Jails, but I'm not sure they're supported on OSX either.

However, the main point of this is coupling 2 different pieces of software together (Drone and Docker), while provisioning should be your own choice.
So either manage your platform with chef, puppet, vagrant, docker, anything you want.

EDIT: just to make it clear - there is a workaround to run docker on your mac, but using a tiny linux VM. since you need a real OSX worker, that's almost useless.

[supermarin]

@bradrydzewski that's a good point. maybe image itself could be renamed to something like docker-image, and if not specified, implicitly assumed as host?

[bradrydzewski]

@davelab6 what is recommended way to virtualize osx? Could you share some links?

Docker will introduce a plugin mechanism that, I'm told, will allow developers to plugin container backends like bsd jails, for example. Perhaps someone will write an OSX backend. Ideally we could let Docker handle the abstraction tier as opposed to Drone.

Windows also has a container solution in development:
http://research.microsoft.com/en-us/projects/drawbridge/

I definitely think that using the same Docker / Container paradigm for all architectures would be great, assuming it could be done with OSX, and assuming Docker adds a plugin layer.

[tchap]

I can share experience with building on OSX. Since you must use Apple HW to run even virtualized Mac OS X (that is their license), and virtualizing Mac OS X is not that broadly supported, at least in the free tools, we just don't bother and buy a few Mac Minis to do the job. This is probably not a solution for a hosted CI targeting many many projects, but that is how we do it since virtualizing Mac OS X is simply expensive and PITA. Wondering how Travis does that. I guess they have money to spare :-)

Otherwise I guess you would want to do something similar to Travis, i.e. use virtual machines and roll them back once a build is finished.

[supermarin]

@tchap that may work, but not with docker. that's the whole point of this issue

[tchap]

@supermarin I am just saying that the situation is quite messy. Since there is no support for containers in Mac OS X as far as I know, the only thing you can do is virtualization, which is pretty expensive. And we are also guessing that Docker could take care of a lot of stuff. Why would they even move to support full-fledged virtualization? I mean I don't know their long-term goals, but this would not make too much sense to me. But if anybody has more data, I would be happy to be proven wrong :-)

[tchap]

@supermarin So yeah, I am basically supporting decoupling from Docker.

[tchap]

@bradrydzewski Considering the YAML format, I am wondering if we need to keep it the same for all the platforms. I mean is that even possible? It would be already a huge step forward to be able to build native apps on Mac OS X and potentially Windows. If you want server apps, there is Linux on Docker waiting for you. We could just implement some kind of build slave for each of these platforms that would have its own set of allowed keys in .drone.yml, which would of course overlap where possible.

[bradrydzewski]

Docker is linux only today. But they have openly discussed supporting a plugin architecture to enable BSD Jails and other container solutions. This suggests Docker could run on BSD. Perhaps one could write a Docker plugin for Jailkit to isolate OSX builds?

I'd like to understand more about Docker and backends before removing Docker integration. I will see the team next week and try to get more information.

[gonzojive]

I, too, think it's best to design for the case where custom hardware is needed for testing. The only official/legal way to test iOS is with Apple hardware, as mentioned earlier. The only way to test some firmware is to load it onto a physical machine. Perhaps Drone can remain coupled with Docker, but even if it does, there's need for additional abstraction in Drone to support more custom build environments.

[huslage]

MacOS has Sandboxes, which are a bit like jails.

I think it's honestly best to not worry about this right now and just make the Linux-only solution shine. Docker is far from finished and counting on this-or-that feature right this minute is probably not a good idea anyway. Focus your efforts to replace Jenkins on Linux...then you can abstract out pieces later if you want to.

[bradrydzewski]

@huslage agreed that we will focus on Docker + Linux first

I do have an idea for those wanting OSX or Windows support today, that doesn't require us to decouple from Docker.

I recommend creating a simple daemon that can masquerade as the Docker daemon, but instead execute custom logic. Drone would think it was talking to the Docker daemon ....

We do this for our unit tests:

func TestSetupErrorServiceInspect(t *testing.T) {
    setup()
    defer teardown()

    mux.HandleFunc("/v1.9/containers/create", func(w http.ResponseWriter, r *http.Request) {
        body := `{ "Id":"e90e34656806", "Warnings":[] }`
        w.Write([]byte(body))
    })

    mux.HandleFunc("/v1.9/containers/e90e34656806/start", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusNoContent)
    })

    mux.HandleFunc("/v1.9/containers/e90e34656806/json", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusBadRequest)
    })

Some examples of how this might work:

1. Drone makes request to build an image, the fake daemon instead creates a directory on the host system and saves all files (build script, ssh key, etc). The directory name could use a sha, which would be returned as the Image ID.
2. Drone makes request to create a container from an image, the fake daemon actually copies the image directory into a new directory. The directory could use a sha, which would be returned as the Container ID
3. Drone makes request to start the container. The fake daemon could just run the build script and stream the results back.
4. Drone makes request to destroy the container and image. The fake daemon would just remove the directories.

It would take some reverse engineering of both systems, but I think it would actually work really well, while keeping Drone itself nice and simple. Since Drone will focus primarily on Docker+Linux for the next few months, this may be the best and quickest way to get this working on OSX.

[spheromak]

Isn't this solved with the docker execution drivers? http://blog.docker.io/2014/03/docker-0-9-introducing-execution-drivers-and-libcontainer/

[0xcaff]

The deployment and publish code should really be made platform independent.

[0xcaff]

@bradrydzewski Instead of the hack which you proposed above, we could add a plugin system for execution environments. Docker would be one of these plugins. The plugin system should allow developers to hook their own build slaves into drone (VMs, Mac Minis, BSD Jails).

If you still want to stick with a docker only approach, execution drivers are a thing which allows you to write your own container engines for docker. Basically it should allow you to do everything above, except with docker. Unfortunately, the documentation for execution drivers is sparse, so unless you have time to read all of the docker source code, you will have to wait.

[tchap]

Recently I've been working on a project that basically allows you to add your own build slaves to any CI server. Check https://github.com/cider/cider#tips-and-tricks . Please note that this is very much work in progress, but any ideas are welcome ;-) I originally came up with this to add Mac OS X and Windows "support" to CircleCI, but why not to use it with Docker. I am not saying that it's the cleanest and the most systematic solution, but it is a solution ;-)

[tchap]

Btw a demo output is available here: https://github.com/cider/cider-example#demo-build-run

[bradrydzewski]

The code is (slowly) being refactored so that Drone can work with different virtualization solutions. Although Drone still only supports Docker, it shouldn't be a stretch to support Vagrant, which in turn means we can support VirtualBox (for Windows builds) and VMWare Fusion (for OSX builds).

[denji]

OS X Yosemite will include:

• Hypervisor (Hypervisor.framework). The Hypervisor framework allows virtualization vendors to build virtualization solutions on top of OS X without needing to deploy third-party kernel extensions (KEXTs). Included is a lightweight hypervisor that enables virtualization of the host CPUs.
• Yosemite dedup memory pages [WK compression algorithms - PDF]
• Docker testing Hypervisor for OSX - qemu, virtualbox, chroot-sanbox
  We see that Apple looks towards utilization of resources

I can also raise the issue of support dedup disk space

[supermarin]

@bradrydzewski is there a branch pushed with that work?

[bradrydzewski]

@supermarin the plan has slightly shifted since Windows announced they will officially support Docker. We now need a simple solution for OSX. My plan is to create a simple server daemon that implements the Docker protocol (simple REST API) that translates those commands into something that works on OSX. It will probably be a simple chroot, but can get more sophisticated over time.

This project doesn't exist yet, but I'm happy to help kick it off.

[supermarin]

@bradrydzewski would that still support vagrant and OSX virtualization?

The plans have slightly changed on our side too, and if it helps, one of the goals is to spawn a new VM for each build (and kill it afterwards).

[denji]

• [Free] VirtualBox - build-in (performance 1/5)
• [Paid] Parallels Desktop (only license product - plugin free, better performance 3/5)
• [Paid] VMware Vagrant (vmware plugin 79USD)

[supermarin]

@denji thank you for your response, but i'm not sure this answers the question

[bradrydzewski]

@supermarin yes and no.

If you need a Windows or Linux build environment you will be able to use Docker. This will let you spawn a new virtual environment for every build and kill it after.

If you need an OSX or IOS build environment we will need to provide a workaround. This workaround will need to look like Docker. Drone will think it is talking to Docker. Under the covers it will need to translate the docker instructions coming from Drone into something that OSX understands. I'm not sure what this will look like yet ...

I don't have much OSX development experience, so I'm open to suggestions. What legal options are available to virtualize OSX? Do we need full virtualization for OSX or would a chroot work?

[supermarin]

@bradrydzewski I'm pretty sure you need a full virtualization.
It's legal to virtualize OSX software on Apple's hardware.

[bradrydzewski]

@supermarin upcoming 0.4 supports build agents. The build agent polls Drone for builds, executes and then sends the results back to the Drone server.

This means someone can write an agent for non-Docker platforms, such as OSX, with or without using virtualization: https://github.com/drone/drone/tree/0.4.0/cmd/drone-agent

The only challenge here is that Drone heavily uses Docker for things like plugins (ie deploy to Heroku, publish to S3). This means that an IOS agent would function, but may have limited functionality compared to Docker-based agents.

But it is a start ...

[denji]

lightweight OS X virtualization, zero overhead:

• https://github.com/mist64/xhyve
• http://www.pagetable.com/?p=831
• https://github.com/coreos/coreos-xhyve#coreos--xhyve

[marceldegraaf]

Hi @bradrydzewski, we're currently looking into using Drone behind our firewall. We're looking for a system that can build our Ruby/Go projects in Docker, but can also run iOS builds on a dedicated Mac Mini that's sitting in our office.

The link to drone-agent you posted above is no longer working. Did build agents make it into the 0.4 release? We'd be happy to contribute a build agent that works on OS/X, but I'm not sure if that's still possible?

[bradrydzewski]

@marceldegraaf agents were pulled from the 0.4 release since they need quite a bit more work. In the short term, have you considered running all your builds on linux machines, and for your IOS builds, you could ssh into the Mac Mini?

So your .drone.yml might look something like this:

build:
  image: unbuntu
  commands:
    - |
      ssh user@1.2.3.4 <<EOF
        cd /some/place/on/mini
        git fetch origin
        git merge origin master
        ./test.sh
      EOF

Note that each private Drone repository has a private key injected into the build environment. You can get the public key in the repository settings screen, and then add to your the authorized_keys file on your mac mini.

I know it isn't ideal, but it at least gives you a temporary solution until w have something permanent figured out.

[marceldegraaf]

Hm, that's not a bad idea actually. Thanks!

[joelchen]

Has anyone looked into https://github.com/jkingyens/docker4xcode?

[shaxbee]

It seems that native Mac OS and Windows Docker is in Beta: https://blog.docker.com/2016/03/docker-for-mac-windows-beta/

[supermarin]

Well this won't work either, as the builds won't get executed on OSX.

The Docker engine is running in an Alpine Linux distribution on top of an xhyve Virtual Machine on Mac OS X or on a Hyper-V VM on Windows, and that VM is managed by the Docker application.

[nathany]

Support for other operating systems would be very helpful in certain circumstances. I'm currently looking at options to test the fsnotify Go library on multiple platforms and architectures: fsnotify/fsnotify#136.

[bradrydzewski]

@nathany Drone is capable of running builds on different operating systems and architectures. There have been efforts, for example, to port Drone and all of the plugins to arm:
https://discuss.drone.io/t/drone-ported-to-arm/55

Docker is actively being ported to Windows and FreeBSD which means Drone should be able to run builds natively on Windows and FreeBSD as well. Below I'll provide some more context on what it would take to port Drone to these other platforms. This can serve as a roadmap for anyone that wants to accelerate these efforts.

FreeBSD

Drone should run on FreeBSD with little or no modification. We would just need to build all of our plugins for FreeBSD and publish to the Docker registry (similar to what was done for arm)

Windows

Drone converts build commands in the yaml file into a shell script. In order to support Windows we will need to convert these to powershell commands. We would also need to compile all of our plugins for windows and publish to the Docker registry.

Generating a powershell compliant script should not be too difficult. It should be a relatively simple change to our shell script transform function. The transform function already receives the platform in anticipation of adding this capability in the future.

Multi Architecture

As I mentioned drone is capable of running linux or arm builds, but is not capable of running linux and arm builds using the same installation. This is because right now all builds are placed on the same queue and pushed to the same pool of workers.

In order to support multiple architectures or platforms in a single installation we would need to have multiple queues (one per platform). This should be a relatively simple change to the queue package

-Publish(Work) error
-Pull() Work
+Publish(platform string, work Work) error
+Pull(platform string) Work

Docker Registry Changes

Today when you run docker pull redis on linux it pulls the Docker image for linux. If you run docker pull redis on windows it pulls the Docker image for linux. This makes it difficult to use the same yaml across multiple platforms because Docker image names are currently unique to a single platform.

Luckily this is something the Docker team is working to improve:
moby/moby#15866

Things We've Already Done

We've already added a bunch of new capabilities in anticipation of running builds on multiple platforms. For example, you can already specify the platform in your yaml file (it gets ignored right now):

platform: linux_amd64

You can use the platform in your build matrix to test a single commit against multiple architectures:

platform: ${platform}

matrix:
  platform:
    - linux_amd64
    - windows_amd64
    - feebsd_amd64

You can even limit steps in the yaml to specific platforms:

script:
  posix:
    image: golang
    commands: ./build.sh
    when:
     platform: [ linux_amd64, freebsd_amd64 ]
  windows:
    image: golang:windows    # windows uses a special image
    commands: ./build.batch  # windows runs a batch file
    when:
     platform: windows_amd64

[ysangkok]

FYI, compiling on FreeBSD currently, i have the following problem:

[janus@xn--kn-1ia ~/gopath/src/github.com/drone/drone] gmake build
cd drone && go build --ldflags '-extldflags "-static" -X github.com/drone/drone/version.VersionDev=' -o drone
# github.com/drone/drone/static
../static/static.go:15: undefined: Asset
../static/static.go:15: undefined: AssetDir
# github.com/drone/drone/template
../template/template.go:35: undefined: AssetDir
../template/template.go:45: undefined: MustAsset
# github.com/drone/drone/vendor/github.com/mattn/go-sqlite3
cc: warning: argument unused during compilation: '-pthread'
# github.com/drone/drone/store/datastore
../store/datastore/store.go:100: undefined: ddl.Asset
../store/datastore/store.go:101: undefined: ddl.AssetDir
Makefile:30: recipe for target 'build_static' failed
gmake: *** [build_static] Error 2

[bradrydzewski]

@ysangkok did you run make deps and make gen to generate the static assets? The compile errors seem to indicate these are missing and have not been generated. Second seems to be just a warning with sqlite3 but not sure that is an issue. Let's move the compiling discussion to our gitter room at https://gitter.im/drone/drone and I can help you in real time!

[bradrydzewski]

I also want to point mention that we are working on porting the git plugin to arm and have a docker image available. Would be great to have the same for freebsd and windows if there are any takers :)
https://github.com/drone-plugins/drone-git/blob/release/0.5-alt/Dockerfile.armhf

[XVilka]

Good day! Are there any plans for this yet? To be able to work without docker? This issue seems unupdated for a while.

[bradrydzewski]

Since this was opened (about 3 years ago) the project has deeply integrated with container runtimes. Unwinding this is not trivial. So at this time there are no plans to execute builds outside of container environments, so it is probably best to close the issue.

If the scope of the project changes we can always reconsider and re-open this issue.

[Adnn]

My 2 cents on a closed issue:
The original proposal of this discussion (integrating with Vagrant in addition to Docker) is much preferable than simply supporting host builds. Vagrant can be used to virtualize Windows and OSX (and others), and some people are building native apps targeting all of those.
[And to agree with one of the first points: the world is in need for a better Jenkins, i.e. an automation system that does not assume your target environment(s). Not saying you necessarily want to be this guy, but just in case :) ]

[droplet-js]

关注drone一年多了，依然未解决osx编译问题，真是让人失望啊。

--- translate ---

It is disappointing to notice that drone is still more than a year old and still fails to solve the OSX compiler problem.

[tboerger]

The upcoming release will be prepared for other build environments, but the usage of plugins is still not resolved that way.

[bradrydzewski]

Drone supports different runtime execution environments, including support for native execution directly on the host. This includes support for Linux, Windows, macOS, BSD and more. See https://twitter.com/droneio/status/1168931431546052608