0.2.0-rc1 - Look Ma I'm not a PoC anymore!

This marks the first RC for Dev-Loop 0.2.0. Dev-Loop 0.2.0 is really about taking dev-loop from a useful PoC, into something that you'd actually want to share at work, and talk about using for evergreen projects.

0.2.0-RC1 is not technically feature complete, however there are numerous bug fixes, so we'd like people to be pointed to this release. 0.1.1 && 0.1.0 config works in full. 0.2.0-RC1 should be used today. The reason this is an RC is because it's not feature complete, not because it isn't stable to be depended on.

---

New Features for RC1

The following features have been added to 0.2.0-rc1 from the latest release 0.1.1. Not as mentioned above, these are not all the features we want to add in 0.2.0.

experimental_permissions_helper attribute for docker executor

When this is enabled the docker executor will create a pseudo user with the uid/gid of the user on the host to run all your tasks as, that as closely as possible resembles the user the container would normally use. This way on linux you don't have files written in your project owned by root, that require a chown to fix owners of. This is meant to smooth the experience between docker && not docker.

This is marked as "experimental" because it requires some tooling inside containers to be used. As far as we know this should work for nearly every container that dev-loop can run in today, but more testing needs to be done on a wide variety of images within containers. Anything inheriting off of ubuntu/debian is guaranteed to work as they distribute all the tool necessary.

We'd like lots of feedback on this! Even if you don't have permission issues in docker, but are using docker enable it, and let us know how it goes! We'd love to make this feature true by default, but that requires lots of confidence which needs to be built through actual usage.

parallel-pipeline task type

Starting in 0.2.0 it is now possible to write a parallel-pipeline task which will always parallelize. This can be useful when you want too run tasks in parallel that still need to be grouped together and done as one task. The syntax is the exact same as pipeline, but instead of offering order guarantees, it specifically splits them up:

---
tasks:
  - name: run-things
    description: run things
    type: parallel-pipeline
    steps:
       - name: step one
         task: step-one
         description: perform step one
       - name: step two
         task: step-two
         description: perform step two

As you can see the syntax is the exact same as pipeline, except it is called "parallel-pipeline".

DL_WORKER_COUNT environment variable

It is now possible to control the number of workers that dev-loop will provision to use with the DL_WORKER_COUNT environment variable. This can be used as a one-off to test a normally parallel tasks in not parallel for debugging (in the case of buggy user scripts), or if you would like to increase the number of workers from say physical cores, to the amount of virtual cores you have so your tasks run faster.

Error-ing Overhaul

Errors have completely been overhauled to now provide actual user friendly error messages. This is in a huge part thanks to @yaahc and her excellent work with color-eyre! Huge props for an amazing library that made this really easy to do, and being super open to contributions. Some notable improvements include:

• rustc esque line showing yaml callouts of where in your configuration you've made a mistake:

• Typo suggestions for most errors:

• Warning messages when docker is taking awhile to do something like say download an image, rather than outputting nothing.

---

Bug Fixes

Numerous bug fixes have been merged into 0.2.0-rc1, which is the main reason we're recommending people use it now. These bug fixes include:

• Numerous Mac OS X fatal errors where Mac OS X would not properly stand up docker containers.

  • This now has continous testing on ever PR to validate this doesn't happen in the future.

• Fixed an issue where the list command would occasionally freeze in certain outputs.

  • This now has an automated unit test to validate this doesn't regress.

• Fixed an issue where dev-loop could not properly resolve certain script paths if not run from the root of the project.

  • This too now has an E2E test on OSX, and Linux to validate it hasn't broken.

---

Features Still Left in 0.2.0

Since I made a fuss about 0.2.0 not being feature complete what can you expect to see from 0.2.0 that isn't here yet?

• The ability to specify a "default" option for oneof tasks. Save those keystrokes!
• Complete overhaul of the docs site!
  • Create the new logo!
  • Update the color scheme, and general feel so it doesn't feel so thrown together. I really like rome's website: https://romefrontend.dev/ so we should take some inspiration.
  • Add in a version selector on the sidebar. When browing old docs there should be a red bar that says "outdated docs" but they should be kept in their original form. When browing an unreleased version there is a green bar that says "beta/unreleased". By default show latest stable.
  • Update the schema docs to actually be parsable: I really like stripe's docs https://stripe.com/docs/api/sigma/scheduled_queries/object specifically notice the "Show child attributes" which would be great for things like the docker executor params which aren't strictly typed.
  • Add in sample migration plans using notoriously hard to build projects. Show how dev-loop can deliver value right away for non new projects.
  • Rewrite the walkthrough so it's a bit easier to digest as someone new to dev-loop. The current was definitely written in a style for someone who is pretty savy with the problems of local builds of mono-repo esque projects. This shouldn't be required to understand why dev-loop is useful, or how to use it.
• Add a trace command which allows you to export chrome tracing files so you can see where dev-loop is spending the time for your execution, as a way of optimizing local build times as well as finding issues in dev-loop.