Embedded JavaScript pipeline PoC #1829

pda · 2022-11-14T12:18:29Z

Experimental PoC adding a buildkite-agent pipeline eval buildkite.js command using github.com/dop251/goja as a JavaScript (“ECMAScript 5.1(+)”) interpreter, plus github.com/dop251/goja_nodejs for CommonJS require("…") and some other conveniences.

require("buildkite/plugin") demonstrates a native module, implementing the plugin(name, ver, config) JavaScript function as a Go Function.

Other require("buildkite/*") modules are provided by an embedded filesystem, located in a new package resources; it currently contains node_modules/buildkite/hello.js which services require("buildkite/hello") and demonstrates console.log() working.

Other require("…") paths are served from the host filesystem (i.e. the build directory).

There's an example JavaScript pipeline at test/fixtures/pipelines/buildkite.js demonstrating some basic module loading, environment usage, plugin reuse etc:

// Example buildkite.js

// native module implemented in Go
plugin = require("buildkite/plugin");

// loaded from embedded filesystem in buildkite-agent binary
require("buildkite/hello");

const dockerCompose = plugin("docker-compose", "v3.0.0", {
  config: ".buildkite/docker-compose.yml",
  run: "agent",
});

pipeline = {
  env: {
    DRY_RUN: !!process.env.DRY_RUN,
  },
  agents: {
    queue: "agent-runners-linux-amd64",
  },
  steps: [
    {
      name: ":go: go fmt",
      key: "test-go-fmt",
      command: ".buildkite/steps/test-go-fmt.sh",
      plugins: [dockerCompose],
    },
  ],
};

module.exports = pipeline;

$ DRY_RUN=1 ./buildkite-agent pipeline eval test/fixtures/pipelines/buildkite.js
[stderr] 2022-11-14 22:44:37 INFO   Reading pipeline config from "test/fixtures/pipelines/buildkite.js"
[stderr] 2022/11/14 22:44:37 hello embedded FS!

agents:
  queue: agent-runners-linux-amd64
env:
  DRY_RUN: true
steps:
- command: .buildkite/steps/test-go-fmt.sh
  key: test-go-fmt
  name: ':go: go fmt'
  plugins:
  - docker-compose#v3.0.0:
      config: .buildkite/docker-compose.yml
      run: agent

This can be piped into buildkite-agent pipeline upload for now, although eventually it would be integrated.

$ ./buildkite-agent pipeline eval test/fixtures/pipelines/buildkite.js | ./buildkite-agent pipeline upload

keithpitt · 2022-11-14T13:40:24Z

clicommand/pipeline_eval.go

+	process.Enable(runtime) // process.env()
+
+	// provide plugin() as a native module (implemented in Go)
+	registry.RegisterNativeModule("buildkite/plugin", func(runtime *goja.Runtime, module *goja.Object) {


Ohhh, this is how you did it. I can imagine expanding on the API would be a bit of a pain if it's all in Golang. Presumably we could RunScript with some existing gear in the VM first?

Ah right, looks like it's built into goja_nodejs already https://github.com/dop251/goja_nodejs/blob/master/require/module.go#L114. I can't quite figure out how to enable it though...

@keithpitt Yeah this registry.RegisterNativeModule("buildkite/plugin", … is just a proof-of-concept of JS functions implemented in Go, but I think you'd want a very good reason to do that; like interacting with existing/complex Go code/libs.

The require.WithLoader(func(name string) ([]byte, error) { … ~20 lines further up loads require("buildkite/*") straight out of resources/node_modules/buildkite/*.js files which get embedded into the buildkite-agent binary at compile time and exposed via a virtual read-only filesystem.

pda · 2022-11-15T00:08:50Z

clicommand/pipeline_eval.go

+	// Add support for require() CommonJS modules.
+	// require("buildkite/*") is handled by embedded resources/node_modules/buildkite/* filesystem.
+	// Other paths are loaded from the host filesystem.
+	registry := require.NewRegistry(
+		require.WithLoader(func(name string) ([]byte, error) {
+			if !strings.HasPrefix(name, "node_modules/buildkite/") {
+				return require.DefaultSourceLoader(name)
+			}
+			res := resources.FS
+			data, err := res.ReadFile(name)
+			if errors.Is(err, fs.ErrNotExist) {
+				return nil, require.ModuleFileDoesNotExistError
+			} else if err != nil {
+				return nil, err
+			}
+			return data, nil
+		}),
+	)


@keithpitt This is the bit which will load require("buildkite/whatever") from resources.FS which is a virtual filesystem exposing resources/node_modules/buildkite/* files embedded in the compiled binary.

moskyb · 2022-11-15T00:35:27Z

clicommand/pipeline_eval.go

+	y, err := yaml.Marshal(v.Export())
+	if err != nil {
+		return err
+	}


it might be cool to evaluate the generated pipeline.yml against the pipeline json schema here to safeguard against generating a wonky pipeline

🤔 i wonder if we should do that on pipeline upload as well, even if we just output warnings

Yeah that's be nice on upload too!

Oh, hell yeah! If we can get this thing talking to npm, maybe it makes sense to keep the schema in a package. If we kept it in sync, then our pipeline editor in the UI could validate against it too.

package resources (embedded filesystem) is also folded into package js.

Now require("buildkite/*") will attempt to load from host filesystem if the requested file doesn't exist in the embedded filesystem.

pda · 2022-11-15T12:34:06Z

I've done a bit of refactoring, and some improvements to require("…") loading, and the debug/error messages thereof. You can pass --debug to the buildkite-agent pipeline eval command to get full traces of what files it's trying to load from where.

davidwheeler123 · 2023-01-17T06:04:26Z

I love this idea - I find the yaml quite challenging to work with, and I love the idea of a Turing complete DSL, with factoring and lovely things like that. But why JavaScript vs typescript? With typescript you'd get IDE integration for free, and discoverability would increase 1000%, plus more problems surfaced at compile time instead of runtime

dabarrell · 2023-03-31T06:57:49Z

Hey @davidwheeler123 – glad you stumbled across this! This was purely a first-pass experiment, and the use of javascript here doesn't mean much – we've also played around with an embedded typescript engine :) This is early days, but certainly a direction we're exploring

Embedded JavaScript pipeline PoC

a844d5e

pda force-pushed the pda/js-pipeline branch from 063c4e4 to a844d5e Compare November 14, 2022 12:22

keithpitt reviewed Nov 14, 2022

View reviewed changes

pda commented Nov 15, 2022

View reviewed changes

moskyb reviewed Nov 15, 2022

View reviewed changes

pda added 3 commits November 15, 2022 17:41

package js extracted from PipelineEvalCommand

a17963d

package resources (embedded filesystem) is also folded into package js.

js: debug logger, JS require() load order tweak

49f09c6

Now require("buildkite/*") will attempt to load from host filesystem if the requested file doesn't exist in the embedded filesystem.

js: better error/debug reporting, especially for require()

e5b88fe

pda force-pushed the pda/js-pipeline branch from 3774966 to e5b88fe Compare November 15, 2022 12:30

DrJosh9000 added the experiment label Feb 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Embedded JavaScript pipeline PoC #1829

Embedded JavaScript pipeline PoC #1829

pda commented Nov 14, 2022 •

edited

keithpitt Nov 14, 2022

keithpitt Nov 14, 2022

pda Nov 15, 2022

pda Nov 15, 2022

moskyb Nov 15, 2022

pda Nov 15, 2022

keithpitt Nov 25, 2022

pda commented Nov 15, 2022

davidwheeler123 commented Jan 17, 2023

dabarrell commented Mar 31, 2023

Embedded JavaScript pipeline PoC #1829

Are you sure you want to change the base?

Embedded JavaScript pipeline PoC #1829

Conversation

pda commented Nov 14, 2022 • edited

keithpitt Nov 14, 2022

Choose a reason for hiding this comment

keithpitt Nov 14, 2022

Choose a reason for hiding this comment

pda Nov 15, 2022

Choose a reason for hiding this comment

pda Nov 15, 2022

Choose a reason for hiding this comment

moskyb Nov 15, 2022

Choose a reason for hiding this comment

pda Nov 15, 2022

Choose a reason for hiding this comment

keithpitt Nov 25, 2022

Choose a reason for hiding this comment

pda commented Nov 15, 2022

davidwheeler123 commented Jan 17, 2023

dabarrell commented Mar 31, 2023

pda commented Nov 14, 2022 •

edited