Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

control: add build API and builder daemon service #408

Merged
merged 7 commits into from
Mar 11, 2022

Commits on Mar 10, 2022

  1. Configuration menu
    Copy the full SHA
    1dc3ad5 View commit details
    Browse the repository at this point in the history
  2. parser: correct use of inference locate()

    The API is no longer fallible, and the Exists::Implicit case should be
    matched over instead.
    jgraettinger committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    d4d6873 View commit details
    Browse the repository at this point in the history
  3. cargo: update all dependencies

    * Fix a number of deprecation warnings.
    * Fix linker errors related to Go <=> Rust sqlite usage.
    jgraettinger committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    838bfa7 View commit details
    Browse the repository at this point in the history
  4. control: initial models, controllers, & tests for builds

    The most interesting bit is the mechanism for dequeing / updating builds
    on behalf of a forthcoming builder daemon service.
    
    It uses transaction-scoped postgres advisory locks to offer a build
    queue which can be processed in parallel, and which is tolerant to
    builder failures.
    
    Issue #400
    jgraettinger committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    1e95f72 View commit details
    Browse the repository at this point in the history
  5. control: add builder service for building builds

    The builder daemon periodically polls the database for a build to run.
    On finding one, it builds it in a local temporary directory using
    `flowctl api build`, and then tests it using `flowctl temp-data-plane`
    and `flowctl api test`.
    
    This commit conceptualizes each of these activities as "tasks", where
    log lines of each task are directly streamed into the build database as
    they occur. The build DB can then be its own source-of-truth for all of
    its outputs during the build process. We also preserve structured log
    information from the build process.
    
    Looking forward, activations and other background tasks will produce
    similar types of logs, and we can use the same mechanism to capture /
    persist / query them.
    
    For now, the build daemon is directly integrated into the control plane
    server. We'll probably want to separate these in the future.
    
    Issue #400
    jgraettinger committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    9b1c28e View commit details
    Browse the repository at this point in the history
  6. control: advisory locks => SELECT ... WITH UPDATE SKIP LOCKED

    Also add an index on builds.account_id.
    jgraettinger committed Mar 10, 2022
    Configuration menu
    Copy the full SHA
    fc1706c View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2022

  1. control: builder cleanups from review

    * Use new url_for pattern
    * State::Done => State::Success
    * Terminology: "task" => "job" for background jobs of the control plane.
    jgraettinger committed Mar 11, 2022
    Configuration menu
    Copy the full SHA
    c62e0d6 View commit details
    Browse the repository at this point in the history