Better SCM CI Integration Overview

• Workflows inside the App
  • Workflow triggering
  • Reporting back final build results
• Workflows Runs
• Workflows
• Testing
• Monitoring

NOTE: For now we only support local packages.

SCM CI Integration helps you to collaborate between OBS and the source code management (SCM) systems like GitHub, GitLab or Gitea. It helps to handle your package sources with those systems. This section describes the integration between SCMs and OBS.

Workflows inside the App

Workflow triggering

1. SCM triggers a webhook pointing to our trigger/workflow endpoint (someone opened or updated a PR).
2. Inside TriggerWorkflowController:
   • Extract the event, SCM, and payload from the incoming request.
   • Create a WorkflowRun
   • Create a ScmWebhook which contains a new payload created after the payload coming from the request.
   • Call Token::Workflow model passing one instance of WorkflowRun and ScmWebhook.
3. Token::Workflow model
   • Download the .obs/workflows.yml file from the target branch, not from the Pull request, using the Workflows::YAMLDownloader.
   • Call the Workflows::YAMLToWorkflowsService which returns a collection of Workflow objects for each workflow defined in the configuration file.
   • Send the initial report back to the SCM with "pending" state. Using ScmInitialStatusReporter.
   • Call each workflow.
   • Send the initial report back to the SCM with "success" state. Using ScmInitialStatusReporter.
4. Workflow model
   1. Destroy the existing project in case the SCM closes a PR.
   2. Restore a previously existing project in case the SCM re-opens a PR.
   3. Handle the steps in other cases:
      • Extract the steps from the workflow (Workflow::Step::BranchPackageStep, Workflow::Step::LinkPackageStep and Workflow::Step::ConfigureRepositoriesStep etc.)
      • Call every step.
      • Collects the artifacts of each step with Workflows::ArtifactsCollector.
5. Workflow::Step model (and its inherited Workflow::Step::*)
   • Perform the step.
   • Create two Event::Subscription (storing the token ID in the event_subscriptions.token_id column) and in that Event::Subscription, we store the required information to be able to report back to the SCM (repository owner, etc... see below in Reporting Back to SCM).

NOTE: To make the service run on a linked package (link_package step) we had to make it a "project service" by adding a package named _project into the target project. The package contains a _service file with a correct definition of a service.

Reporting back final build results

1. Whenever a build finishes in the Backend, a new Event::BuildFail or Event::BuildSuccess is created and a new ReportToScmJob is queued in scm queue. Follow this link for a better understanding of the Events system.
   • ReportToScmJob inherits from CreateJob.
   • CreateJob is only for jobs dealing with events, since its perform method expects an event_id.
   • For the jobs to be created, event classes like Event::BuildFail and Event::BuildSuccess need to have the following code: create_jobs :report_to_scm_job. This will then be picked up in the after_create :perform_create_jobs callback in the Event::Base model.
   • If the provided symbol in the create_jobs method is a valid job inheriting from CreateJob, then the corresponding job will be queued.
   • The scm queue service is defined in systemd/obs-delayedjob-queue-scm.service
   • We pass the event_id to the ReportToScmJob. Using the event_id we are fetching the event object, we search for the event subscriptions that match the given event object (eventtype and package), loop over them and call GitlabStatusReporter / GithubStatusReporter / GiteaStatusReporter (depending on the SCM) for each one.
2. GitlabStatusReporter / GithubStatusReporter / GiteaStatusReporter service, called by ReportToScmJob,
   • Report status back to the SCM for the corresponding commit
   • Create an instance of ScmStatusReport, which represents the communication between OBS and the SCM. It stores the status (success or fail) and the error message in case of failure. It's associated with the workflow run.

Communication Between OBS and the SCM

We use Octokit as client for GitHub API.

We use this wrapper for the GitLab API.

For Gitea we created our own module: GiteaAPI::V1::Client (app/apis/gitea_api/v1/client.rb).

Workflow Runs

Whenever a workflow token is triggered by an incoming webhook, a workflow run is created to track what is happening, thus helping users understand how their workflow behaves. In addition to the headers and payload of the incoming webhook, workflow runs store the response we send back to the SCM and to which URL it is sent. Finally, the status is saved, so users know if the workflow run succeeded or potentially failed due to an error.