Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report ML-powered query enablement in the init status report #907

Merged
merged 10 commits into from Feb 7, 2022

Conversation

henrymercer
Copy link
Contributor

@henrymercer henrymercer commented Feb 3, 2022

Adds information about whether ML-powered JS queries are enabled to the init status report. This will improve our observability by allowing us to compare metrics between runs with ML-powered JS queries on and off.

Specifically, we add a ml_powered_js_queries field to the init status report that contains:

  • The version string if the analysis will use a specific version of the pack
  • "latest" if the analysis will use the latest version of the pack
  • "false" if the analysis won't run any ML-powered JS queries, or if the analysis will run multiple ML-powered JS query packs

We report this information in the init status report rather than the analyze status report so we can gather data about timeouts.

Example status report:

{
  "workflow_run_id": 1790730464,
  "workflow_name": "Code Scanning - Action",
  "job_name": "CodeQL-Build",
  "analysis_key": ".github/workflows/codeql-analysis.yml:CodeQL-Build",
  "commit_oid": "6a99ab511b076566d0b9259a4102ac7f5c4b3e0c",
  "ref": "refs/heads/main",
  "action_name": "init",
  "action_ref": "1cddec9558c522dd219349d2149a48c46e203a8a",
  "action_oid": "unknown",
  "started_at": "2022-02-03T17:08:30.984Z",
  "action_started_at": "2022-02-03T17:08:30.984Z",
  "status": "success",
  "completed_at": "2022-02-03T17:08:37.758Z",
  "matrix_vars": "null",
  "disable_default_queries": "",
  "languages": "javascript",
  "ml_powered_js_queries": "~0.0.2" /* NEW */,
  "paths": "",
  "paths_ignore": "",
  "queries": "security-extended",
  "tools_input": "latest",
  "tools_resolved_version": "20220120",
  "workflow_languages": "javascript"
}

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Confirm the readme has been updated if necessary.
  • Confirm the changelog has been updated if necessary.

We report this information in the `init` status report rather than the
`analyze` status report so we can gather data about timeouts.
@henrymercer henrymercer marked this pull request as ready for review February 3, 2022 17:15
@henrymercer henrymercer requested a review from a team as a code owner February 3, 2022 17:15
Copy link
Contributor

@aeisenberg aeisenberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor thing, otherwise looks good.

src/util.ts Outdated Show resolved Hide resolved
src/util.test.ts Outdated Show resolved Hide resolved
@edoardopirovano
Copy link
Contributor

I believe you also need a PR such as https://github.com/github/hydro-schemas/pull/1845 to add the new field to the database and also a github/github PR to propagate the fields?

@henrymercer
Copy link
Contributor Author

I believe you also need a PR such as github/hydro-schemas#1845 to add the new field to the database and also a github/github PR to propagate the fields?

Thanks, these are in the works (see the internal tracking issue)!

When multiple ML-powered JS packs are in scope (an unsupported
scenario), the status report is "multiple".
src/config-utils.ts Outdated Show resolved Hide resolved
src/util.ts Outdated Show resolved Hide resolved
Some platforms that ingest this status report charge based on the
cardinality of the fields, so here we restrict the version strings we
support to a fixed set.
Also be more explicit about which version strings are reportable in
the code.
@henrymercer henrymercer merged commit 4eb03fb into main Feb 7, 2022
@henrymercer henrymercer deleted the henrymercer/report-ml-powered-query-enablement branch February 7, 2022 17:16
@github-actions github-actions bot mentioned this pull request Feb 7, 2022
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants