Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: daily workflow testing more versions in back-compat #5180

Open
bradleystachurski opened this issue May 1, 2024 · 1 comment
Open

CI: daily workflow testing more versions in back-compat #5180

bradleystachurski opened this issue May 1, 2024 · 1 comment
Labels
CI Continuous integration

Comments

@bradleystachurski
Copy link
Member

bradleystachurski commented May 1, 2024

Now that we're using self-hosted CI runners, we should create a daily workflow that runs backwards-compatibility tests against more versions.

CI for PRs only test compatibility vs the oldest supported version (v0.2.1 as-of creating this issue). We originally decided to run against fewer versions to improve developer experience for PRs, but this makes it possible for backwards-incompatible changes to go undetected (see: #5171).

We should also consider other tests that don't fit within PR CI workflows but would be useful to run in daily workflows, e.g. upgrade tests, but that's outside the scope of this issue.

Other considerations:

  • Use the full matrix instead of the partial
    • Might take too long against multiple versions, so we may still use the partial matrix
  • Run the daily cron during low volume hours (midnight PT?)
@bradleystachurski bradleystachurski added the CI Continuous integration label May 1, 2024
@elsirion
Copy link
Contributor

elsirion commented May 2, 2024

Yeah, I think we could make the hourly check more extensive but reduce frequency, possibly to daily or a few times around low demand hours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Continuous integration
Projects
None yet
Development

No branches or pull requests

2 participants