Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain why we run tests against dev versions #5244

Merged
merged 2 commits into from Jan 13, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
46 changes: 46 additions & 0 deletions .github/workflows/cross-version-testing.md
Expand Up @@ -112,6 +112,52 @@ the previous section:
| 1.0.1 | ✅ | The latest micro version of `1.0` |
| 1.1.dev | ✅ | The version installed by `install_dev` |

## Why do we run tests against development versions?

In cross-version testing, we run daily tests against both publicly available and pre-release
development versions for all dependent libraries that are used by MLflow.
This section explains why.

### Without dev version test

First, let's take a look at what would happen **without** dev version test.

```
|
├─ XGBoost merges a change on the master branch that breaks MLflow's XGBoost integration.
|
├─ MLflow 1.20.0 release date
|
├─ XGBoost 1.5.0 release date
├─ ❌ We notice the change here and might need to make a patch release if it's critical.
|
v
time
```

- We didn't notice the change until after XGBoost 1.5.0 was released.
- MLflow 1.20.0 doesn't work with XGBoost 1.5.0.

### With dev version test

Then, let's take a look at what would happen **with** dev version test.

```
|
├─ XGBoost merges a change on the master branch that breaks MLflow's XGBoost integration.
├─ ✅ Tests for the XGBoost integration fail -> We can notice the change and apply a fix for it.
|
├─ MLflow 1.20.0 release date
|
├─ XGBoost 1.5.0 release date
|
v
time
```

- We can notice the change **before XGBoost 1.5.0 is released** and apply a fix for it **before releasing MLflow 1.20.0**.
- MLflow 1.20.0 works with XGBoost 1.5.0.

## When do we run cross version tests?

1. Daily at 7:00 UTC using a cron scheduler.
Expand Down