Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Explain why we run tests against dev versions #5244

Merged
merged 2 commits into from Jan 13, 2022
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
45 changes: 45 additions & 0 deletions .github/workflows/cross-version-testing.md
Expand Up @@ -112,6 +112,51 @@ the previous section:
| 1.0.1 | ✅ | The latest micro version of `1.0` |
| 1.1.dev | ✅ | The version installed by `install_dev` |

## Why do we run tests against development versions?

In cross-version testing, we daily run tests against not only publicly available versions but also
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"... we run daily tests against both publicly available and pre-release development versions for all dependent libraries that are used by MLflow. This section explains why."

development versions installed from GitHub or nightly wheels. This section explains why.

### Without dev version test

First, let's take a look at what would happen **without** dev version test.

```
|
├─ XGBoost merges a change on the master branch that breaks MLflow's XGBoost integration.
|
├─ MLflow 1.20.0 release date
|
├─ XGBoost 1.5.0 release date
├─ ❌ We notice the change here and might need to make a patch release if it's critical.
|
v
time
```

- We can't notice the change until XGBoost 1.5.0 is released.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"We didn't notice the change until after XGBoost 1.5.0 was released"

- MLflow 1.20.0 doesn't work with XGBoost 1.5.0.

### With dev version test

Then, let's take a look at what would happen **with** dev version test.

```
|
├─ XGBoost merges a change on the master branch that breaks MLflow's XGBoost integration.
├─ ✅ Tests for the XGBoost integration fail -> We can notice the change and apply a fix for it.
|
├─ MLflow 1.20.0 release date
|
├─ XGBoost 1.5.0 release date
|
v
time
```

- We can notice the change **before XGBoost 1.5.0 is released** and apply a fix for it **before releasing MLflow 1.20.0**.
- MLflow 1.20.0 works with XGBoost 1.5.0.

## When do we run cross version tests?

1. Daily at 7:00 UTC using a cron scheduler.
Expand Down