New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Evaluate Api examples #5186
Merged
Merged
Evaluate Api examples #5186
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
81b59e5
init
WeichenXu123 ebf1cba
permlink
WeichenXu123 15f5f71
Merge branch 'master' into eval-api-examples
WeichenXu123 ec81ae4
update
WeichenXu123 07e9c3d
add readme
WeichenXu123 4b5b08b
fix lint
WeichenXu123 fcb2e75
update
WeichenXu123 e464eed
update
WeichenXu123 dc11f53
update
WeichenXu123 d1f49bd
update
WeichenXu123 3613b43
update
WeichenXu123 2fcf860
add artifact __repr__
WeichenXu123 File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
### MLflow evaluation Examples | ||
|
||
The examples in this directory illustrate how you can use the `mlflow.evaluate` API to evaluate a PyFunc model on the | ||
specified dataset using builtin default evaluator, and log resulting metrics & artifacts to MLflow Tracking. | ||
|
||
- Example `evaluate_on_binary_classifier.py` evaluates an xgboost `XGBClassifier` model on dataset loaded by | ||
`shap.datasets.adult`. | ||
- Example `evaluate_on_multiclass_classifier.py` evaluates a scikit-learn `LogisticRegression` model on dataset | ||
generated by `sklearn.datasets.make_classification`. | ||
- Example `evaluate_on_regressor.py` evaluate as scikit-learn `LinearRegression` model on dataset loaded by | ||
`sklearn.datasets.fetch_california_housing` | ||
|
||
#### Prerequisites | ||
|
||
``` | ||
pip install scikit-learn xgboost shap>=0.40 matplotlib | ||
``` | ||
|
||
#### How to run the examples | ||
|
||
Run in this directory with Python. | ||
|
||
``` | ||
python evaluate_on_binary_classifier.py | ||
python evaluate_on_multiclass_classifier.py | ||
python evaluate_on_regressor.py | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
import xgboost | ||
import shap | ||
import mlflow | ||
from sklearn.model_selection import train_test_split | ||
|
||
# train XGBoost model | ||
X, y = shap.datasets.adult() | ||
|
||
num_examples = len(X) | ||
|
||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42) | ||
|
||
model = xgboost.XGBClassifier().fit(X_train, y_train) | ||
|
||
eval_data = X_test | ||
eval_data["label"] = y_test | ||
|
||
with mlflow.start_run() as run: | ||
mlflow.sklearn.log_model(model, "model") | ||
model_uri = mlflow.get_artifact_uri("model") | ||
result = mlflow.evaluate( | ||
model_uri, | ||
eval_data, | ||
targets="label", | ||
model_type="classifier", | ||
dataset_name="adult", | ||
evaluators=["default"], | ||
) | ||
|
||
print(f"metrics:\n{result.metrics}") | ||
print(f"artifacts:\n{result.artifacts}") | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
import mlflow | ||
from sklearn.linear_model import LogisticRegression | ||
from sklearn.datasets import make_classification | ||
from sklearn.model_selection import train_test_split | ||
|
||
mlflow.sklearn.autolog() | ||
|
||
X, y = make_classification(n_samples=10000, n_classes=10, n_informative=5, random_state=1) | ||
|
||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42) | ||
|
||
with mlflow.start_run() as run: | ||
model = LogisticRegression(solver="liblinear").fit(X_train, y_train) | ||
model_uri = mlflow.get_artifact_uri("model") | ||
result = mlflow.evaluate( | ||
model_uri, | ||
X_test, | ||
targets=y_test, | ||
model_type="classifier", | ||
dataset_name="multiclass-classification-dataset", | ||
evaluators="default", | ||
evaluator_config={"log_model_explainability": True, "explainability_nsamples": 1000}, | ||
) | ||
|
||
print(f"run_id={run.info.run_id}") | ||
print(f"metrics:\n{result.metrics}") | ||
print(f"artifacts:\n{result.artifacts}") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
import mlflow | ||
from sklearn.datasets import fetch_california_housing | ||
from sklearn.linear_model import LinearRegression | ||
from sklearn.model_selection import train_test_split | ||
|
||
mlflow.sklearn.autolog() | ||
|
||
california_housing_data = fetch_california_housing() | ||
|
||
X_train, X_test, y_train, y_test = train_test_split( | ||
california_housing_data.data, california_housing_data.target, test_size=0.33, random_state=42 | ||
) | ||
|
||
with mlflow.start_run() as run: | ||
model = LinearRegression().fit(X_train, y_train) | ||
model_uri = mlflow.get_artifact_uri("model") | ||
|
||
result = mlflow.evaluate( | ||
model_uri, | ||
X_test, | ||
targets=y_test, | ||
model_type="regressor", | ||
dataset_name="california_housing", | ||
evaluators="default", | ||
feature_names=california_housing_data.feature_names, | ||
evaluator_config={"explainability_nsamples": 1000}, | ||
) | ||
|
||
print(f"metrics:\n{result.metrics}") | ||
print(f"artifacts:\n{result.artifacts}") | ||
WeichenXu123 marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line's output looks like:
We might want to implement
__str__
forEvaluationArtifact
for better string representation (e.g.<EvaluationArtifact file_name.png>
).cc @dbczumar
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can do this in follow-up updates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I add a
__repr__
for evaluation artifact , format is likeImageEvaluationArtifact(uri='...')