Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation Default evaluator #5092

Merged
merged 128 commits into from Jan 10, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
e2c5aff
init
WeichenXu123 Nov 15, 2021
91a6e1c
update
WeichenXu123 Nov 15, 2021
071e86b
update
WeichenXu123 Nov 16, 2021
e7d8a76
update
WeichenXu123 Nov 16, 2021
7499ac6
update
WeichenXu123 Nov 16, 2021
f1819a4
update
WeichenXu123 Nov 16, 2021
aa397e1
update
WeichenXu123 Nov 16, 2021
981164a
update
WeichenXu123 Nov 16, 2021
593bec8
update
WeichenXu123 Nov 17, 2021
2240d0a
update
WeichenXu123 Nov 18, 2021
0e54b97
update
WeichenXu123 Nov 18, 2021
6ee4367
update
WeichenXu123 Nov 19, 2021
15712e0
update
WeichenXu123 Nov 19, 2021
63b1d2b
update
WeichenXu123 Nov 21, 2021
afc52e8
update
WeichenXu123 Nov 21, 2021
feddc1e
rename module
WeichenXu123 Nov 21, 2021
27814cf
address comments
WeichenXu123 Nov 22, 2021
fc9cf1a
address comments
WeichenXu123 Nov 22, 2021
1f0c12d
Merge branch 'master' into eval_api
WeichenXu123 Nov 22, 2021
d642309
revert black change
WeichenXu123 Nov 22, 2021
2a67297
change module path
WeichenXu123 Nov 22, 2021
66f760d
update
WeichenXu123 Nov 22, 2021
f7d6738
update
WeichenXu123 Nov 23, 2021
c7f0360
update
WeichenXu123 Nov 23, 2021
c2b673e
update
WeichenXu123 Nov 23, 2021
b046af3
address comments
WeichenXu123 Nov 25, 2021
d7620e6
Merge branch 'master' into eval_api
WeichenXu123 Nov 27, 2021
14c8e82
fix
WeichenXu123 Nov 28, 2021
25101f9
refactor
WeichenXu123 Nov 28, 2021
a4db525
lazy load pyspark
WeichenXu123 Nov 28, 2021
035346f
revert export
WeichenXu123 Nov 28, 2021
3422781
fix curcit import
WeichenXu123 Nov 28, 2021
0ad527e
update tests
WeichenXu123 Nov 28, 2021
2ea29c6
fix conftest.py
WeichenXu123 Nov 29, 2021
6bcbb0c
Revert "fix conftest.py"
WeichenXu123 Nov 29, 2021
46f3264
fix tests
WeichenXu123 Nov 29, 2021
d310359
update doc
WeichenXu123 Nov 29, 2021
1b3519b
update
WeichenXu123 Nov 30, 2021
09099de
default evaluator
WeichenXu123 Nov 30, 2021
2f3faa6
update
WeichenXu123 Nov 30, 2021
9aff5e8
fix
WeichenXu123 Nov 30, 2021
1dd0617
fix
WeichenXu123 Dec 1, 2021
57012d3
address comments
WeichenXu123 Dec 1, 2021
cf5cb34
merge master
WeichenXu123 Dec 1, 2021
21538cb
fix doc
WeichenXu123 Dec 1, 2021
812f0df
fix doc
WeichenXu123 Dec 1, 2021
cc2ac8e
update import
WeichenXu123 Dec 1, 2021
226d54b
fix doc
WeichenXu123 Dec 1, 2021
c20ccdc
update hash algo
WeichenXu123 Dec 1, 2021
68842fe
update import
WeichenXu123 Dec 2, 2021
755639b
address comment
WeichenXu123 Dec 2, 2021
73cd704
add tests
WeichenXu123 Dec 2, 2021
010f225
fix lint
WeichenXu123 Dec 2, 2021
e2f9c89
add tests
WeichenXu123 Dec 3, 2021
22b0bbd
Merge branch 'master' into eval_api
WeichenXu123 Dec 3, 2021
b1b34f8
add more tests
WeichenXu123 Dec 3, 2021
a51bd1f
add tests
WeichenXu123 Dec 3, 2021
c82eaa6
fix lint
WeichenXu123 Dec 3, 2021
96709eb
update shap explainer
WeichenXu123 Dec 3, 2021
04fb796
merge master
WeichenXu123 Dec 8, 2021
dc1b32d
address comments
WeichenXu123 Dec 8, 2021
11a388b
update
WeichenXu123 Dec 9, 2021
be4724f
update
WeichenXu123 Dec 9, 2021
14adf5d
update
WeichenXu123 Dec 9, 2021
4c38651
update
WeichenXu123 Dec 12, 2021
cbc0c32
update
WeichenXu123 Dec 13, 2021
503d26e
update
WeichenXu123 Dec 13, 2021
eae32d4
update
WeichenXu123 Dec 13, 2021
a4b8b60
remove scikitplot dep
WeichenXu123 Dec 13, 2021
1bc630b
add pr curve
WeichenXu123 Dec 13, 2021
06ce7e8
add shap.summary_plot
WeichenXu123 Dec 13, 2021
1016fa0
log explainer
WeichenXu123 Dec 13, 2021
3152dd5
Merge branch 'master' into eval-api-default-eval
WeichenXu123 Dec 14, 2021
c055060
address comments
WeichenXu123 Dec 14, 2021
849656d
address comments
WeichenXu123 Dec 14, 2021
1705fbc
improve explainer code
WeichenXu123 Dec 15, 2021
721b824
address comments
WeichenXu123 Dec 15, 2021
ebd495f
update
WeichenXu123 Dec 15, 2021
a3c5669
update
WeichenXu123 Dec 16, 2021
0936484
address comments
WeichenXu123 Dec 16, 2021
20f5d12
update shap init
WeichenXu123 Dec 16, 2021
a55dcd8
update explainer creating
WeichenXu123 Dec 16, 2021
4b23209
update predict_proba
WeichenXu123 Dec 16, 2021
2471afb
address comments
WeichenXu123 Dec 17, 2021
3abbe04
refactor
WeichenXu123 Dec 20, 2021
74775e1
add multi-class metrics artifacts
WeichenXu123 Dec 20, 2021
db63d79
update doc
WeichenXu123 Dec 20, 2021
1618b8a
add log_loss metric
WeichenXu123 Dec 20, 2021
1b49dc9
lazy load pyspark
WeichenXu123 Dec 20, 2021
df71505
address ben comments
WeichenXu123 Dec 21, 2021
21a38d0
Merge branch 'master' into eval-api-default-eval
WeichenXu123 Dec 21, 2021
d386233
fix
WeichenXu123 Dec 21, 2021
17e6b0c
prevent show shap logo, add tests
WeichenXu123 Dec 22, 2021
836f748
support spark model
WeichenXu123 Dec 22, 2021
19fb010
add tests
WeichenXu123 Dec 22, 2021
0241e0a
add shap version check
WeichenXu123 Dec 23, 2021
c94612e
update docs, loose classifier label limit
WeichenXu123 Dec 23, 2021
47bde7b
add tests
WeichenXu123 Dec 23, 2021
653b1f3
multiclass classifier merge metrics/plots
WeichenXu123 Dec 28, 2021
c6b7ed6
zfill feature name
WeichenXu123 Dec 29, 2021
f414cc0
update doc
WeichenXu123 Dec 29, 2021
ea921d7
add config max_num_classes_threshold_logging_roc_pr_curve_for_multicl…
WeichenXu123 Dec 29, 2021
e63e197
refactor
WeichenXu123 Dec 29, 2021
946a7cc
update tests
WeichenXu123 Dec 29, 2021
3b9553a
improve label handling
WeichenXu123 Dec 31, 2021
5368caf
refactor
WeichenXu123 Jan 1, 2022
c5f93cc
add tests
WeichenXu123 Jan 2, 2022
19d23aa
black
WeichenXu123 Jan 2, 2022
34a3bf7
Merge branch 'master' into eval-api-default-eval
WeichenXu123 Jan 4, 2022
74cc310
update
WeichenXu123 Jan 4, 2022
2924f8c
increase plot dpi
WeichenXu123 Jan 4, 2022
cf5d095
fix test fixture
WeichenXu123 Jan 5, 2022
923a6a9
fix pylint
WeichenXu123 Jan 5, 2022
f899fcf
update doc
WeichenXu123 Jan 5, 2022
fd6465b
use matplot rc_context
WeichenXu123 Jan 5, 2022
2f90049
fix shap import
WeichenXu123 Jan 5, 2022
4c4e909
refactor EvaluationDataset
WeichenXu123 Jan 5, 2022
4acf9ea
limit user specify shap algos
WeichenXu123 Jan 5, 2022
09d4842
clean
WeichenXu123 Jan 5, 2022
8e018f4
update evaluation dataset
WeichenXu123 Jan 6, 2022
985447a
use svg fig
WeichenXu123 Jan 6, 2022
57dec63
revert svg
WeichenXu123 Jan 6, 2022
c497381
curve dashline, legend display ap/roc, legend move out
WeichenXu123 Jan 6, 2022
9707f74
linewidth 1
WeichenXu123 Jan 6, 2022
21411b3
keyword arguments for evaluate, fix tests
WeichenXu123 Jan 9, 2022
84c5398
mark abc.abstractmethod, kw args for ModelEvaluator methods
WeichenXu123 Jan 10, 2022
4641183
fix pylint
WeichenXu123 Jan 10, 2022
f561e4f
fix pylint
WeichenXu123 Jan 10, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
23 changes: 23 additions & 0 deletions mlflow/models/evaluation/artifacts.py
@@ -0,0 +1,23 @@
import pandas as pd

from mlflow.models.evaluation.base import EvaluationArtifact


class ImageEvaluationArtifact(EvaluationArtifact):
def save(self, output_artifact_path):
self._content.save(output_artifact_path)

def _load_content_from_file(self, local_artifact_path):
from PIL.Image import open as open_image
WeichenXu123 marked this conversation as resolved.
Show resolved Hide resolved

self._content = open_image(local_artifact_path)
return self._content


class CsvEvaluationArtifact(EvaluationArtifact):
def save(self, output_artifact_path):
self._content.to_csv(output_artifact_path, index=False)

def _load_content_from_file(self, local_artifact_path):
self._content = pd.read_csv(local_artifact_path)
return self._content