Evaluation Default evaluator (#5092)

* init

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* rename module

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* revert black change

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* change module path

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* refactor

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* lazy load pyspark

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* revert export

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix curcit import

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix conftest.py

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* Revert "fix conftest.py"

This reverts commit 2ea29c6.

* fix tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* default evaluator

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update import

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update hash algo

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update import

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comment

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix lint

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add more tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix lint

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update shap explainer

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* remove scikitplot dep

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add pr curve

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add shap.summary_plot

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* log explainer

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* improve explainer code

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update shap init

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update explainer creating

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update predict_proba

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* refactor

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add multi-class metrics artifacts

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add log_loss metric

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* lazy load pyspark

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* address ben comments

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* prevent show shap logo, add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* support spark model

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add shap version check

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update docs, loose classifier label limit

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* multiclass classifier merge metrics/plots

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* zfill feature name

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add config max_num_classes_threshold_logging_roc_pr_curve_for_multiclass_classifier

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* refactor

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* improve label handling

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* refactor

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* add tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* black

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* increase plot dpi

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix test fixture

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix pylint

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update doc

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* use matplot rc_context

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix shap import

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* refactor EvaluationDataset

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* limit user specify shap algos

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* clean

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* update evaluation dataset

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* use svg fig

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* revert svg

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* curve dashline, legend display ap/roc, legend move out

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* linewidth 1

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* keyword arguments for evaluate, fix tests

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* mark abc.abstractmethod, kw args for ModelEvaluator methods

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix pylint

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

* fix pylint

Signed-off-by: Weichen Xu <weichen.xu@databricks.com>

mlflow/models/evaluation/artifacts.py

@@ -0,0 +1,23 @@
+import pandas as pd
+
+from mlflow.models.evaluation.base import EvaluationArtifact
+
+
+class ImageEvaluationArtifact(EvaluationArtifact):
+    def save(self, output_artifact_path):
+        self._content.save(output_artifact_path)
+
+    def _load_content_from_file(self, local_artifact_path):
+        from PIL.Image import open as open_image
+
+        self._content = open_image(local_artifact_path)
+        return self._content
+
+
+class CsvEvaluationArtifact(EvaluationArtifact):
+    def save(self, output_artifact_path):
+        self._content.to_csv(output_artifact_path, index=False)
+
+    def _load_content_from_file(self, local_artifact_path):
+        self._content = pd.read_csv(local_artifact_path)
+        return self._content