Error message "Object of type ndarray is not JSON serializable" when requesting a prediction to a Onnx model

[mcarmona99]

We found an error when trying to request inference to an ONNX model registered in MLflow (using MinIO as artifacts repository).

When sending a request to a MLflow server that deploys an Onnx model, the following response is shown:

{'status': {'code': -1, 'info': 'Object of type ndarray is not JSON serializable', 'reason': 'MICROSERVICE_INTERNAL_ERROR', 'status': 1}}

The manifest used to create the SeldonDeployment object is the following:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  name: pruebaonnx
  namespace: test-despliegue-01
spec:
  name: pruebaonnx
  predictors:
  - componentSpecs:
    - spec:
        containers:
        - env:
          - name: masked_proxy_vars
            value: masked
          livenessProbe:
            failureThreshold: 200
            httpGet:
              path: /health/ping
              port: http
              scheme: HTTP
            initialDelaySeconds: 80
            periodSeconds: 5
            successThreshold: 1
          name: classifier
          readinessProbe:
            failureThreshold: 200
            httpGet:
              path: /health/ping
              port: http
              scheme: HTTP
            initialDelaySeconds: 80
            periodSeconds: 5
            successThreshold: 1
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "sed -i 's/mlflow <= 2.5.0/mlflow <= 2.8.1/g' /microservice/requirements.txt"]
    graph:
      children: []
      implementation: MLFLOW_SERVER
      envSecretRefName: mlflow-server-seldon-init-container-s3-credentials
      modelUri:  s3://mlflow/.../artifacts_folder
      name: classifier
      type: MODEL
    name: default
    replicas: 1
  protocol: seldon

This snippet:

          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh", "-c", "sed -i 's/mlflow <= 2.5.0/mlflow <= 2.8.1/g' /microservice/requirements.txt"]

was needed to avoid the already fixed issue mlflow/mlflow#9185.

We have been investigating and the model was registered properly in Mlflow. We have confirmed this by using just mlflow to make the prediction, and it works (note that we are using specific functions created by us that call the mlflow and minio SDKs, custom_mlflow is just mlflow overriding the s3 artifact repository class):

The path saved at downloaded_model contains the artifacts generated by mlflow (saved in MinIO and downloaded to the host)

from modules.minio import functions as functions_minio
from modules.mlflow import functions
from modules.mlflow.custom_mlflow import custom_mlflow
import numpy as np

functions_minio.cached_get_minio_client(url="MASKED", user="MASKED", password="MASKED")
mlflow_client = functions.cached_get_mlflow_client(url="MASKED", minio_url="MASKED")

sample_data = np.array([[5.1, 3.5, 3.4, 2.2]])

# ...

# Load model as a PyFuncModel.
downloaded_model = "tests/mlflow/models/onnx_model_folder"
loaded_model = custom_mlflow.mlflow.pyfunc.load_model(downloaded_model)


print(loaded_model.predict(sample_data))

The output is:

{'output_label': array([1], dtype=int64), 'output_probability': [{0: 0.009999999776482582, 1: 0.6399998664855957, 2: 0.3499999940395355}]}

which is actually the expected output and what must be shown when requesting the prediction to the seldondeployment server.

We have also added the following snippet:

          - name: SELDON_DEBUG
            value: 'true'
          - name: LOG_LEVEL_ENV
            value: 'DEBUG'

and no extra information is shown in the logs.

$ microk8s kubectl logs ...-classifier-... -n test-despliegue-01
...
2023-11-17 15:26:39,217 - werkzeug:_log:224 - INFO:  <masked> - - [17/Nov/2023 15:26:39] "GET /health/ping HTTP/1.1" 200 -
2023-11-17 15:26:41,864 - seldon_core.wrapper:handle_generic_exception:53 - ERROR:  {'status': {'status': 1, 'info': 'Object of type ndarray is not JSON serializable', 'code': -1, 'reason': 'MICROSERVICE_INTERNAL_ERROR'}}
2023-11-17 15:26:41,864 - werkzeug:_log:224 - INFO:  <masked> - - [17/Nov/2023 15:26:41] "POST /predict HTTP/1.1" 500 -
2023-11-17 15:26:44,214 - werkzeug:_log:224 - INFO:  <masked> - - [17/Nov/2023 15:26:44] "GET /health/ping HTTP/1.1" 200 -
2023-11-17 15:26:44,215 - werkzeug:_log:224 - INFO:  <masked> - - [17/Nov/2023 15:26:44] "GET /health/ping HTTP/1.1" 200 -

To reproduce

Steps shown above.

Expected behaviour

We must get a response with the prediction. We have already tested with models created by other frameworks: keras, sklearn, pytorch, etc (all of them registered in MLFlow) and all the deployments work.

Environment

Charmed Kubeflow

• Kubernetes Cluster Version: 1.24

• Deployed Seldon System Images:

$ microk8s kubectl get --namespace seldon-system deploy seldon-co
ntroller-manager -o yaml  | grep seldonio
          value: docker.io/seldonio/seldon-core-executor:1.17.1
        image: docker.io/seldonio/seldon-core-operator:1.17.1

Model Details

Model generated with the following function:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

def generate_model():
    iris = load_iris()
    X, y = iris.data, iris.target
    X = X.astype(np.float32)
    X_train, X_test, y_train, y_test = train_test_split(X, y)
    clr = RandomForestClassifier()
    clr.fit(X_train, y_train)

    # Convert into ONNX format.
    from skl2onnx import to_onnx

    onnx_file_path = "tests/mlflow/models/onnx_model.onnx"
    onx = to_onnx(clr, X[:1])
    with open(onnx_file_path, "wb") as f:
            f.write(onx.SerializeToString())