Support KFServing API V2 predict protocol

[parano]

About KFServing API V2 predict protocol: https://github.com/kubeflow/kfserving/tree/master/docs/predict-api/v2

The Predict Protocol, version 2 is a set of HTTP/REST and GRPC APIs for inference / prediction servers. By implementing this protocol both inference clients and servers will increase their utility and portability by being able to operate seamlessly on platforms that have standardized around this protocol.

The protocol is composed of a required set of APIs that must be implemented by a compliant server. This required set of APIs is described in required_api.md. The GRPC proto specification for the required APIs is available.

This is to add an option to BentoML API server, that enables a set of special endpoints that are compatible with KFServing API V2 protocol. It will make BentoML API server work much nicer with other tools in the kubeflow eco-system.

• Predict protocol V2 spec: https://github.com/kubeflow/kfserving/blob/master/docs/predict-api/v2/required_api.md#tensor-data-types
• KFServing roadmap on V2 protocol support is targeting Q3 2020 https://github.com/kubeflow/kfserving/blob/master/ROADMAP.md#v05-api-stabilization-and-tco-reduction-eta-end-of-q3

[pncnmnp]

@yubozhao
Kishore and I are interested in contributing to this issue.

[parano]

Hi @pncnmnp @Kishore - it probably makes more sense to first implement gRPC support in bentoML #703 before supporting KFServing V2's predict protocol

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[yonil7]

Is this feature already supported?

[parano]

@yonil7 not yet, is this feature a blocker for you? would love to learn more

[yonil7]

I was just looking for a standard inference REST API and I was surprised to see there is no such standard.
The closest I could find was KServe predict protocol v2. But I think their infer API is unnecessarily too complex and verbose.

On the other hand, GCP Vertex AI predict API (which is almost identical to GCP AI Platform predict API and Tensorflow Serving predict API) has the exact same features/abilities in a much more elegant API - any single instance / prediction and the request parameters is just a JSON object (can be number, null, bool, string, (nested) list, (nested) object)

[parano]

@yonil7 bentoML is trying to establish such a standard, and it takes a very different approach compared to Kserve's protocol. Essentially bentoML defines how an HTTP Request/Response is converted to and from a Python object that data scientist's code will consider as input to their inference function.

[wolvever]

KServe predict protocol is supported in KServe, Seldon Core and Triton Inference Server. I think it make sense to support this protocol to allow users switch different frameworks. We are currently using Triton Inference Server to serve our own models for historical reason. And we also want to provide BentoML to our collaborators to simplify serving and deployment. But our product relied on this predict protocol, and we can't provide BentoML now.