You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After exporting the model for prediction, I found that setting de.enable_inference_mode() is required.
and comment is : de.enable_inference_mode()
TrainableWrapper.read_value` is not thread-safe that causes threads
competition and Out-Of-Bound exception in concurrent serving scenario.
To resolve this, we define the ModelMode APIs to instruct
the TrainableWrapper to build a different thread-safe sub-graph
for 'TrainableWrapper.read_value' on inference mode.
But after export the model and use tensorflow serving or trion to inference,how to set enable_inference_mode(),or is it necessary ?
The text was updated successfully, but these errors were encountered:
enable_inference_mode is used to change the graph building logic inner TFRA. It would be reduce two times memory copy in TrainableWrapper which are IDs and Embedding Values.
After exporting the model for prediction, I found that setting de.enable_inference_mode() is required.
and comment is :
de.enable_inference_mode()
TrainableWrapper.read_value` is not thread-safe that causes threads
competition and Out-Of-Bound exception in concurrent serving scenario.
To resolve this, we define the
ModelMode
APIs to instructthe
TrainableWrapper
to build a different thread-safe sub-graphfor 'TrainableWrapper.read_value' on inference mode.
But after export the model and use tensorflow serving or trion to inference,how to set enable_inference_mode(),or is it necessary ?
The text was updated successfully, but these errors were encountered: