A/B Model Traffic split is not working as expected after edit of Experiment CR

[DineshKrishnanG]

Describe the bug

A/B Model Traffic split is not working as expected after edit of Experiment CR.

Based on the Config Dump from envoy, the weighted routes keeps getting appended. It should replace the old weight with updated weight. Instead it maintains all the previous configuration.
The issue might be related to this, as the traffic split happens correctly when create Experiment with 100% and 0% first time itself.

To reproduce

1. Create a Experiment with 50% traffic split ( Reference: https://docs.seldon.io/projects/seldon-core/en/v2.6.0/contents/examples/k8s-examples.html#experiment)
2. Verify the request goes to both the models - SUCCESS
3. Modify the Experiment to 0% traffic split to 'iris' and 100% traffic split to iris2
4. Invoke the inference calls using REST/GRPC
5. Verify the request goes to only model with iris2(100%) and not to iris(0%) - FAIL
   - > Traffic is routing to both iris and iris2 model

Expected behaviour

The traffic split should happen based on the edited configuration.

E.g., When weight is updated from 50% to 0%, traffic should not be routed to the model.
Expected Behaviour - Traffic to be only routed to iris2 model
Current Behaviour - Traffic is routing to both iris and iris2 model

Based on the Config Dump from envoy, the weighted routes keeps getting appended. It should replace the old weight with updated weight. Instead it maintains all the previous configuration.
The issue might be related to this, as the traffic split happens correctly when Experiment is created with 100% and 0% first time itself.

Environment

Kubernetes version : 1.27.7
Seldon Version : 2.6.0

Model Details

• Images of your model: [Output of: kubectl get seldondeployment -n <yourmodelnamespace> <seldondepname> -o yaml | grep image: where <yourmodelnamespace>]
• Logs of your model: [You can get the logs of your model by running kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>]

[KengoA]

@DineshKrishnanG Thank you for raising this. I created the experiment as per the documentation using 2.7.0 and the traffic split seems to work fine after the editing Experiment CR. Could you expand on how you're modifying the experiment and provide more logs if possible?