[k8s] Ability to test against bad network conditions

[pepoviola]

The Vision

Zombienet clusters and internal testnets are usually spawned within a single data center, where network speeds are often close to optimal. Unfortunately, in the real world things are not so ideal. Therefore, I'd propose to extend zombienet functionality w/ ability to simulate bad & adversary network conditions.
(extracted from #903).

The Plan

We want to first explore this scenarios using k8s (with chaos-mesh, since is already installed in our cluster) but we don't expose any way to configure the scenario for the test.

Completed tasks

- [x] https://github.com//issues/927 - [x] https://github.com/paritytech/zombienet-sdk/issues/18 - [x] https://github.com/paritytech/zombienet-sdk/issues/17

1302 Umbrella issue for tracking Delay tasks (latency/jitter) in both network config and test-runner.

1336 Umbrella issue for partition/isolation in both network config / test-runner.

[emamihe]

Done @pepoviola
I checked the configuration of installed chaos-mesh and I found it installed wrongly after I did some test and found that the configs are not working. Hence I reinstalled the chaos-mesh with correct config meaning that it should be considered that GKE uses containerd so it is very important that this setting take place that simulation happens correctly.

so in order to let's say adding the latency to the network within the namespace like hamid2 the config below is needed:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: delay
spec:
  action: delay
  mode: all
  selector:
    namespaces:
      - hamid2
  delay:
    latency: '100ms'
    correlation: '100'
    jitter: '0ms'

and here is the result after applying this manifest per namespace:

so as you see above the 100ms latency added to the connection.

clsoing the issue.

[pepoviola]

Hi @emamihe, is also possible to add latency to pods using those as selector right?
Thx!

[emamihe]

yes @pepoviola

delay:
    latency: '100ms'

does this job

[pepoviola]

yes @pepoviola

delay:
    latency: '100ms'

does this job

Just to clarify, we should apply a manifest with the selector for the pods. E.g.

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: delay
spec:
  action: delay
  mode: all
  selector:
    pod:
      - hamid2
  delay:
    latency: '100ms'
    correlation: '100'
    jitter: '0ms'

Can you confirm please?
Thx!

[emamihe]

Regarding the selector part if you want to mention exact pod name it should be used the term pods not pod
and as the value document says:

KIND:     NetworkChaos
VERSION:  chaos-mesh.org/v1alpha1

FIELD:    pods <map[string][]string>

DESCRIPTION:
     Pods is a map of string keys and a set values that used to select pods. The
     key defines the namespace which pods belong, and the each values is a set
     of pod names.

Hence the final manifest looks like:

apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: delay
spec:
  action: delay
  mode: all
  selector:
    pods:
       zombie-xyzasdasdsqweasdasd:
       - alice
       - bob
  delay:
    latency: '100ms'
    correlation: '100'
    jitter: '0ms'

[pepoviola]

Closing here, I will re-org this initiative in other cards/issues.

[pepoviola]

I will re-open this issue to use it as umbrella issue for the pending tasks.