Introducing network partitions #3068
-
|
Hey, Is it possible to introduce network partitions in shadow? Judging from the docs, which state that 'Shadow requires that the network graph is connected such that there exists at least one path (a series of one or more edges) between every pair of nodes', this does not seem to be the case. But given the description of Shadow ('Shadow is a discrete-event network simulator that directly executes real application code, enabling you to simulate distributed systems with thousands of network-connected processes in realistic and scalable private network experiments '), it seems that modelling network partitions would be a useful feature for such a system. So is this either planned or even currently possible, or would this feature violate some key design invariant of Shadow? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
|
Can you describe what you're looking for from partitioning the network? If the network was partitioned such that two groups of hosts couldn't communicate, it would effectively be like running two separate simulations, one with each sub-network. |
Beta Was this translation helpful? Give feedback.
-
|
I suspect they're talking about network partitions that occur during the course of execution (which wouldn't work anyway with current Shadow, since the network is statically constructed during bootstrap). |
Beta Was this translation helpful? Give feedback.
-
|
Yes this is what I'm looking for. Testing e.g. how a leader-based system with replication performs when a leader is dynamically disconnected from its replicas seems like a common test scenario for such a distributed system. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah the network model is just a static connectivity graph and we don't currently have a rich routing layer in which it would be natural to introduce dynamic route changes. One way to work around this and simulate failures of connections from leader nodes to replica nodes would be to introduce a forwarding proxy that forwards traffic between pairs of nodes, and then you could implement your failure logic inside of that proxy. The proxy would be pretty simply, either "blocking" connections between some nodes if certain conditions are met, or else "allowing" connections by forwarding the application traffic. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. |
Beta Was this translation helpful? Give feedback.
Yeah the network model is just a static connectivity graph and we don't currently have a rich routing layer in which it would be natural to introduce dynamic route changes. One way to work around this and simulate failures of connections from leader nodes to replica nodes would be to introduce a forwarding proxy that forwards traffic between pairs of nodes, and then you could implement your failure logic inside of that proxy. The proxy would be pretty simply, either "blocking" connections between some nodes if certain conditions are met, or else "allowing" connections by forwarding the application traffic.