Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multicast UDP traffic appears to cause Booster to encounter watchdog resets #337

Open
ryan-summers opened this issue Oct 11, 2023 · 10 comments
Labels
bug Something isn't working

Comments

@ryan-summers
Copy link
Member

ryan-summers commented Oct 11, 2023

While testing out some multicast UDP traffic, I observed booster reboot twice. The service then indicated that a watchdog occurred. It's not clear if this was related to the Multicast traffic or not, but seemed to only occur when executing the Stabilizer streaming HITL with a streaming IP of 224.192.168.1

> service
Version             : Unspecified [release]
Hardware Revision   : v1.5
Rustc Version       : rustc 1.72.0 (5680fa18f 2023-08-23)
Features            :
Detected Phy        : W5500
Panic Info          : None
Watchdog Detected   : true

>
@ryan-summers ryan-summers added the bug Something isn't working label Oct 11, 2023
@ryan-summers
Copy link
Member Author

This was running on 1ea1124

@jordens
Copy link
Member

jordens commented Oct 11, 2023

n.b.
I'm also streaming multicast in the same net on 239.34.16.10.
I would not use multicast in 224.0.0.0/8. Let's stick to the admin scoped 239.0.0.0/8.

@ryan-summers
Copy link
Member Author

This appears pretty reproducible with 239.192.168.1 in my network. I'm looking into what's happening

@ryan-summers
Copy link
Member Author

Disabling the watchdog indicates that booster remains operational throughout the whole event and doesn't encounter a true lockup. What I believe may be happening is that there's so much traffic incoming on the PHY that smoltcp and the W5500 need to process that it slows down processing of the application in general. I wonder if there's a way we can handle events like this where there's excessive ethernet traffic.

@jordens
Copy link
Member

jordens commented Oct 12, 2023

It should not be seeing that traffic at all. The switch doesn't flood.
It might see IGMP traffic but that's not much at all and always has been there.

@ryan-summers
Copy link
Member Author

I see the green ethernet LED remain consistently on during the Stabilizer stream period, indicating that Booster is indeed receiving excessive packets.

I opened smoltcp-rs/smoltcp#848 around this, but this could be due to a cheap switch that isn't handling multicast properly as well.

@jordens
Copy link
Member

jordens commented Oct 12, 2023

That doesn't seem to be the case here. I'm seeing close to zero traffic towards stabilizer (while another one is streaming multicast). As expected.

@jordens
Copy link
Member

jordens commented Oct 12, 2023

This is likely then a W5500 or w5500 specific bug.

@ryan-summers
Copy link
Member Author

Do you have a router and/or managed switch in between? The switch I'm using with my local network is just some cheap unmanaged unit. I suspect that it's forwarding the multicast traffic to all of the ports regardless of subscription to the multicast group.

@jordens
Copy link
Member

jordens commented Oct 12, 2023

There is a cisco 2960L in between. Whether that one can be called "managed" or not is debatable. But it does behave properly regarding non-flooding of multicast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants