Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tailscale.com/tstest/integration.TestTwoNodes is flaky #11960

Open
andrew-d opened this issue May 1, 2024 · 1 comment · Fixed by #12026
Open

tailscale.com/tstest/integration.TestTwoNodes is flaky #11960

andrew-d opened this issue May 1, 2024 · 1 comment · Fixed by #12026

Comments

@andrew-d
Copy link
Member

andrew-d commented May 1, 2024

Seen in the following action run, among others:
https://github.com/tailscale/tailscale/actions/runs/8913374664/job/24478740954?pr=11958

=== RUN   TestTwoNodes
=== PAUSE TestTwoNodes
=== CONT  TestTwoNodes
    integration_test.go:1005: DERP httpsrv listener: 127.0.0.1:45959
    integration_test.go:362: node1 SOCKS5 addr: 127.0.0.1:37391
    integration_test.go:363: node2 SOCKS5 addr: 127.0.0.1:33433
    integration_test.go:367: Running /tmp/1534209587/tailscale --socket=/tmp/TestTwoNodes1962935569/001/tailscale.sock up --login-server=http://127.0.0.1:33403/ --reset ...
FAIL	tailscale.com/tstest/integration
andrew-d added a commit that referenced this issue May 1, 2024
@andrew-d
tstest/integration: mark TestTwoNodes as flaky
39e6649 
I couldn't reproduce this locally, but it's failing pretty consistently
for me in CI.

Updates #11960

Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Id78b063a3b8768e2dfe47b5ab1c0cab36dcb53eb
@andrew-d andrew-d mentioned this issue May 1, 2024
Closed
bradfitz added a commit that referenced this issue May 6, 2024
@bradfitz
cmd/tailscale/cli: don't start WatchIPNBus until after up's initial S…
7e1b0d8 
…tart

The CLI "up" command is a historical mess, both on the CLI side and
the LocalBackend side. We're getting closer to cleaning it up, but in
the meantime it was again implicated in flaky tests.

In this case, the background goroutine running WatchIPNBus was very
occasionally running enough to get to its StartLoginInteractive call
before the original goroutine did its Start call. That meant
integration tests were very rarely but sometimes logging in with the
default control plane URL out on the internet
(controlplane.tailscale.com) instead of the localhost control server
for tests.

This also might've affected new Headscale etc users on initial "up".

Fixes #11960
Fixes #11962

Change-Id: I36f8817b69267a99271b5ee78cb7dbf0fcc0bd34
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
bradfitz added a commit that referenced this issue May 6, 2024
@bradfitz
cmd/tailscale/cli: don't start WatchIPNBus until after up's initial S…
acf4768 
…tart

The CLI "up" command is a historical mess, both on the CLI side and
the LocalBackend side. We're getting closer to cleaning it up, but in
the meantime it was again implicated in flaky tests.

In this case, the background goroutine running WatchIPNBus was very
occasionally running enough to get to its StartLoginInteractive call
before the original goroutine did its Start call. That meant
integration tests were very rarely but sometimes logging in with the
default control plane URL out on the internet
(controlplane.tailscale.com) instead of the localhost control server
for tests.

This also might've affected new Headscale etc users on initial "up".

Fixes #11960
Fixes #11962

Change-Id: I36f8817b69267a99271b5ee78cb7dbf0fcc0bd34
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
bradfitz added a commit that referenced this issue May 6, 2024
@bradfitz
cmd/tailscale/cli: don't start WatchIPNBus until after up's initial S…
0d267e5 
…tart

The CLI "up" command is a historical mess, both on the CLI side and
the LocalBackend side. We're getting closer to cleaning it up, but in
the meantime it was again implicated in flaky tests.

In this case, the background goroutine running WatchIPNBus was very
occasionally running enough to get to its StartLoginInteractive call
before the original goroutine did its Start call. That meant
integration tests were very rarely but sometimes logging in with the
default control plane URL out on the internet
(controlplane.tailscale.com) instead of the localhost control server
for tests.

This also might've affected new Headscale etc users on initial "up".

Fixes #11960
Fixes #11962

Change-Id: I36f8817b69267a99271b5ee78cb7dbf0fcc0bd34
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
@bradfitz bradfitz mentioned this issue May 6, 2024
Merged
@maisem
Copy link
Collaborator

maisem commented May 6, 2024 •
edited

Still happening

=== RUN   TestTwoNodes
=== PAUSE TestTwoNodes
=== CONT  TestTwoNodes
    integration_test.go:1154: DERP httpsrv listener: 127.0.0.1:40985
    integration_test.go:390: node1 SOCKS5 addr: 127.0.0.1:43593
    integration_test.go:391: node2 SOCKS5 addr: 127.0.0.1:44891
    integration_test.go:394: n1 is listening
    integration_test.go:396: n2 is listening
    integration_test.go:397: Running /tmp/15[196](https://github.com/tailscale/tailscale/actions/runs/8976587553/job/24653726841?pr=12024#step:8:197)36403/tailscale --socket=/tmp/TestTwoNodes1375173484/001/tailscale.sock up --login-server=http://127.0.0.1:33717 --reset ...
    integration_test.go:1235: panic: [unexpected] controlclient: TryLogin called on https://controlplane.tailscale.com
    integration_test.go:1235: 
    integration_test.go:1235: goroutine 116 [running]:
    integration_test.go:1235: tailscale.com/control/controlclient.(*Direct).TryLogin(0xc00026f[200](https://github.com/tailscale/tailscale/actions/runs/8976587553/job/24653726841?pr=12024#step:8:201), {0x1ada828, 0xc0000404b0}, 0x0, 0x2)
    integration_test.go:1235: 	tailscale.com/control/controlclient/direct.go:402 +0x294
    integration_test.go:1235: tailscale.com/control/controlclient.(*Auto).authRoutine(0xc000338c80)
    integration_test.go:1235: 	tailscale.com/control/controlclient/auto.go:341 +0x7f3
    integration_test.go:1235: created by tailscale.com/control/controlclient.(*Auto).Start in goroutine 8
    integration_test.go:1235: 	tailscale.com/control/controlclient/auto.go:228 +0x8e
    integration_test.go:397: up: unexpected EOF
        , exit status 1
    integration_test.go:383: writing tailscaled logs to n1.log and n2.log
    stuntest.go:63: STUN server shutdown
--- FAIL: TestTwoNodes (4.19s)

https://github.com/tailscale/tailscale/actions/runs/8976587553/job/24653728228?pr=12024

@maisem maisem reopened this May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cmd/tailscale/cli: don't start WatchIPNBus until after up's initial Start tailscale/tailscale
2 participants
@andrew-d @maisem