Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql: TestValidationWithProtectedTS failed #123447

Closed
cockroach-teamcity opened this issue May 2, 2024 · 1 comment · Fixed by #123750
Closed

sql: TestValidationWithProtectedTS failed #123447

cockroach-teamcity opened this issue May 2, 2024 · 1 comment · Fixed by #123750
Assignees
@fqazi
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
SQL Foundations
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented May 2, 2024 •
edited by cockroach-jira-scripts

sql.TestValidationWithProtectedTS failed on master @ 80c77c6651754d31dfb2c852de419d032e0d412b:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestValidationWithProtectedTS (14m56s)

Stack:

goroutine 807742 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error 

=== RUN   TestValidationWithProtectedTS
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestValidationWithProtectedTS3301513052
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)

Parameters:

  • attempt=1
  • run=7
  • shard=4
Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-38354
@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels May 2, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone May 2, 2024
@fqazi fqazi self-assigned this May 2, 2024
@fqazi
Copy link
Collaborator

fqazi commented May 2, 2024

It looks like one of the go routines in the test hit an error and failed earlier, let me see if I can figure out what we hit. But at the very least we should enhance the test so it doesn't hang if that happens

@rafiss rafiss added P-2 Issues/test failures with a fix SLA of 3 months and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels May 3, 2024
fqazi added a commit to fqazi/cockroach that referenced this issue May 7, 2024
@fqazi
sql: fix TestValidationWithProtectedTS flake
ff99247 
Previously, the test TestValidationWithProtectedTS would hang if a Go
routine would hit an error, since a channel would not be posted on a
error. It would also not gracefully handle TS after replica GC errors on
transactions, which could happen on overloaded machines. To address this
patch makes the test more resilient to the Go routine hitting errors so
that it doesn't time out, and adds retry logic for TS after replica GC
errors.

Fixes: cockroachdb#123447
Release note: None
@fqazi fqazi mentioned this issue May 7, 2024
Merged
craig bot pushed a commit that referenced this issue May 9, 2024
@fqazi
Merge #123750
8042adf 
123750: sql: fix TestValidationWithProtectedTS flake r=fqazi a=fqazi

Previously, the test TestValidationWithProtectedTS would hang if a Ggio routine would hit an error, since a channel would not be posted on a error. It would also not gracefully handle TS after replica GC errors on transactions, which could happen on overloaded machines. To address this patch makes the test more resilient to the Go routine hitting errors so that it doesn't time out, and adds retry logic for TS after replica GC errors.

Fixes: #123447
Release note: None

Co-authored-by: Faizan Qazi <faizan@cockroachlabs.com>
@craig craig bot closed this as completed in b8a8565 May 9, 2024
SQL Foundations automation moved this from Triage to Done May 9, 2024
blathers-crl bot pushed a commit that referenced this issue May 14, 2024
@fqazi
sql: fix TestValidationWithProtectedTS flake
53bf974 
Previously, the test TestValidationWithProtectedTS would hang if a Go
routine would hit an error, since a channel would not be posted. It
would also not gracefully handle TS after replica GC errors on
transactions, which could happen on overloaded machines. To address this
patch makes the test more resilient to the Go routine hitting errors so
that it doesn't time out, and adds retry logic for TS after replica GC
errors.

Fixes: #123447
Release note: None
@blathers-crl blathers-crl bot mentioned this issue May 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fqazi fqazi

Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
SQL Foundations
  
Done
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

sql: fix TestValidationWithProtectedTS flake fqazi/cockroach
3 participants
@rafiss @fqazi @cockroach-teamcity