roachtest: ycsb/F/nodes=3/cpu=32 failed

[cockroach-teamcity]

roachtest.ycsb/F/nodes=3/cpu=32 failed with artifacts on release-21.2 @ 99a4816fc272228a63df20dae3cc41d235e705f3:

		  |   113.0s        0         6523.2         4500.1      1.0     26.2    184.5    285.2 read
		  |   113.0s        0         6384.2         4499.8     10.0     22.0    109.1    352.3 readModifyWrite
		  |   114.0s        0         6008.2         4513.3      1.0     18.9    209.7    302.0 read
		  |   114.0s        0         6136.2         4514.2      9.4     19.9    125.8    419.4 readModifyWrite
		  |   115.0s        0         6031.3         4526.6      1.0     22.0    192.9    268.4 read
		  |   115.0s        0         6008.3         4527.2      9.4     22.0    125.8    671.1 readModifyWrite
		  |   116.0s        0         6414.7         4542.8      1.0     21.0    176.2    285.2 read
		  |   116.0s        0         6269.7         4542.2      9.4     21.0    121.6    771.8 readModifyWrite
		  |   117.0s        0         6044.7         4555.7      1.0     18.9    201.3    302.0 read
		  |   117.0s        0         6015.7         4554.8      9.4     21.0    142.6    838.9 readModifyWrite
		  |   118.0s        0         6215.2         4569.7      1.0     24.1    184.5    260.0 read
		  |   118.0s        0         6105.2         4567.9      9.4     22.0    151.0    838.9 readModifyWrite
		  |   119.0s        0         6350.0         4584.7      1.0     23.1    201.3    302.0 read
		  |   119.0s        0         6230.1         4581.9      9.4     21.0    142.6    671.1 readModifyWrite
		  |   120.0s        0         6071.1         4597.1      1.0     14.7    234.9    369.1 read
		  |   120.0s        0         6283.9         4596.1      9.4     21.0    176.2    704.6 readModifyWrite
		Wraps: (4) exit status 20
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *exec.ExitError

	monitor.go:128,ycsb.go:73,ycsb.go:90,test_runner.go:777: monitor failure: monitor task failed: t.Fatal() was called
		(1) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).WaitE
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:116
		  | main.(*monitorImpl).Wait
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:124
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerYCSB.func1
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/ycsb.go:73
		  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerYCSB.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/ycsb.go:90
		  | main.(*testRunner).runTest.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
		Wraps: (2) monitor failure
		Wraps: (3) attached stack trace
		  -- stack trace:
		  | main.(*monitorImpl).wait.func2
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:172
		Wraps: (4) monitor task failed
		Wraps: (5) attached stack trace
		  -- stack trace:
		  | main.init
		  | 	/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/monitor.go:81
		  | runtime.doInit
		  | 	/usr/local/go/src/runtime/proc.go:6309
		  | runtime.main
		  | 	/usr/local/go/src/runtime/proc.go:208
		  | runtime.goexit
		  | 	/usr/local/go/src/runtime/asm_amd64.s:1371
		Wraps: (6) t.Fatal() was called
		Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError

Reproduce

See: roachtest README

Same failure on other branches

• roachtest: ycsb/F/nodes=3/cpu=32 failed [bad connection] [similar to 69523] #69545 roachtest: ycsb/F/nodes=3/cpu=32 failed [bad connection] [similar to 69523] [C-test-failure O-roachtest O-robot branch-master]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

[tbg]

	  | Error: driver: bad connection

[tbg]

Moving to SQL-Queries for the same reasons as the master sibling, see #69545 (comment)

[rytaft]

This does not seem like a blocker, but I'm also not sure this is a SQL Queries issue. @rafiss you seem to have the most familiarity with this issue -- can you comment on this?

[rafiss]

Sure - we have seen the same type of failure in #68574. That was (supposed to be) fixed by #68665 which is because of a lib/pq issue. I don't yet know how it still could be happening, but I'll look more closely again.

[rytaft]

@rafiss did you get a chance to take a look at this at all?

[rafiss]

I did look for a few hours and couldn't repro or find anything helpful yet.