Attempt to fix random CI failures #1495
Conversation
![hound[bot]](https://avatars.githubusercontent.com/in/3598?s=60&v=4){kind=small}
| config.before(:each, type: :feature) do | ||
| # :rack_test driver's Rack app under test shares database connection | ||
| # with the specs, so continue to use transaction strategy for speed. | ||
| driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test |
There was a problem hiding this comment.
Metrics/LineLength: Line is too long. [82/80]
| the spec. The app's database connection would not be able to access | ||
| uncommitted transaction data setup over the spec's database connection. | ||
| MSG | ||
| end |
There was a problem hiding this comment.
Layout/EmptyLineAfterGuardClause: Add empty line after guard clause.
|
I don't have permissions to re-run the build, so I'll be rebasing this as new commits are added to master, and logging here how the build went each time. (I could probably work around this, but I'm lazy now). First push: 🍏 pass 🍏 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
d170c73
to
6848bc3
Compare
|
Second push: 🍏 pass 🍏 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
6848bc3
to
fca6f5d
Compare
|
Third push: 🍏 pass 🍏 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
fca6f5d
to
70bbebb
Compare
|
Fourth push: 🍏 pass 🍏 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
70bbebb
to
8b3cfc0
Compare
|
Fifth push: 🍏 pass 🍏 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
8b3cfc0
to
04ce1d1
Compare
|
Sixth push: 🍏 pass 🍏 |
|
I'm not merging this yet because I'm not seeing other builds fail. |
|
Actually no, here's a failure: https://circleci.com/gh/thoughtbot/administrate/5608 It's the Ruby 2.5 build for a50bbae |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
04ce1d1
to
51946e9
Compare
|
Seventh push: 🍏 pass 🍏 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
51946e9
to
cc4e534
Compare
|
Eight push: 🍏 pass 🍏 (This is getting a bit silly). |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
cc4e534
to
8b26229
Compare
|
Failure elsewhere: https://circleci.com/gh/thoughtbot/administrate/5637 |
|
Completely unexpected failure here! https://circleci.com/gh/thoughtbot/administrate/5639 However it's not to do with Postgres. Appears to be a different sort of fluke. I'll ignore it. |
|
Another Postgres failure elsewhere! https://circleci.com/gh/thoughtbot/administrate/5636 |
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
8b26229
to
2bcd73f
Compare
|
Another 🍏 pass 🍏 |
We get frequent random CI failures with errors on feature specs, like
the following:
```
Failures:
1) Search admin searches with an unknown filter
Failure/Error: DatabaseCleaner.clean
ActiveRecord::StatementInvalid:
PG::TRDeadlockDetected: ERROR: deadlock detected
DETAIL: Process 12270 waits for AccessExclusiveLock on relation 16412 of database 16386; blocked by process 12192.
Process 12192 waits for AccessShareLock on relation 16400 of database 16386; blocked by process 12270.
HINT: See server log for query details.
: TRUNCATE TABLE "public"."blog_posts", "public"."series", "public"."countries", "public"."log_entries", "public"."payments", "public"."product_meta_tags", "public"."customers", "public"."products", "public"."orders", "public"."line_items" RESTART IDENTITY CASCADE;
# ./spec/support/database_cleaner.rb:19:in `block (2 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# PG::TRDeadlockDetected:
# ERROR: deadlock detected
# DETAIL: Process 12270 waits for AccessExclusiveLock on relation 16412 of database 16386; blocked by process 12192.
# Process 12192 waits for AccessShareLock on relation 16400 of database 16386; blocked by process 12270.
# HINT: See server log for query details.
# ./spec/support/database_cleaner.rb:19:in `block (2 levels) in <top (required)>'
Finished in 29.41 seconds (files took 3.65 seconds to load)
418 examples, 1 failure
```
This may or may not be related to `DatabaseCleaner`. As an attempt to
fix it, I'm trying this configuration, copied straight from their
documentation at https://github.com/DatabaseCleaner/database_cleaner/blob/v1.7.0/README.markdown
pablobm
force-pushed
the
fix-random-ci-failures
branch
from
2bcd73f
to
7855f74
Compare
|
Another Postgres-related failure elsewhere: https://circleci.com/gh/thoughtbot/administrate/5647 |
|
What do you think about this one now? Is it solving it enough that it might be worth merging and seeing when the problem comes up again? |
|
Some stats:
So I guess without this change there's a Postgres deadlock every 12.8 builds, while there are none with this change. Is that statistically significant enough? No idea, but at least it doesn't seem to be making it worse. Gah. Merging. |
We get frequent random CI failures with errors on feature specs, like
the following:
This may or may not be related to
DatabaseCleaner. As an attempt tofix it, I'm trying this configuration, copied straight from their
documentation at https://github.com/DatabaseCleaner/database_cleaner/blob/v1.7.0/README.markdown