Truncate tables before test runs is hanging on Linux

[dhh]

When running a multi-process test run on Linux, I'm seeing the system hang as part of the truncate_tables call in database_tasks.rb. It seems that the problem is around abandon_results!, which waits for a timeout to proceed. Here's the backtrace when I halt execution:

activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:63:in `abandon_results!': Interrupt
activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:63:in `block (2 levels) in execute_batch'
activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:997:in `block in with_raw_connection'
activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:969:in `with_raw_connection'
activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:61:in `block in execute_batch'
activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:60:in `each'
activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:60:in `execute_batch'
activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:230:in `block (2 levels) in truncate_tables'
activerecord/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:219:in `disable_referential_integrity'
activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:228:in `block in truncate_tables'
activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:94:in `block in with_multi_statements'
activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:997:in `block in with_raw_connection'
activesupport/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:969:in `with_raw_connection'
activerecord/lib/active_record/connection_adapters/mysql2/database_statements.rb:91:in `with_multi_statements'
activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:227:in `truncate_tables'
activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:26:in `truncate_tables'
activerecord/lib/active_record/tasks/database_tasks.rb:226:in `block in truncate_tables'
activerecord/lib/active_record/connection_adapters/abstract/connection_pool.rb:399:in `with_connection'
activerecord/lib/active_record/tasks/database_tasks.rb:514:in `block in with_temporary_connection'
activerecord/lib/active_record/tasks/database_tasks.rb:546:in `with_temporary_pool'
activerecord/lib/active_record/tasks/database_tasks.rb:513:in `with_temporary_connection'
activerecord/lib/active_record/tasks/database_tasks.rb:225:in `truncate_tables'

First instinct was to just make truncate_tables optionally via a toggle. Truncation is only needed if you have tests that don't run inside the default transaction, which is rare. But while that might still make sense for an additional speedup (truncating tables still takes time, even if it's executing normally), we surely should fix the underlying issue.

The truncate call was introduced by @jhawthorn in d8b8171.

[justinko]

It seems that the problem is around abandon_results!, which waits for a timeout to proceed.

I don't think abandon_results! utilizes any sort of timeout: https://github.com/brianmario/mysql2/blob/58c8190dd423779eae230e2ff803e2d0c465f67b/ext/mysql2/client.c#L821

It just tells mysql you don't care about the results of the query ("truncation" in this case) so it can move on to other things.

[justinko]

When it's hanging, does mysql SHOW PROCESSLIST; or SHOW OPEN TABLES WHERE In_use > 0; reveal anything?

On another note, it would be interesting to benchmark this to see if there's any change to your ~10 second test run increase:

SET FOREIGN_KEY_CHECKS=0;
TRUNCATE TABLE your_table;
SET FOREIGN_KEY_CHECKS=1;

Apparently disabling foreign key checks can greatly speedup truncation.