Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync parallel test DBs to schema using SHA #36873

Merged
merged 1 commit into from Aug 9, 2019

Conversation

jhawthorn
Copy link
Member

Previously, every time we ran tests in parallel we would load the schema for all N databases, and then drop them at the end of testing.

Now that we have the ability to sync DBs with the schema file exactly using metadata and a SHA, we can instead only load the schemas when they change. In order for this to work we no longer drop the databases at the end of the tests. If the schema is in sync, we truncate the tables to ensure we start with an empty DB.

I believe this will make setup fast enough to close #36807

Builds on #36870, obsoletes #36826

def self.create_and_load_schema(i, env_name:)
old, ENV["VERBOSE"] = ENV["VERBOSE"], "false"

ActiveRecord::Base.configurations.configs_for(env_name: env_name).each do |db_config|
db_config.config["database"] += "-#{i}"
ActiveRecord::Tasks::DatabaseTasks.create(db_config.config)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where'd this go? Nothing in reset_to_schema sounds obviously likely to do the create if needed (and from its name, I don't think I'd expect it to anyway)

Copy link
Member Author

@jhawthorn jhawthorn Aug 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

purge does (re)create the database. I was thinking reset in the "rake db:reset" sense, which recreates the database, but reading it again I agree that might be confusing

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel strongly about avoiding 'reset' -- that parallel makes sense too.

.. but where does the create actually happen now?

Copy link
Member Author

@jhawthorn jhawthorn Aug 8, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack! You're right. This does not work when there is no DB 😳 I thought purge would create. I'll fix this and improve the naming.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went with reconstruct_from_schema, which I like (I didn't want recreate because create doesn't imply loading the schema, but reconstruct does sound like it loads the DB's structure).

I've made sure this now also creates DBs.

Previously, every time we ran tests in parallel we would load the schema
for all N DBs.

Now that we have the ability to sync DBs with the schema file exactly
using metadata and a SHA, we can instead only load the schemas when they
change.
@jhawthorn jhawthorn merged commit 599c170 into rails:master Aug 9, 2019
jhawthorn added a commit to jhawthorn/rails that referenced this pull request Aug 9, 2019
Sync parallel test DBs to schema using SHA
@dhh
Copy link
Member

dhh commented Aug 9, 2019

😍⭐️

@dhh
Copy link
Member

dhh commented Aug 9, 2019

I think this should be applied to 6-0-stable as well, so it'll be part of 6.0 final.

@kaspth
Copy link
Contributor

kaspth commented Aug 9, 2019

@dhh, already done: #36905

Thanks @jhawthorn, works great 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Setup time for parallelize worker testing is excessive for singe test case runs
5 participants