Precompilation & Load Path Cache

[metaskills]

I am working on optimizing some cold starts for Rails in AWS Lambda (https://lamby.custominktech.com) and wonder if you could describe why the precompilation command does not also add the load-path-cache. My goal is to have this in place for the Docker image deployed to Lambda so Rails boots faster.

I've come up with a workaround to do just that. But hoping y'all can add some context as to how the load path cache is supposed to work.

[casperisfine]

I think I didn't add it because that cache is dependent on the $LOAD_PATH, and that it's hard to predict an app $LOAD_PATH without booting it.

But if you have some ideas or rough patches, feel free to submit them.

[metaskills]

Another thing to note is that Lambda is a read-only file system. My method to build the load-path-cache is to simply add something like this to the Dockerfile.

RUN bundle exec bootsnap precompile --gemfile app/ lib/ && \
    LAMBY_BUILD=1 bundle exec ruby app.rb

Where app.rb here does the typical requires for Rails boot, application, environment and defines the Lambda handler. This does generate a good looking load-path-cache within the Lambda Container. Still testing benchmarks.

However, one big hurdle I had to overcome is that this file is opened and re-written on boot. I've done some general testing and it appears to not change. Can you share why this file is mutated and ends up @dirty during commit_transaction? I have to add code like this to the app.rb to work around it while I test things further.

unless ENV['LAMBY_BUILD']
  require 'bootsnap'
  Bootsnap::LoadPathCache::Store.class_eval do
    def commit_transaction ; end
  end
end

[casperisfine]

to simply add something like this to the Dockerfile.

That's pretty much what we do too. Mostly to run rake assets:precompile but it generates the load path cache.

Can you share why this file is mutated and ends up @dirty during commit_transaction?

Hum, that sounds weird. I'd have to dig that up. Not sure I'll have the time today though.

Does your app ever mutate $LOAD_PATH ?

[metaskills]

Thanks. No, the application does not mutate $LOAD_PATH

[casperisfine]

Maybe you happen to have a backtrace relating to that?

[metaskills]

Yup!

Init error when loading handler app.handler
{
"errorMessage": "Read-only file system @ rb_sysopen - /var/task/tmp/cache/bootsnap/load-path-cache.8.4481.tmp",
"errorType": "Init<Errno::EROFS>",
"stackTrace": [
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:88:in `initialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:88:in `open'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:88:in `dump_data'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:58:in `commit_transaction'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:49:in `block in transaction'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:45:in `synchronize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/store.rb:45:in `transaction'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:139:in `push_paths_locked'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:122:in `block in reinitialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:116:in `synchronize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:116:in `reinitialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache/cache.rb:16:in `initialize'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache.rb:44:in `new'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/load_path_cache.rb:44:in `setup'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap.rb:62:in `setup'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap.rb:108:in `default_setup'",
"/var/task/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.7.4/lib/bootsnap/setup.rb:4:in `<top (required)>'",
"/var/task/app.rb:9:in `require'",
"/var/task/app.rb:9:in `<top (required)>'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'",
"/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'"
]
}

[casperisfine]

Hum, I'm afraid I'm not able to reproduce. On my test app the cache is consistent and isn't marked as dirty on subsequent boots.

If you are able to reproduce this in isolation I'm interested in digging this up, but at this stage I'm afraid this issue isn't actionable for me.

[metaskills]

Would a different $LOAD_PATH on the Docker container vs the Lambda runtime environment cause this? For example, I looked at both and saw these differences.

>> d - l
=> ["/var/lang/lib/ruby/gems/2.7.0/gems/bundler-2.2.16/lib"]
>> l - d
=> ["/var/task", "/var/runtime/lib", "/opt/ruby/lib", "/var/runtime/gems/bundler-2.2.16/lib"]

If not, where else could I look to reproduce this?

[casperisfine]

Would a different $LOAD_PATH on the Docker container vs the Lambda runtime environment cause this?

Yes. The content of $LOAD_PATH is hashed to generate the key. If the $LOAD_PATH isn't the same in both environments, the cache won't work.

[casperisfine]

won't work.

I meant a new entry will be written.

[metaskills]

Thanks! Would you be open to a configuration change to avoid writing the new entry?

[casperisfine]

I'd be open to a patch that would ignore the error. That would be inline with recent changes in bootsnap where we make the cache best effort.

[metaskills]

Thanks. Done deal. #358

[casperisfine]

I think this was fixed.