Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thread puma threadpool terminated with exception machine stack overflow in critical region (fatal) #891

Open
darrenterhune opened this issue Mar 9, 2020 · 23 comments

Comments

@darrenterhune
Copy link

darrenterhune commented Mar 9, 2020

Issue description

Our CI server is having a problem with the recent major bump to 2.0. Locally we are fine and pdf's work. If I downgrade to 1.4.0 our CI is fine.

Seems this was also an issue in the past: https://stackoverflow.com/questions/55280153/machine-stack-overflow-in-critical-region-fatal
#810

Randomized with seed 34635
Capybara starting Puma...
* Version 4.3.3 , codename: Mysterious Traveller
* Min threads: 0, max threads: 4
* Listening on tcp://127.0.0.1:34469
#<Thread:0x0000561dc0601600@puma threadpool 001@/home/circleci/project/vendor/bundle/ruby/2.6.0/gems/puma-4.3.3/lib/puma/thread_pool.rb:89 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
/home/circleci/project/vendor/bundle/ruby/2.6.0/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each': machine stack overflow in critical region (fatal)

https://github.com/puma/puma/blob/master/lib/puma/thread_pool.rb#L89
https://github.com/rack/rack/blob/master/lib/rack/urlmap.rb#L58

Expected or desired behavior

CI runs and passes

System specifications

wicked_pdf 2.0.1
wkhtmltopdf-binary-edge 0.12.4.0 because of pallymore/wkhtmltopdf-binary-edge#15 and extra build config just to get it to work so we locked it
Ubuntu 16.04
ruby 2.6.5
rails 6.0.2.1

@unixmonkey
Copy link
Collaborator

The actual code changes between 1.4.0 and 2.0.1 are pretty small, and I suspect this issue may have to do with a conflict with some other gem that hooks into render, like remotipart (do you have this in your project?).

Please read this comment and look at the diff to see what I'm talking about. I'm curious why this affects CI, but not development. Is CI running the same version of Ruby you develop with? Does it precompile assets, where maybe you don't locally?

@darrenterhune
Copy link
Author

darrenterhune commented Mar 9, 2020

Interesting, yeah I didn't see that comment. Let me dive back into this and I'll get back to you. Thanks for some direction 👍

Yes we are using the same ruby locally, on CI and on our production servers. Yes we do have remotipart, it looks like it's a dependency of rails_admin, which we are using.

@darrenterhune
Copy link
Author

Ok @unixmonkey I had some more time this avo to debug this. I forked this gem and made this change darrenterhune@ccc8615 and it's now working and passing CI. I just wanted to make sure this comment was true https://github.com/mileszs/wicked_pdf/pull/869/files#r386040878

I also found this where some people are saying having ActiveSupport.on_load ran in multiple gems causes them issues rails/rails#27013 which might be a problem (but I have no idea why) because remotipart also does this https://github.com/JangoSteve/remotipart/blob/master/lib/remotipart/rails/railtie.rb#L33-L36

We just need to 🔪 rails_admin but we are still months off from that and I suspect others will also have this issue.

@unixmonkey
Copy link
Collaborator

Yep. Thanks for confirming that. I'm working on a replacement system that avoids any kind of conflict, but I suspect that won't be released for a while either. You might be better off using your fork or monkey-patching the initializer as a no-op, and including/prepending in the controllers you need it in until then.

@Jeremie1707
Copy link

hey, could you elaborate on the way to quick fix this issue : "monkey-patching the initializer as a no-op, and including/prepending in the controllers you need it in until then", while waiting for the next release. i just added rails_admin and I have the same issue as @darrenterhune .

@unixmonkey
Copy link
Collaborator

@Jeremie1707 I tried to quickly setup an app to reproduce this, and was not able to. I looked through the commits of rails_admin and see they recently made some updates to improve compatibility with the zeitwerk loader, so maybe it's better now? You could try upgrading rails_admin and remotipart with:

bundle update rails_admin remotipart

And try again.

Another thing that might help and be minimally invasive is to defer requiring wicked_pdf until railties are booted, and your app initializers run. You can do this by editing Gemfile like this:

gem 'wicked_pdf', require: false

Then requiring wicked_pdf before it's needed in the initializer:

require 'wicked_pdf'
WickedPdf.config = {
  exe_path: '/usr/local/bin/wkhtmltopdf'
}

Then only in the controllers it's needed, add the PDF helpers:

class MyController < ApplicationController
  prepend WickedPdf::PdfHelper
  include WickedPdf::WickedPdfHelper::Assets

  def show
  end
end

Or, you can use @darrenterhune 's fork by just changing out your Gemfile

gem 'wicked_pdf', git: 'https://github.com/darrenterhune/wicked_pdf'

Please let me know what you end up doing and what worked or not.

@darrenterhune
Copy link
Author

@Jeremie1707 I'd advise against my branch just because I'm not supporting it if there are issues etc. That said it did work for us. However we simply downgraded wicked_pdf for now and locked it so Dependabot stopped trying to upgrade it. The comments about zeitwerk are interesting and I'm going to give this a test today to see what's up because we just in the last couple days had some issues with this new loader stuff. I'll report back after.

@darrenterhune
Copy link
Author

Just bumped this gem again. Deferring works but without it, it doesn't work on CI. I have to either use the 1.4 version, defer or use my branch. We've since switched to just deferring since we currently only use this in 1 controller and 1 action in that controller. Adding to application_controller wouldn't be a problem either so probably the better route to go.

@andrebras
Copy link

andrebras commented Jun 30, 2020

@darrenterhune, @unixmonkey I had the same issue adding the view_component gem (has a monkey patch on the render) and @Okatawa's patch (#919) worked for me.

@darrenterhune can you give it a try? Maybe it's the same issue.

@dingzj
Copy link

dingzj commented Jul 22, 2020

I got "fatal - exception reentered" error and stuck on loading when showing my rails home page, even the controller/view don't call to generate/display PDF.

Basically, whenever I installed the wicked_pdf gem v2.x above version (just adding to Gemfile, nothing else), the error showed; but, if installed wicked_pdf v 1.4.0 , the error goes away.

My setup:
Ruby 2.6.5
Rails 5.2.4.2
wkhtmltopdf-binary: 0.12.6.2
macOS: Mojave
I do use: rails_admin (if it matters or not)

After applying @unixmonkey 's comment above, by using the following forked gem, my error goes away.

gem 'wicked_pdf', git: 'https://github.com/darrenterhune/wicked_pdf'

Hope this helps to someone.

@rkiller
Copy link

rkiller commented Oct 9, 2020

I see the same error:

fatal - machine stack overflow in critical region

Setup:
Ruby 2.6.6
Rails 6.0.3.3
wkhtmltopdf-binary 0.12.6.5
macOS: Catalina

I've use version 1.4 of wicked_pdf and no issues at all. Whenever I upgrade to latest this bug shows up and keeps me from upgrading.

@feliperaul
Copy link

Does anyone in this thread can reliably reproduce this bug?

I'm also facing it, but only randomly.

Nate Berkopec, a Puma maintainer, is looking into this on puma/puma#2552, but he also can't reproduce the "fatal - machine stack overflow in critical region" error.

Please report in that issue if you can help.

@almokhtarbr
Copy link

@feliperaul did you solve it ? i have the same problem !!!

@feliperaul
Copy link

@almokhtarbr Nope, and I can't even reproduce it, it's totally random (every once 15 days for us).

For now I have switched to PDFKit, the APIs are very similar.

@almokhtarbr
Copy link

@feliperaul the same thing really weird, thanks for suggestion

@sposin
Copy link

sposin commented Mar 2, 2021

I am able to reliably get the
fatal - machine stack overflow in critical region
error.

I encountered it when trying to add Scout APM to our app.
scout_apm 4.0.4
ruby 3.0.0
rails 6.0.3.5
wicked_pdf 2.0
wkhtmltopdf-binary (0.12.5.4)

I tried wicked_pdf >= 2.0 and they all have the problem. I downgraded to wicked_pdf 1.4 and the error does not exist.

@davegudge
Copy link
Contributor

I was experiencing :

fatal (exception reentered):


UncaughtThrowError: uncaught throw :app_exception
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:134:in `throw'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:134:in `rescue in call_app'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:131:in `call_app'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:28:in `block in call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:17:in `catch'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/web-console-4.1.0/lib/web_console/middleware.rb:17:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in `call_app'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:26:in `block in call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/tagged_logging.rb:80:in `block in tagged'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/tagged_logging.rb:28:in `tagged'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/tagged_logging.rb:80:in `tagged'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:26:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/sprockets-rails-3.2.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:82:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/webpacker-6.0.0.beta.7/lib/webpacker/dev_server_proxy.rb:25:in `perform_request'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/rack-proxy-0.7.0/lib/rack/proxy.rb:63:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
	from /Users/dave/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
rails (~> 6.0.3.7)
wicked_pdf (2.1.0)
remotipart (1.4.4)

I've added further details to #111 (comment)

@mathieujobin
Copy link
Contributor

After encountering the #810 issue....
I upgraded my Rails app from 4.2 to 5.2 while locking down wicked_pdf to 1.1.0
now I am having issues upgrading Rails to 6.1, and I cannot find a working version of wicked_pdf

/home/mathieu/.rvm/rubies/ruby-2.6.8/lib/ruby/gems/2.6.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each': machine stack overflow in critical region (fatal)

@davegudge
Copy link
Contributor

Have you tried using the master branch of wicked_pdf rather than the latest version?

@mathieujobin
Copy link
Contributor

Issue appears resolved with master branch 5cee578

At least CI is happy, I have not test deployed but should be alright.

@mathieujobin
Copy link
Contributor

@davegudge any plan on making a new release ?

@davegudge
Copy link
Contributor

Hey @mathieujobin,

I do not have permission to create a new release, however, @unixmonkey may be able to indicate when a new version is planned for release?

@davegudge
Copy link
Contributor

davegudge commented Jun 24, 2022

This is no longer an issue since v2.6.0 was released back in May.

I've just removed this from my Gemfile:

# Tracking master to resolve: https://github.com/mileszs/wicked_pdf/issues/891
gem "wicked_pdf", github: "mileszs/wicked_pdf", branch: "master"

and upgraded to version v2.6.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests