6.14.0 creates infinite recursion on "net request" with rack-mini-profiler present
#510
Comments
|
We have a similar issue. I haven't found the exact cause however it took down our sign up page. I believe we had some error that threw a "Reason stack level too deep" and this caused NewRelic to throw an uncaught error. We did not have an issue with We also we unable to reproduce this on a non-production environment |
|
We have the same in development: Rails 6.0.3.4
|
|
I had the same problem, the commit that introduced the regression was: cde7d10 Setting |
|
This should be fixed by making the option |
|
We also have the same issue, we're not using rack-mini-profiler so might be another gem or depency. |
|
@ro-savage Thank you for bringing this issue to our attention. Could you share your environment, please? Also would adding the option prepend_net_instrumentation: false, work for you? |
|
Disabling prepend_net_instrumentation works for us too.
Our gemfile is quite huge so I’ll try to reduce the list of gems that could cause the issue.
|
|
@angelatan2 - We just removed New Relic completely from our system. The fact that feature version bump causes such a major issue for us means we decided New Relic wasn't worth the potentially damage. So I won't be able share any more details other than below. But basically we had an impossible to find where it occurred error, that ended up with We are just a standard RoR app, and there was no reference to new relic. We just had to revert gems until we pinpointed the new relic agent. We think it may have happened when we were sending off analytics and subscription data to third party services. As it seemed to have happened after our sign up form submission, but before the account was finished being created. |
|
@ro-savage, it is a delicate balance between maintaining the old and moving forward with the evolving changes of the Ruby language and environment. We are well aware of just how sensitive such changes can be and work hard to test and vet each release to ensure exactly this experience does not happen. We obviously have a blindspot in our test suite that failed to surface this issue. For that, I sincerely apologize. We will certainly retrospect to understand just how this issue got past our tests and work to plug similar blindspots across all the instrumentation libraries we're supporting. Meanwhile, I am looking into this and trying to understand just how our change caused this issue with the addressable gem. For background info, we are working to reduce corner cases where the Ruby agent's instrumenting approach of using method chaining leads to other gems not being usable in conjunction with the agent running as described in this blog post. Our long-term strategy calls for eliminating method-chaining as a means of patching into libraries and using prepend instead. Doing so will also help us eliminate issues with method parameters delegation that we're facing with Ruby 2.7+ as described here #502. It looks like unfortunately in this case, our attempt to get "out of the way" led to the very thing we're trying to prevent in the first place. At the moment, this looks to be strictly confined to Ruby 2.7 and Addressable 2.7.0. if anyone is seeing outside Ruby 2.7, I would love to hear from you and especially see a stack trace. Meanwhile, @PikachuEXE's suggested configuration change is the fastest way to revert to using the method chaining approach that doesn't exhibit this problem. |
|
@mwlang we're having the issue with ruby 2.6.6 here is the stacktrace: Please don't hesitate to let me know if you need more informations. |
|
It's definitely an issue with 'net/http' and/or adressable. We have an initializer in our app that get a certificate from a webserver using Net::HTTP.start and a get request. Let me know if you need more details. |
|
@jeremybdk Thank you for sharing the additional details. We are trying to reproduce this issue with our test app but unfortunately, we are not having much success. We are wondering if there is another gem dependency that could cause this issue. Are you willing/able to share your gemfile with us? |
|
@angelatan2 better than that! I spent some time, creating a new rails app, adding all my gems, reproducing the crash, and removing most of my gems to see which ones was making the app crash. The weird thing is that it's not one gem that trigger the crash, it's only when I have the gems left in this Gemfile. You can find a link to the app here: https://github.com/jeremybdk/crash_newrelic I hope that will help you :) |
|
@jeremybdk thank you for that reproduction. I am able to at least see the crash locally with the app you put together. |
|
UPDATE: It's a two-way contention with the Ruby agent and Airbrake both monkey patching
This is a potential fix and has not yet passed all tests in continuous integration. Please do check the status of PR #511 before proceeding to install or test this. gem 'newrelic_rpm',
:git => 'https://github.com/newrelic/newrelic-ruby-agent.git',
:ref => '77e553b11301dc1e3d3376fb0a545fc2b29196ce' |
|
I think for my case it's It seems like they already have solution for Will post back after testing this method. |
|
I confirm if I set I can run network requests without FastImage.type(
"https://duckduckgo.com/assets/logo_homepage.alt.v108.svg",
timeout: 5,
)
HTTPClient.new.get("https://duckduckgo.com/assets/logo_homepage.alt.v108.svg").code |
|
I did some more testing today regarding Airbrake, the issue disappear when upgrading to Airbrake v10.0.1 |
|
@jeremybdk and @PikachuEXE Thank you both for testing further and confirming the fix works. As for Airbrake 10.0.1, it looks like they too are moving away from method chaining to a prepend strategy. All further work to move away from method chaining to prepend will take place in our next major release (i.e. 7.0.0). We've been getting a lot of requests to move to prepend strategy for instrumenting various gems, but the risk/reward is too high to continue in the 6.x series of the agent. |
|
@mwlang thanks for your help explaining and resolving the issue. We had the same problem using Skylight, because they are still using |
^^ Provide a general summary of the issue in the title above. ^^
Description
We use
fastimageto check the type of image behind the image URLs input by user before saving them.Example code would be something like:
Expected Behavior
It should work without infinite recursion (
SystemStackError: stack level too deep)Troubleshooting or NR Diag results
Error Backtrace
Steps to Reproduce
Your Environment
Include as many relevant details about your environment as possible including the running version of the Ruby agent, the Ruby version being used and any other relevant environment information
2.7.2, Rails6.0.3.4rack-mini-profiler (2.2.0)newrelic_rpm (6.14.0)Additional context
newrelic_rpm (6.13.1)has no such issueThis can only be reproduced on production like environment
(This issue first affects production env & later reproduced on staging)
The text was updated successfully, but these errors were encountered: