The number of Crashes and ANRs has increased

[ilakhin]

Environment

How do you use Sentry?

• Self-hosted 22.11.0, 23.2.0

Which version of the SDK?

• 1.2.0, 1.5.0

How did you install the package? (Git-URL, Assetstore)

• Git-URL

Which version of Unity?

• 2021.3.25f1, 2020.3.46f1

Is this happening in Unity (editor) or on a player like Android, iOS, Windows?

• Android

Actual Result

Since September 11, the number of Crashes and ANRs has increased significantly.

Any logs or screenshots

┆Issue is synchronized with this Jira Improvement by Unito

[markushi]

Hi @ilakhin, thanks for reaching out!
Could you share some more details about the crashes you're seeing? E.g. a full stack trace would be very helpful. In case they contain sensitive information feel free to send them to me via E-Mail: markus.hintersteiner@sentry.io

[ilakhin]

Hi @markushi!
I sent more information by email.

[romtsn]

hi @ilakhin, could you please provide more details what exactly happened on Sept. 11 (or whenever the crash/anr-rate had started growing)? Have you updated the SDK version from 1.2.0 to 1.5.0 on that day? Or perhaps updated the self-hosted sentry version?

Also, do you use our Gradle plugin in your project by any chance?

[ilakhin]

Hi @romtsn

• We released a new version of the game on September 4;
• Since September 11, the number of Crashes and ANRs has increased.
• We use Sentry 1.2.0 (1.5.0 - on the other project). The Sentry package has not been updated for the last few months;
• The last self-hosted Sentry update was last year;
• We don't use the Sentry Android Gradle Plugin
• Issue is only on Android 12, 12L, 13, 14.

[romtsn]

Hi @romtsn

• We released a new version of the game on September 4;
• Since September 11, the number of Crashes and ANRs has increased.
• We use Sentry 1.2.0 (1.5.0 - on the other project). The Sentry package has not been updated for the last few months;
• The last self-hosted Sentry update was last year;
• We don't use the Sentry Android Gradle Plugin
• Issue is only on Android 12, 12L, 13, 14.

Thanks for the detailed info! And just to confirm, both versions 1.2.0 and 1.5.0 cause the growth of crashes and ANRs, but you haven't updated those SDK versions for a while, only the app code? Have you changed something in the SDK configuration perhaps?

[ilakhin]

@romtsn

• Yes, 1.2.0 / 1.5.0 cause the growth of crashes and ANRs;
• Only app code has been updated;
• SDK configuration has not changed.

[markushi]

Adding https://issuetracker.google.com/issues/300840851 for reference here, seems like this is related to some Google Play System Service update as well.

[markushi]

Hi @ilakhin, thanks for providing us with all the details, much appreciated! We've been able to reproduce the crash on a phone which has the Google Play System Update from August installed. I'll keep you posted on the progress!

One more question: Within your application, are you collecting stacktraces for non-fatal events? E.g. by using sentry_value_new_stacktrace() (link)

[markushi]

Quick update: We're still looking into this issue. If anyone is able to re-produce this issue within their local dev environment please let me know!

[garethknowles]

I believe this may be the same issue was also tracked/reported over on react-native here: facebook/react-native#39505 as well as on the above google tracker.

As detailed on the google issue tracker - we've been able to reproduce locally in testing (specifically it seems to be more easily reproduced with Android 14 beta devices - likely linked to the google play system services / update versions). We've been able to link the issue directly to the react native sentry sdk in our app - with removal resolving the crashes.

[markushi]

@garethknowles thanks for linking the react-native issue, yes - this looks like being the same.
So far we only have been able to reproduce the issue on a device with Android 14 Beta + August Google Play System updates. And only sporadically with a customers release build, but unfortunately not within our environment.

Any chance you could try a build with sentry debugging enabled? Although we think the culprit lies within the Play System update, we'd love to find a workaround within our SDK in order to fix this.

Sentry.init({
  dsn: "...",
  // ... your existing config
  
  debug: true
});

[chrisbobbe]

Hi! 👋 Copying a bit that might be useful from our investigation over at zulip/zulip-mobile (a React Native app that uses Sentry):

https://chat.zulip.org/#narrow/stream/48-mobile/topic/Android.20crashes/near/1642674

Meanwhile I just went looking to see if there's a useful changelog for those Google Play system updates. It looks like there isn't exactly, but the code is open source - which I hadn't expected - and so it is possible to identify what changed though with some effort:
https://www.esper.io/blog/building-a-google-play-system-update-changelog

(We haven't yet gone and done that effort, or I would post the result here.)

[garethknowles]

@markushi I've enabled the debug setting, here is some logcat output around the crash triggered on a Android 14 beta device I'm testing on.

--------- beginning of crash
2023-09-20 10:00:31.624 22170-22422 sentry-native           <redacted>                 I  entering signal handler
2023-09-20 10:00:31.624 22170-22170 libc                    <redacted>                 A  stack corruption detected (-fstack-protector)
2023-09-20 10:00:31.624 22170-22170 sentry-native           <redacted>                 I  entering signal handler
2023-09-20 10:00:31.624 22170-22422 sentry-native           <redacted>                 I  entering signal handler
2023-09-20 10:00:31.631 22170-22422 sentry-native           <redacted>                 D  captured backtrace with 9 frames
2023-09-20 10:00:31.632 22170-22422 sentry-native           <redacted>                 D  merging scope into event
2023-09-20 10:00:31.632 22170-22422 sentry-native           <redacted>                 D  trying to read modules from /proc/self/maps
2023-09-20 10:00:31.721 22170-22422 sentry-native           <redacted>                 D  read 335 modules from /proc/self/maps
2023-09-20 10:00:31.723 22170-22422 sentry-native           <redacted>                 D  adding attachments to envelope
2023-09-20 10:00:31.723 22170-22422 sentry-native           <redacted>                 D  sending envelope
2023-09-20 10:00:31.723 22170-22422 sentry-native           <redacted>                 D  serializing envelope into buffer
2023-09-20 10:00:31.723 22170-22422 sentry-native           <redacted>                 I  crash has been captured
2023-09-20 10:00:31.723 22170-22422 libc                    <redacted>                 A  Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x100000027 in tid 22422 (mqt_native_modu), pid 22170 (<redacted>)
2023-09-20 10:00:31.751 22170-22170 sentry-native           <redacted>                 D  captured backtrace with 4 frames
2023-09-20 10:00:31.751 22170-22170 sentry-native           <redacted>                 D  merging scope into event
2023-09-20 10:00:31.751 22170-22170 sentry-native           <redacted>                 D  adding attachments to envelope
2023-09-20 10:00:31.751 22170-22170 sentry-native           <redacted>                 D  sending envelope
2023-09-20 10:00:31.751 22170-22170 sentry-native           <redacted>                 D  serializing envelope into buffer
2023-09-20 10:00:31.752 22170-22170 sentry-native           <redacted>                 I  crash has been captured
2023-09-20 10:00:31.752 22170-22503 Sentry                  <redacted>                 D  Envelope sent successfully.
2023-09-20 10:00:31.754 22170-22503 Sentry                  <redacted>                 D  Discarding envelope from cache: /data/user/0/<redacted>/cache/sentry/1634f398-fc4a-4f13-b9d9-6e44bb90fe0f.envelope
2023-09-20 10:00:31.755 22170-22503 Sentry                  <redacted>                 D  Envelope flushed
2023-09-20 10:00:31.894 22170-22422 libc                    <redacted>                 A  stack corruption detected (-fstack-protector)

We can give you further logs & details via the support ticket we've opened in your support system?
Ticket #103215

[chrisbobbe]

@garethknowles, can you find out the version of Sentry Native that's printing those "sentry-native" lines? I think it'll be a dependency of Sentry Android which in turn is a dependency of Sentry React Native (you mentioned your app is built with RN). I wonder if the crashes reproduce with the current latest Sentry Native.

[markushi]

@garethknowles this definitely helps, thank you! Yeah for sure, you can also send it via e-mail directly to markus.hintersteiner@sentry.io

[garethknowles]

@garethknowles, can you find out the version of Sentry Native that's printing those "sentry-native" lines? I think it'll be a dependency of Sentry Android which in turn is a dependency of Sentry React Native (you mentioned your app is built with RN). I wonder if the crashes reproduce with the current latest Sentry Native.

There are more details including the affected sentry versions in the perviously linked Google issue tracker and react-native github issue pages but to summarise - we're using @sentry/react-native 3.4.3 but there are also reports of it on 5.1.1 - I believe those use io.sentry:sentry-android:5.7.0 and io.sentry:sentry-android:6.15.0

[romtsn]

If anyone would like to give it a try, the AOSP changelog is published here https://mikeng.github.io/aosp.changelog.to/, but I couldn't find anything in the art/ subproject which is remotely relevant. Seems like the newer tags were not even published, as the latest change I've seen in art is dated Jul 25...

[chreck]

We released our react native app on 3th October and having now over 10k active users. We do not see any crashes like before. On Google Play Console we also have no crashes.

We use the setup and version:

"@sentry/react-native": "^5.10.0",

In AndroidManifest.xml we set:

<meta-data android:name="io.sentry.ndk.enable" android:value="false" />

In the Javascript part we setup Sentry like this:

Sentry.init({
  dsn: "https://123456@sentry.io/100",
  enableNdk: false,
})

[TheRogue76]

@andersk Yes, that's right - thanks for clarifying! If you have access to your build.gradle file you can exclude our ndk dependency - that even prevents sentry-native from being packaged and thus it won't be loaded.

configurations.configureEach {
  exclude(group = "io.sentry", module = "sentry-android-ndk")
}

See our docs for more details. Now I'm wondering: do you still see a number of elevated crashes when libsentry.so is loaded but init() is not called?

@chrisbobbe Yes, we already looked into this, unfortunately the tags for the Play System August weren't published up until recently - let me double check on this.

I have had this change in production for about two days now. The crashes have stabilised, and i am not seeing anything new in Google play store console. Even though most of the issues had mentions of other libraries, i feel like It was a cascading issue and they might have still originated from Sentry's NDK. Before excluding the NDK, i was running on the latest version and had enableNDK prop set to false, but the issues persisted.

[andrey-vishnitskiy-hypemasters]

@markushi any updates with the fix?

[markushi]

Hey everyone, here’s a quick update from our side:
We’re still looking into this issue, following every lead we receive. So far we’ve been able to identify a scenario where our signal handler appears to be receiving a signal it’s not meant to receive. It’s still unclear why this scenario is happening at all and in we’re in close contact with some Google engineers to further investigate the root cause.

[Sevyls]

Any status update to this issue? We consider to remove Sentry from our Android app as crashes increased again.

[kahest]

@Sevyls did you try disabling NDK integration or excluding the NDK module? If you did and it didn't prevent these crashes, we'd love to hear from you.

[kahest]

Quick update - we have a promising lead into a possible solution that we are currently testing and validating with Google.

[kahest]

We just released sentry-java 6.33.2-beta.1 with a fix for this issue. Our tests show very promising results, but we released this as a beta to get some more data. Please try out the beta if you can and let us know the results. 🙏

[kahest]

sentry-react-native 5.13.1-beta.1 is now also available for testing.

[kahest]

sentry-unity 1.7.1-beta.1 is now also available for testing.

[kahest]

We just released sentry-java 6.34.0, which is the first stable release that includes the fix for this issue. I'm closing this now.

We will release stable versions for sentry-unity and sentry-react-native with the fix this week.

[kahest]

sentry-react-native 5.14.0 has landed

[kahest]

sentry-unity 1.7.1 is now also available