DataStore Crash in OutgoingMutationQueue

[fzy-github]

Describe the bug

This is regression/the same crash as in #3325

Error reported:

OS Version: iOS 16.6.1 (20G81)
Report Version: 104

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: SEGV_ACCERR at 0x000000000000001a
Crashed Thread: 10

Application Specific Information:
Signal 11, Code 2 >
Attempted to dereference garbage pointer at 0x1a.

Thread 10 Crashed:
0   Foundation                      0x31d969644         __NSOQSchedule
1   Foundation                      0x31d8f89a4         -[NSOperationQueue setSuspended:]
2   SampleApp                       0x202eadc50         OutgoingMutationQueue.doStart (OutgoingMutationQueue.swift:143)
3   SampleApp                       0x202eb0f78         OutgoingMutationQueue.queryMutationEventsFromStorage (OutgoingMutationQueue.swift:349)
4   SampleApp                       0x202e74b90         SQLiteStorageEngineAdapter.query<T> (StorageEngineAdapter+SQLite.swift:339)
5   SampleApp                       0x202e755e4         SQLiteStorageEngineAdapter.query<T> (StorageEngineAdapter+SQLite.swift:308)
6   SampleApp                       0x202e77280         SQLiteStorageEngineAdapter
7   SampleApp                       0x202eadf90         OutgoingMutationQueue.queryMutationEventsFromStorage (OutgoingMutationQueue.swift:336)
8   SampleApp                       0x202ead7b0         OutgoingMutationQueue.doStart (OutgoingMutationQueue.swift:140)
9   SampleApp                       0x202ead0c0         OutgoingMutationQueue.respond (OutgoingMutationQueue.swift:108)
10  SampleApp                       0x202eb3e8c         [inlined] OutgoingMutationQueue.init (OutgoingMutationQueue.swift:77)
11  SampleApp                       0x202eb3e8c         OutgoingMutationQueue.init
12  SampleApp                       0x202e2fdc4         thunk for closure
13  libdispatch.dylib               0x337cda31c         _dispatch_call_block_and_release
14  libdispatch.dylib               0x337cdbea8         _dispatch_client_callout
15  libdispatch.dylib               0x337ce3530         _dispatch_lane_serial_drain
16  libdispatch.dylib               0x337ce40a0         _dispatch_lane_invoke
17  libdispatch.dylib               0x337ceecd8         _dispatch_workloop_worker_thread
18  libsystem_pthread.dylib         0x3e8fbadd8         _pthread_wqthread

Steps To Reproduce

N/A crash happens sometimes. 

Looking at available logs crash happened after `syncQueriesStarted` event was emitted by DataStore,.

Expected behavior

no crash

Amplify Framework Version

2.25.2

Amplify Categories

DataStore

Dependency manager

Swift PM

Swift version

5.7

CLI version

12.8.2

Xcode version

15.1

Relevant log output

N/A

Is this a regression?

Yes

Regression additional context

Previusly reported and supposedly fixed: #3325

Platforms

iOS

OS Version

16, 17

Device

various devices

Specific to simulators

no

Additional context

Sync expression configuration:

let syncExpressions = [
            DataStoreSyncExpression.syncExpression(Table1.schema, where: {
                Table1.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table2.schema, where: {
                if let dateLimit = dateLimit {
                    Table2.keys.user_id.eq(userId).and(Table2.keys.time_stamp.gt(dateLimit.temporalDateTime))
                } else {
                    Table2.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(Table3.schema, where: {
                Table3.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table4.schema, where: {
                Table4.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table5.schema, where: {
                Table5.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table6.schema, where: {
                Table6.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table7.schema, where: {
                Table7.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table8.schema, where: {
                if let dateLimit = dateLimit {
                    Table8.keys.user_id.eq(userId).and(Table8.keys.data_timestamp.gt(dateLimit.temporalDateTime))
                } else {
                    Table8.keys.user_id.eq(userId)
                }
            }),
            DataStoreSyncExpression.syncExpression(Table9.schema, where: {
                Table9.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table10.schema, where: {
                Table10.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table11.schema, where: {
                Table11.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table12.schema, where: {
                Table12.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table13.schema, where: {
                Table13.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table14.schema, where: {
                Table14.keys.user_id.eq(userId)
            }),
            DataStoreSyncExpression.syncExpression(Table15.schema, where: {
                Table15.keys.data_timestamp.eq(nil)
            }),
            DataStoreSyncExpression.syncExpression(Table16.schema, where: {
                Table16.keys.time_stamp.eq(nil)
            })
        ]
        try Amplify.add(plugin: AWSDataStorePlugin(
            modelRegistration: DataStoreModels(),
            configuration: .custom(errorHandler: onError, syncMaxRecords: UInt.max, syncExpressions: syncExpressions)
        ))

When initial sync queries complete we run following code to update sync expressions and re-sync data:

DataStoreManager.dateLimit = nil
            Task {
                try await Amplify.DataStore.stop()
                try await Amplify.DataStore.start()
            }

[fzy-github]

full crash log:
620fa117e58f4fdf86d1ac249c3179e1-symbolicated.txt

[harsh62]

@fzy-spyro Thanks for opening the issue. Our team is looking into it and will provide an update.

[madej10]

@harsh62 @lawmicha Could you please leave us an update. We're closing in on the public release, and the app keeps crashing regularly.

[5d]

Hello @madej10 @fzy-spyro,

Apologies, we were unable to replicate the crash. To address this issue:

1. Sending a diagnose report by running amplify diagnose --send-report
2. To gather more details about the crash, run Instruments.App and provide Leak and Zombie reports for the crashing app
3. Additionally, could you share a sample app that reproduce this issue?

Thank you.

[PrzemekIntent]

@5d @harsh62 @lawmicha

I've sent a diagnose report:
✅ Report saved: /var/folders/st/dtn7dgkx5t7ggh7m72c0421h0000gn/T/*******/report-1705578808834.zip ✔ Done Project Identifier: 2fe7f2619666633cbea23b61a6aaf9fd

I couldn't reproduce the issue by myself yet, but we're still noticing the crash about 2 times / day, having ~30 users.

I'll deliver more details if I figure out how to reproduce it but for onw, maybe the diagnose report will help.

[madej10]

@5d @harsh62 @lawmicha were you able to take a look at the diagnose report? We can't reproduce with Instruments.App as it makes the app barely usable. Please let us know if you're looking at this issue at all

[lawmicha]

Hi @PrzemekIntent / @madej10, are these crashes coming from the same app?

• Crash in AtomicDictionary "Error _ContiguousArrayBuffer.init - EXC_BREAKPOINT (SIGTRAP) Could not allocate memory" #3407 Could not allocate memory - crash in AtomicDictionary
• Crash in SyncMutationToCloudOperation.finish #3468 Attempted to dereference garbage pointer - crash in SyncMutationToCloudOperation.finish
• DataStore Crash in OutgoingMutationQueue #3467 (this issue) - Attempted to dereference garbage pointer at 0x1a. - crash in OutgoingMutationQueue.doStart

[madej10]

@lawmicha yes

[lawmicha]

Thanks for confirming, the schema looks the same as the redacted one from #3407 (comment) i'm not sure if we have more information here

[madej10]

We're working on a sample app that hopefully will allow you to reproduce in-house. We will keep you posted

[lawmicha]

Thanks, keep us posted. Feel free to reach out on Discord https://discord.com/invite/jWVbPfC if it is more convenient for you! We also monitor the swift-help channel

[fzy-github]

Hey @lawmicha
We keep experiencing this issue. One thing we notices it that most often this crash happens when we write data to datastore and app is in background or moves to background.

We write data to datastore when app is in background eg. as a result of bluetooth device sending data to our app

[harsh62]

@fzy-spyro Thanks for sharing this information. We will try to recreate the issues with the steps you mentioned.