DataStore subscriptionProcessor failed

[mvishnureddy007]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.

Language and Async Model

Java

Amplify Categories

Authentication, GraphQL API, DataStore

Gradle script dependencies

implementation 'com.amplifyframework:core:2.13.0'
implementation 'com.amplifyframework:aws-api:2.13.0'
implementation 'com.amplifyframework:aws-datastore:2.13.0'
implementation 'com.amplifyframework:aws-auth-cognito:2.13.0'
implementation 'com.amplifyframework:rxbindings:2.13.0'

Environment information

implementation 'com.amplifyframework:core:2.13.0' implementation 'com.amplifyframework:aws-api:2.13.0' implementation 'com.amplifyframework:aws-datastore:2.13.0' implementation 'com.amplifyframework:aws-auth-cognito:2.13.0' implementation 'com.amplifyframework:rxbindings:2.13.0'

Please include any relevant guides or documentation you're referencing

No response

Describe the bug

Hi,
We are getting this error after user has been logged in and out for multiple times.

DataStoreException{message=DataStore subscriptionProcessor failed to start., cause=GraphQLResponseException{message=Subscription error for MessageRecipient: [GraphQLResponse.Error{message='subscription exceeds maximum value limit 20 for operator containsAny.', locations='null', path='null', extensions='{errorCode=400}'}], errors=[GraphQLResponse.Error{message='subscription exceeds maximum value limit 20 for operator containsAny.', locations='null', path='null', extensions='{errorCode=400}'}], recoverySuggestion=See attached list of GraphQLResponse.Error objects.}, recoverySuggestion=Check your internet.}

GraphQLResponseException{message=Subscription error for MessageRecipient: [GraphQLResponse.Error{message='subscription exceeds maximum value limit 20 for operator containsAny.', locations='null', path='null', extensions='{errorCode=400}'}], errors=[GraphQLResponse.Error{message='subscription exceeds maximum value limit 20 for operator containsAny.', locations='null', path='null', extensions='{errorCode=400}'}], recoverySuggestion=See attached list of GraphQLResponse.Error objects.}

Reproduction steps (if applicable)

No response

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

amplifyconfiguration.json

No response

GraphQL Schema

// Put your schema below this line

Additional information and screenshots

No response

[tylerjroach]

@mvishnureddy007 You appear to be running into a limit on the AppSync side. Do you have >20 containsAny predicates?

[mvishnureddy007]

@tylerjroach We don't have containsAny predicate in our code. I'm providing our observer Code here

QueryPredicate predicate = MessageRecipient.USER_ID.eq(ApplicationData.shared.userProfileData.getId());
QuerySortBy sortBy = new QuerySortBy("MessageRecipient", "sentAt", QuerySortOrder.ASCENDING);
List sortByList = new ArrayList();
sortByList.add(sortBy);
ObserveQueryOptions options = new ObserveQueryOptions(predicate, sortByList);
Amplify.DataStore.observeQuery(MessageRecipient.class,
options,
observationStarted,
onQuerySnapshot,
onObservationError,
onObservationComplete);

Our scenario is happening because of Same user logged into multiple devices.

I'm providing schema
type MessageRecipient @model @auth(rules: [
{allow: groups, groupsField: "permittedTo"}
]) {
id: ID!
messageId: ID! @Index(name:"byMessage")
groupId: String! @Index(name: "byGroup")
userId: Int! @Index(name:"byUser")
status: MessageStatus!
sentAt: AWSDateTime!
readAt: AWSDateTime
permittedTo: [String!]
}

[tylerjroach]

Even though the Android client only uses client-side filtering, the service still uses Enhanced Subscription Filtering on the service side to enforce dynamic auth rules.

This rule

@auth(rules: [
  {allow: groups, groupsField: "permittedTo"}
]) 

Gets applied service-side in the subscription auth resolver with the containsAny operator to match the groups the user is in to the groups define in the group field permittedTo. The limit is listed here.

Most likely the root cause is: either some of the records have too many groups listed in the permittedTo field or the other way around, the user is a member of too many groups.

I'm not exactly sure how multiple sign-in/sign-outs, or multiple devices would cause the failure. As far as I understand, the limit is per subscriber.

If the explanation above doesn't help resolve the issue, please reply back with information around how Amplify is configured. How DataStore is started/stopped, and any other information you believe may be relevant, such as specific scenarios around the mentioned multiple device/sign-ins.

[mvishnureddy007]

If the user can be part of more than 20 groups, which is common situation in our workflow.
Can you help us, how to handle this?

[iartemiev]

There's no way to accommodate this when using AppSync subscriptions due to the way subscription filtering is implemented in the service. This is a hard limit in the service and it's not possible to increase it on a case-by-case basis.

You may be able to disable subscriptions for models that use dynamic group auth ({allow: groups, groupsField: "permittedTo"}) and then rely on DataStore sync queries to retrieve new/updated records, similar to a polling approach, but it's not possible to get this working with real-time subscription messages without reducing the number of groups to <= 20.

Note that the same limit applies for a list of owners or any other dynamic, list-based row-level authorization.

[mvishnureddy007]

We have removed this

({allow: groups, groupsField: "permittedTo"})

Subscription is working fine.

But we got another issue. ie., After every fresh login the Data store query sync is taking long time to get data from cloud.
Step we are following

1. Once logged into mobile we are checking auth session.
2. After that we are doing this for start data store query
   Amplify.DataStore.start(
   () -> Log.i("MyAmplifyApp", "DataStore started"),
   error -> Log.e("MyAmplifyApp", "Error starting DataStore", error)
   );
3. Once i go to chat at first time data store query retring empty or 1 record only.

[tylerjroach]

@mvishnureddy007 Please update to Amplify Android 2.14.11. The last few Amplify releases have provided significant performance improvements for subscription and sync.

Please report back after updating.

[mvishnureddy007]

After updated Amplify Android 2.14.11 also the Data store query sync is taking long time to get data from cloud. And observer query also delayed to get the real time data for user messages

[mvishnureddy007]

@mvishnureddy007 Please update to Amplify Android 2.14.11. The last few Amplify releases have provided significant performance improvements for subscription and sync.

Please report back after updating.

After updated Amplify Android 2.14.11 also the Data store query sync is taking long time to get data from cloud. And observer query also delayed to get the real time data for user messages

[mvishnureddy007]

@mvishnureddy007 Please update to Amplify Android 2.14.11. The last few Amplify releases have provided significant performance improvements for subscription and sync.

Please report back after updating.

As soon as I logged in to the app this logs I got continuously

2024-05-07 10:33:36.303 29894-30946 amplify:aws-datastore SQLCommandProcessor executeExists in 1 ms inTransaction: true SQL: SELECT EXISTS(SELECT 1 FROM ModelMetadata WHERE ModelMetadata.id = ?);
2024-05-07 10:33:36.305 29894-30946 amplify:aws-datastore V Could not find getId() on com.amplifyframework.datastore.appsync.ModelMetadata. Fallback to direct field access.
2024-05-07 10:33:36.305 29894-30946 amplify:aws-datastore V Could not find get_deleted() on com.amplifyframework.datastore.appsync.ModelMetadata. Fallback to direct field access.
2024-05-07 10:33:36.306 29894-30946 amplify:aws-datastore V Could not find get_lastChangedAt() on com.amplifyframework.datastore.appsync.ModelMetadata. Fallback to direct field access.
2024-05-07 10:33:36.306 29894-30946 amplify:aws-datastore V Could not find get_version() on com.amplifyframework.datastore.appsync.ModelMetadata. Fallback to direct field access.
2024-05-07 10:33:36.306 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = MessageRecipient|***************************************
2024-05-07 10:33:36.306 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = null
2024-05-07 10:33:36.306 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = *************************
2024-05-07 10:33:36.306 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = 2
2024-05-07 10:33:36.307 29894-30946 amplify:aws-datastore V SQLCommandProcessor execute in 1 ms inTransaction: true SQL: INSERT INTO ModelMetadata (id, _deleted, _lastChangedAt, _version) VALUES (?, ?, ?, ?)
2024-05-07 10:33:36.310 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = 8d84c7ee-041b-4fe8-b472-a6b862761396
2024-05-07 10:33:36.310 29894-30946 amplify:aws-datastore V SQLCommandProcessor executeExists in 0 ms inTransaction: true SQL: SELECT EXISTS(SELECT 1 FROM MessageRecipient WHERE MessageRecipient.id = ?);
2024-05-07 10:33:36.321 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = *******************************
2024-05-07 10:33:36.321 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = 2024-03-20T06:25:53.389000000Z
2024-05-07 10:33:36.321 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = **********************************
2024-05-07 10:33:36.321 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = **********************************
2024-05-07 10:33:36.321 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = ["",""]
2024-05-07 10:33:36.322 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = null
2024-05-07 10:33:36.322 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = 2024-03-20T06:25:51.668000000Z
2024-05-07 10:33:36.322 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = READ
2024-05-07 10:33:36.322 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = 2024-03-20T06:25:53.639000000Z
2024-05-07 10:33:36.322 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = ***
2024-05-07 10:33:36.322 29894-30946 amplify:aws-datastore V SQLCommandProcessor execute in 1 ms inTransaction: true SQL: INSERT INTO MessageRecipient (id, createdAt, groupId, messageId, permittedTo, readAt, sentAt, status, updatedAt, userId) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
2024-05-07 10:33:36.325 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = 8
2024-05-07 10:33:36.325 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = *************************************
2024-05-07 10:33:36.325 29894-30946 amplify:aws-datastore V SQLCommandProcessor executeExists in 0 ms inTransaction: true SQL: SELECT EXISTS(SELECT 1 FROM MessageRecipient WHERE (MessageRecipient.userId = ? AND MessageRecipient.id = ?));
2024-05-07 10:33:36.327 29894-30946 amplify:aws-datastore V SQLCommandProcessor.bindValueToStatement(..., value = MessageRecipient|**************************************

Once it is finished the real time messaging and data store query is working fine with out delay

Can any once help us what is the problem? How to fix it.