RFC: Support Datastore for Web and Desktop

[abdallahshaban557]

We are creating this issue to get insights and feedback into our plans for supporting offline first and caching experiences with our API offerings when targeting Web and Desktop. We want to use this issue to collect feedback from you on the use cases you want to use Offline first and caching for, so that we can validate the path forward for us as we explore it further.

Please help us with providing us with the exact use cases you want to use Datastore for on Web and Desktop, and how it would benefit your users. For example:

• Cache data retrieved from API calls.
• Create a faster first-time loading experience by grabbing the latest retrieved data to users.
• Allow users to browse the app even if their network disconnects by using the latest retrieved data.
• Automatically re-connect and retrieve data updates when the internet connection is re-established.

[jamontesg]

Are you kidding, are you just going to start the project?

[abdallahshaban557]

Hi @jamontesg - We have created this feature request to make sure we capture all feedback on Datastore specifically, since our other categories have shipped for Web and Desktop support.

[dorontal]

Why we think Datastore + Web would be great for our (music collaboration) app? Here are a couple of more points to add to the bullet points at the top of this issue (which are also all benefits that we see):

1. Offline first apps are accessible by more people on earth: we want a musicians living in a remote village with no Internet to be able to walk into an Internet cafe, download some tracks, then go back to their village and record along with those tracks with no Internet, until they are ready to upload the recorded result back, say a month later when they return to the Internet cafe. Datastore can make it much easier to create an offline-first app due to its DB caching.

2. Apps running in the browser bypass the judgment / vetting of Apple or Google's app stores. When your app runs in the browser, there is less risk that one of those two companies would decline your app. There is less overhead from not having to wait for the approval process.

3. We hope Datastore will become fully optimized (e.g, DB updates that send partial data with only the fields that have changed rather than the entire model instance), saving us the time of having to do those optimizations ourselves.

4. We hope Datastore can help our client apps make fewer network requests, in general, by use of extensive caching - this will make it less costly to have any app up and running on AWS - it lowers the barrier to entry for any app publisher because it lowers the overhead costs of doing business because the app's users consume less cloud resources.

[abdallahshaban557]

Thank you @dorontal for this feedback!

[dkliss]

On device storage is important for all above use cases mentioned. In context of a cross-platform application, if all platforms cannot support a unified offline storage system it will become a challenge to remotely sync data to AWS from desktop or web devices. Offline first also ensures that in areas with limited bandwidth, data sync is scheduled at off-peak or user defined times to ensure user experience stays positive.

[loint]

@abdallahshaban557
Chat is the core feature for almost enterprise web and apps so we can have a better experiences and consistency (multi-platform app) in single implementation - our user is able to read message even their connections are interrupted.
We're waiting for this support for a long time because hard to maintain both web and app with same chat data synchronization.
Local storage is also benefit us in case we want to store all in local first and sync to server later to have a better experiences without depends direct on a real time connection.

[dgagnon]

My notes:

1. Make sure to include a fully working example of datastore + authenticator UI + cognito auth.
2. The datastore documentation needs to be explicit about which feature from appsync/graphQL are not supported, i.e. multiple owners. At the moment, the datastore doc only contains part of the information needed to manage the graphql models, so users are directly to the API(graphql) doc which very often will not work with the datastore. Alternatively, keeping the datastore to feature parity with appsync would help greatly.
3. Please make sure in include performance tests in your integration suite: above 20 models 10s of thousands of records, everything because really slow, when it even works. (think an online store with different kind of speciality products).
4. The datastore starts on configure, which creates exception when not logged in ( since you need to configure amplify to do the authentication. This creates confusions for developers. Unless a public API category is configured. datastore should not start until auth happened. Adding a flag to delay starting datastore would work as well.

As for the uses cases: it is, and always has been, offline support. This is the only reason we are using AWS. We built three version of our app so far and have had to completely removed datastore, which breaks the only advantage amplify has over the competition.

I will update my message as new ideas comes in from the team.

[jamilsaadeh97]

I think that offline first capabilities is best when it comes to querying and syncing data to the cloud. I'm using DataStore extensively and I must say, it's great but, at the same time, lacking in some areas like starting time, ability to handle a lot of models and data effectively, the differences with GraphQL API like multi owner....

My use case is a reservations application and offline first is the way to go since it offers a better user experience while everything is syncing to the cloud in the background. Sadly I can't have my app in the Web since DataStore is not implemented there.

For the past week I've been testing the possibility to migrate everything to GraphQL API and most probably I will, but I saw that DataStore makes everything easier, especially observeQuery() so that's why I like it but in my opinion it needs to be better in order to be production-ready for applications that offers services.

If the support of Web and Desktop means that DataStore will be rewritten in Dart, and, as result, will become more effective in iOS and Android, then Amplify Flutter would be the best out there.

[YUzushio]

I have high expectations for DataStore to operate on the Web. The characteristic of building apps with Flutter is its operation across all platforms.

While it's possible to implement with the GraphQL API, the usability of DataStore, which can be used on iOS and Android, is exceptionally excellent.

I am considering creating an app with Amplify Flutter, but the fact that it doesn't support the Web is causing significant issues for my business plans.

I can't propose whether to go online first or not, but having it implemented and executable before any discussion of cost and search efficiency is the most significant benefit for me. It would also be good news for many others who have given up on implementing Amplify Flutter.

By supporting the web, DataStore will undoubtedly make AmplifyFlutter the best solution in the world.

[Heilum]

We need amplify_datastore on macOS/Windows very much. please implement them officially. thanks in advance!

[CaptainDario]

@abdallahshaban557, is this being worked on?
I am planning an offline first sync feature and would like to know if this is just a proposal or if it is actively being worked on.

[jamontesg]

@abdallahshaban557 any dates ?

[thomasklaush]

Would be good, to get an approximate timeline for the feature.
I think a mix of Native App and Web for the Backend is a common topic.
Since Datastore and API don't work in parallel due to the Conflict Detection, this feature would be awesome for web as well!
I will go online soon with the web portal and looking forward to using it.

[abdallahshaban557]

Hi @CaptainDario - to be fully transparent, it is at a stage or two beyond a proposal right now, and are experimenting on how to consistently achieve the required feature-set across all the platforms we support.

@jamontesg and @thomasklaush We do not have an exact timelines yet, since we have not aligned on the best implementation path yet. We will continue to provide updates here when we have a clearer path forward.

[alterhuman]

Any update on this?

[nickordoodle]

From a Flutter perspective, the specific features are less relevant and the actual support as soon as possible for Web and Desktop is way more important. Whatever gets this to production fastest I think is the answer.

If DataStore becomes supported for remaining platforms, Flutter developers can confidently build cross-platform solely relying on Amplify's APIs.

I have faced roadblock after roadblock due to lack of platform support although it seems web support overall has increased drastically since inception.

I would also like to hear an update.

ex-Amazonian

[fabriquetonvoyage]

Hi @abdallahshaban557,

👍👍👍 Allow users to browse the app even if their network disconnects by using the latest retrieved data.
👍👍 Automatically re-connect and retrieve data updates when the internet connection is re-established.

It would be great for my "Travel Roadbook App" to have datastore support for web. Indeed, when the connection is lost (often during a trip), the users need to get all information about there travel and day by day schedule.

So, do you plan any release of this feature soon or do we have to forget it and build a custom code for web support ? ... It's so frustrating !

Thanks.

PS : More and more users don't want to install one more native app...

[abdallahshaban557]

Hi @fabriquetonvoyage - I am no longer part of the Amplify Flutter team, but @haverchuck can help provide more insights into the future of the library!

[thomasklaush]

@haverchuck please some information.
We go online soon.
Would be nice, to know the way and if I have to go with the problematic workaround in production.

[jamontesg]

Hi @haverchuck , any future for this project?

[BrandowBuenos]

Hello, everyone. I have an app in which Amplify has been working extremely well. However, I need to develop the web version and I just wanted to port the existing app to the web, but the datastore is not supported yet.

Is there any way to use it nonetheless? If not, is there a timeline for adding this support to the datastore library?

app: www.budgi.it

[thomasklaush]

@BrandowBuenos I used for web the API library.
But now I have two solutions and no data stored in the web version.
But I can use the same GraphQl interface and Data.
Only the conflict resolution makes issues, because Datastore uses Versioning.
If data is changed from Web, this makes issues.

[BrandowBuenos]

@thomasklaush thanks for helping me with your solution. It's definitely a good solution to think about.

What worries me is that maintaining two platforms, which even with identical codes, still requires disabling the library and making requests directly in GraphQL.

But anyway, thank you very much for the idea. Maybe start this way and discard the secondary project when DataStore is supported.

[thomasklaush]

@BrandowBuenos
I use injectable with two different flags to switch between two libraries (API and Datastore) with one interface.

My web app shares the backend, but has a different fronted, so I use the other one.

You can try to switch with an if and kIsWeb Boolean

[BrandowBuenos]

@thomasklaush, what methods do you use to make the queries? A custom one or ModelQueries.list? The latter is not having the expected behavior, returning only one result if there are many results.

[thomasklaush]

@BrandowBuenos

final request = ModelQueries.list(
        EquipmentEntry.classType,
        where: queryPredicate,
      );
      final response = await Amplify.API.query(request: request).response;

Don't forget to go through all pages of the result, this was an issue when I startet.

PaginatedResult<EquipmentEntry>? result = response.data;
bool hasNextResult = result!.hasNextResult;

while (hasNextResult == true) {
          logger.w("Next");
          final nextRequest = result.requestForNextResult;
          final secondResult = await Amplify.API.query(request: nextRequest!).response;
.
.
.
}

[BrandowBuenos]

@BrandowBuenos

final request = ModelQueries.list(
        EquipmentEntry.classType,
        where: queryPredicate,
      );
      final response = await Amplify.API.query(request: request).response;

Don't forget to go through all pages of the result, this was an issue when I startet.

PaginatedResult<EquipmentEntry>? result = response.data;
bool hasNextResult = result!.hasNextResult;

while (hasNextResult == true) {
          logger.w("Next");
          final nextRequest = result.requestForNextResult;
          final secondResult = await Amplify.API.query(request: nextRequest!).response;
.
.
.
}

Thanks @thomasklaush!

[ZoroLH]

Our app is running on the boat with weak internet connectiong. Offline ability is necessary for us. Our data is on cloud firestore now and we are planning to integrate aws iot. Datastore can make our framework simple.

We are using web app for product, Macos for development would help.

[thomasklaush]

Still no timeline?

An Easy way would be, to make the field version and deleted accessible for APIGraphQL.
This would make it possible to work with both.
@haverchuck any possibility for that?

Currently, I have this workflow:
I update the backend by amplify push.
Since I use Datastore, I have to enable Conflict Resolution.

This breaks GraphQL API, since I cannot read and write version field and cannot read deleted field.
I have to manually deactivate Conflict Resolution in the console for 10 functions.

If I want to use a custom Conflict Resolution via Lambda, amplify push reports an error on the backend update, so this unfortunately does not work.

Additionally, I take the risk for Data corruption.
If I get one wrong Entry, Datastore will not load any of the Table.

I will get in Production in April, and hopefully I can find a robust solution until this date.

[jamontesg]

This request is abandoned, the person who started it no longer works on the team and the person in charge shows no signs of life.
I have started to implement the web datastore, for my own.
Does anyone have an approach to the topic of how to work without deactivating conflict resolution and being able to remain in sync with the mobile part ?

Regards

[fabriquetonvoyage]

It's just crazy that AWS give up this feature in Amplify ...
How can we (the community of users) raise this point to the Amplify Team ? Maybe when they do their "Amplify Office Hours" meetings on discord ?

[BrandowBuenos]

It would be a great help to provide support without the offline caching part.

Perhaps even using secure storage for simplified storage.

[dgagnon]

It would be a great help to provide support without the offline caching part.

Perhaps even using secure storage for simplified storage.

datastore without offline data storage is not datastore, it's appsync. I am pretty sure you can use appsync already on web.

[thomasklaush]

@dgagnon you can do it, but you cannot use Datastore on Mobile and AppSync on Web combined.
Main issue is, that with Amplify generated fields for Datastore, you cannot access via AppSync the fields Version and Deleted.
This leads to a situation, where you cannot update and delete elements from web.

[fabriquetonvoyage]

Hi @haverchuck , any news or insights about this feature ? Thanks.

[BrandowBuenos]

Hey everyone,

I wanted to provide an update on recent changes made to our app and web platform. Specifically, I've completely removed the datastore and implemented a custom synchronization system from scratch.

Previously, we relied on the datastore for data storage and synchronization, but it often proved to be unstable, failing to sync data reliably. In response to this issue, I've developed a new approach that offers more robust synchronization capabilities and compatibility with web.

Here's a breakdown of what I've implemented:

Datastore Removal: The existing datastore functionality has been entirely removed from both the app and web platforms.
Custom Sync System: I've developed a custom synchronization system to replace the datastore. This system operates as follows:

• Data is saved locally using FlutterSecureStorage, ensuring secure storage on the device and web compatibility.
• Whenever an item is edited, it is added to a synchronization queue.
• Synchronization occurs immediately upon editing. If successful, the changes are propagated immediately. If not, the system retries synchronization every 30 seconds.

[fabriquetonvoyage]

Really nice Brandow ... It could be great if Amplify team could add this feature to the framework...
Do you publish the code in github or gitlab ? Thank you so much

[jamontesg]

Thanks for the news @BrandowBuenos , we are very excited about this functionality.
We are ready for help you with testing

Regards