A couple of weeks ago I proposed a flutter app architecture which I humbly called, “The Perfect Reactive, Local-First Mobile Application”.

Boy, that didn’t age well…

In the weeks that followed, I took a deeper dive into flutter reactivity and “local-first” patterns and realized that the architecture I called “perfect” actually had a ton of flaws. To name a few:

• Each remote data source ends up being implemented in exactly the same way, just with table names swapped out. This violates the DRY principle.

• Services are often times an unnecessary intermediary for simple database transactions

• There is redundancy and memory inefficiency in having the central App State hold all data that view models need to react to. All that data is already available in the local database.

• Repositories unnecessarily depend on both local and remote data sources when, realistically, we should only ever be writing to and reading from the local database.

• It does not answer the question of “how should we divide these services?”

• If we scope out services neatly, there’s actually relatively little inter-service communication, obviating the need for an event bus.

So with my head bowed, I went back to the drawing board and this time I got a couple of other engineers — including the mind behind a very popular flutter framework — to give my new architecture their stamp of approval.

So here it is! What I’m cautiously calling “The not-perfect-but-good-enough-for-now-and-other-engineers-think-so-too Reactive, Local-First Mobile Application Architecture”

Here it is!

For the app my friends and I are building a video-first social app, we’re using the following stack…

• Flutter as the mobile app framework

• Stacked as a Flutter

• Isar DB for the local DB

• Supabase for the remote database and file storage

…but the core concepts should be the same no matter your stack.

Solution TL;DR

“Local-First” for Structured Data

• A copy of all DB data relevant to a user is stored in the local DB. Several years worth of data for a user is likely to be < 100 MB.

• View Models and Services are only ever writing and reading data from the local database.

• In the background, a service keeps the local DB in sync with the remote DB, resolving conflicts where necessary.

“Local-First” for File Uploads/Downloads

• When the user uploads an asset (profile photo, video post, etc.) it is simply stored locally and uploaded in the background. The UI responds to the locally stored file immediately.

• When the background DB sync service detects updates to rows with assets (e.g. a new video post or an updated group image) it queues the file for background fetching.

• Unlike with DB data, we cannot store all files locally. So we still need a repository/service as an abstraction for providing files to view models (no matter if that file is stored locally or remotely).

Reactivity

• Since a copy of all data a user needs will be stored in the local DB, view models can watch for changes to the local database and rerun queries to generate the updated state and refresh their view.

• This is organized like so: ViewModel → App Services (business logic) → Local DB Service (abstracts Isar) → Isar DB

  • The local DB service is an API for performing CRUD operations against the local DB and listening for changes.

  • The App Services listen for changes and notify view models via the ListenableServiceMixin.

  • View models request the data they need from app services and use listenableServices to react when that data changes.

“Local-First” Implementation

When a user makes a database operation (updating their display name, adding a group image, inviting a member, posting a video, etc.), it is performed against the local database.

We then have a BackgroundSyncManager service that writes these changes to the remote database and resolves conflicts.

In the short term, we can naively implement the BackgroundSyncManager to run on an interval instead of listening to real-time updates from the local and remote databases. This makes things much simpler in the short-term and is totally acceptable tradeoff (a few seconds delay on seeing someone’s reply or video is just fine).

DB Syncing Strategy

Handling New & Updated DB Rows:

• Each DB table gets a last_updated timestamptz column.

• Whenever a database record is created or updated in the remote DB, its last_updated column is updated. Similarly for newly created/updated rows in the local DB

• When the BackgroundSyncManager runs, it queries for all rows where last_updated > lastSyncTime in both the local and remote DB.

• Local DB updates are written upstream. Remote DB updates are written locally. If there are conflicts, we resolve this with a “Last Write Wins” strategy.

Handling Deleted DB Rows:

• A new table called deletions is added in both the remote and local DBs with the columns:

  • table_name

  • record_id

  • deleted_at

• When syncing, we fetch all rows from the remote DB where deleted_at > lastSyncTime. Similarly, we fetch all rows locally where deleted_at > lastSyncTime and delete those rows in the remote DB.

File Syncing Strategy

User-created files (videos, images, etc.) are named with a uuid for several purposes that will be discussed throughout.

Handling New or Uploaded Files Created Locally:

• A new local DB collection fileUploadQueue is added with the columns:

  • tableName

  • recordId

  • columnName

  • filename

  • createdAt

• When a new row is created, or an asset is updated, the lastUpdated column is set to the current timestamp, and the new asset is added to the FileUploadQueue collection.

• Now when the BackgroundSyncManager encounters a local DB record that it wants to insert/update in the remote DB, it will check if there are any records in the FileUploadQueue for that correspond to this tableName, recordId and columnName. If so, it will upload those files and delete them from the queue before upserting the row.

• The upload is prioritized so there are never new rows in the remote DB pointing to files that do not yet exist. Using uuid names ensures that both files can exist at the same time.

• Files are never deleted from the cache until they are uploaded.

Handling New or Uploaded Files Created Remotely:

• A new local DB collection FileDownloadQueue is added with the columns:

  • tableName

  • recordId

  • columnName

  • filename

  • priority

  • createdAt

• When the BackgroundSyncManager fetches new/updated rows with asset columns (e.g. image_uri, video_uri, thumbnail_uri, etc.), it will add it to the FileDownloadQueue.

Background Fetching:

• The FilesRepository goes through the FileUploadQueue from highest priority to lowest priority. For example, thumbnails are highest priority to display on the grid, while videos are slightly lower priority. We don’t want download of thumbnails, profile pics, etc. to get stuck behind large videos.

• It first checks that the filename column still matches the <asset>Uri specified in tableName.recordId.columnName. If not, that means this record was updated and this file is no longer needed. If so, it will delete the row in the FileUploadQueue and move on to the next one.

  • We use uuid filenames to ensure that we can detect when an asset has been updated, instead of giving it the same name as before. 

Deleting Files in the Remote Bucket:

• A DB trigger fires on updates for all tables which have columns that point to assets. When a database row with an asset is updated, it checks to see if the assetUri string has changed. If so, a webhook is invoked to delete the old file from the bucket with the old uri.

  • This makes the assumption that a bucket asset is only ever referenced in a single row, never multiple.

• Again, this is why we are using uuids for filenames – so we can detect updates.

Deleting Files from Local Device:

• The LocalFileStorageService manages the LRU cache. It reads from a local collection called LocalAssets which tracks:

  • filename

  • sizeBytes

  • lastUsedAt

• When it needs to clear space before storing a new file, it will attempt to delete the least recently used files first. Before doing so, it will check that the file does not exist in the FileUploadQueue. If so, it will skip this file and try to clear another. 

Fetching Files for Display:

• All image and video widgets use the FilesService to get their file 

• This service checks the local cache for the file. If it does not exist, it tries to get a signedUrl from the remote database.

“Reactive Views” Implementation

The local database is kept in sync with remote database changes on a fairly regular interval. View Models react immediately to updates to the local database.

ViewModel → App Services (business logic) → Local DB Service (abstracts Isar) → Isar DB

Isar DB

Local Database that stores all the data the user needs. A sync service writes changes from the remote DB to the local DB. 

Isar has “watchers” which allow other classes in our application to listen for changes in specific collections, queries, or individual objects in the DB.

Local DB Service

This is a “Facade Service” (using Stacked terminology) for the Isar package, abstracting away our choice of Isar as the local DB. 

In addition to simple CRUD methods, it also provides a couple of methods watchTable and watchRow which enable consumers of this service to subscribe for notifications when data changes in the local DB.

App Services

These “App Services” (again, using Stacked terminology) contain the business logic of our application.

They use the local DB service to execute queries and use the ListenableServiceMixin to notify listeners (i.e. view models) when data changes.

View Models

View models extend the ReactiveViewModel class. 

View models request the data they need from app services and listen to listenableServices getter tells the ViewModel to rebuild whenever any of the listed services call notifyListeners().

It is the responsibility of the viewmodel to create the unique data structures needed by their view. They fetch the data they need from app services in order to do so. 

Optimizations

• For cache busting purposes, every photo and video is given  a UUID as its name in the backend. This ensures that when an asset is updated (e.g. the group’s image), other users are not fetching the same file locally. 

• Use batching for fetching or syncing local changes (e.g. 200 rows at a time). Ensure lastSyncTime is set to the last_updated time of the most recently synced record.

• Remove or archive rows in the deletions table after some amount of time so the table does not grow unbounded.

• Instead of persistent polling, use web sockets or Supabase real-time to decrease syncing delays and also decrease load on the DB servers.

• Index on the last_updated column

• Add createMany and updateMany so that local updates can be written in batches.

• Sort and sync updates from oldest to newest, setting lastSyncTime after each successful update. This makes the system resilient to crashes.

Edge Cases/Open Questions

• Does conflict resolution solve itself? If we are resolving updates from oldest to newest, then the “last write wins” strategy should be enforced by default.

• What do you do if the DB server rejects our update (RLS policies, changed schema, etc.)

• How do we handle cases where the lack of RLS policies allows us to do things that we otherwise wouldn’t be able to do upstream?

• How do we handle the lack of DB triggers? E.g., when a group is created, a group membership should be created too.

Quick reminder - If you appreciate my writing, please reply to this email or “add to address book”. These positive signals help my emails land in your inbox.

If you don't want these emails, you can unsubscribe below. If you were sent this email and want more, you can subscribe here.

See you next week — Rayhan

P.S. [Link to a piece of content they would be interested in]