Time-Based Replication
When handling data in fast-paced industries, such as airlines, retail, quick-service restaurants, and other time‑sensitive environments, implement a design pattern that focuses on efficient memory management and performance optimization.
A common implementation is the time-based replication design pattern, in which you use a Time‑to‑Live (TTL) Eviction strategy to set document retention logic; for example, logic specifying that only data written to the local Ditto store within the past 24 hours persists on the device.
The implementation process of the time-based replication design pattern with a TTL Eviction strategy varies based on your approach to TTL-based eviction management:
- Big Peer Management (Recommended) — Through regular interval HTTP API calls, the Big Peer cloud deployment delegates all TTL-based evictions mesh-wide. (For instructions, see Big Peer Handling: Implementing TTL Eviction) By opting for the Big Peer approach, you ensure that locally stored documents sync with the Big Peer before eviction.
- Small Peer Local Management — Each Small Peer device manages its own TTL-based evictions. (For instructions, see Small Peer Handling: Implementing TTL Eviction)
As Ditto's recommended approach, by opting to not only do you ensure that locally stored documents sync with the Big Peer before eviction, but you also streamline the eviction process by reducing the frequency of needing to perform subscription updates on Small Peers to align with the latest TTL queries.
To Implement TTL-based evictions using the Big Peer for centralized management, do the following using /api/v4/store/write endpoint with a POST request:
Using a top-level args variable, declare your TTL.
Include the necessary authorization headers. For information on access credentials, see Onboarding.
In your request body, indicate which documents to target for eviction. (See Specifying Targets for Purge)
Configure your HTTP server to run the TTL-based eviction operation at regular intervals using HTTP API calls.
For example, set it to run daily, removing all documents older than 24 hours from the current day.
Make sure the Small Peer devices are evicting and subscribing to the right documents. (See Configuring Small Peers)
- Pass the update command.
- Specify the target collection; for instance, the “orders” collection.
- Using a query, define time‑based eviction criteria to select documents where the createdAt time is less than or equal to $TTL.
- Ensure only eligible documents, those not previously evicted, are considered by setting evictionFlag in your query to true.
Once you've configured for centralized management using the Big Peer, configure Small Peers to make sure their devices are evicting and subscribing to the right documents:
Using the Find method, evict all documents that have the evictionFlag set to true.
Using the Evict method, execute the eviction process on the documents that match your time-based eviction criteria.
Once eviction is complete, using the Find method, subscribe to all documents that have the evictionFlag set to false.
Using the Subscribe method, execute the sync subscription on the documents that match your query criteria.
Make sure to remove the timestamp range from your subscription query.
This is because, by using the Big Peer to centralize management, Small Peers no longer need to perform regular subscription updates to align their time-based eviction criteria in their queries.
In most cases, choosing to use the Big Peer for TTL-based eviction management is ideal; however, in select scenarios where concerns like data availability, integrity, privacy, and security require local management, opting for the Small Peer local management approach may be the better choice.
Before invoking the Evict method to purge documents older than the TTL, cancel any active subscriptions that may be impacted by the eviction. (Canceling Affected Subscriptions)
By stopping any subscriptions affected by the evict update, you ensure that Ditto does not attempt to purge documents actively syncing across the mesh.
For more information about subscriptions, see Platform Manual > Data Sync.
Once you ensure that any ongoing subscriptions affected by the eviction update are canceled, run the eviction process. (See Invoking TTL-Based Eviction)
Once the eviction process is complete, set up a new subscription with your updated time‑based eviction criteria (Creating New Subscriptions)
To cancel a subscription:
To maintain data integrity, you must cancel any active subscription that the eviction update may impact before executing your eviction process.
Otherwise, an undesirable loop may result in which Ditto continuously evicts documents from the local Ditto store and then automatically re‑syncs them, leading to increased memory usage.
Using a top-level args variable, declare your TTL.
Using the Find method, define your time‑based eviction criteria.
Using the Evict method, execute the eviction process on documents matching your criteria.
Notice that the query used for eviction is the opposite of the query used for subscription, enclosed as parameters in the Find method; that is, they do not select the same documents.
By using opposite queries, you prevent potential conflict by ensuring that the queries you use for eviction and subscription are distinct and do not overlap.
To update a subscription with your updated time‑based eviction criteria, do the following:
You must declare your subscription object from the top-most scope of your app to ensure access throughout the life cycle of your app.
Otherwise, you cannot modify or cancel your subscription from any part of your code, resulting in difficulty and potential errors when managing the subscription's lifecycle.
Specify the name of the collection to watch; for example, the "flights" collection.
Using the Find method, define the query that retrieves documents where the "createdAt" field is greater than the ttl.
Using the Subscribe method, initiate your subscription.