In a distributed data system like Ditto, deletion follows a “soft delete” pattern. This means that when you mark an item as deleted, it’s not immediately removed from the collection or local storage. Instead, the item is flagged as deleted to ensure that other devices in the network remain in sync. To fully remove the item and free up space, you’ll need to evict these “soft deleted” items. Implementing a well-thought data storage strategy can ensure that your app remains efficient, resulting in faster load times, improved battery life, and a more responsive UI.

​
Using Eviction

Use the EVICT API method to remove data from the local disk. You will need to set up an automatic process that evicts documents on a regular basis, such as once per day. Evict documents that are accessed less frequently or are no longer relevant in order to free up disk space.

​
Eviction can slow down the Mesh

Eviction triggers a full resync with every connected peer, which can create excessive network traffic and processing overhead.Call EVICT on a regular basis, but no more than once per day during periods of minimal disruption, such as after hours and at night.

​
Time-to-Live Eviction Strategy

Implement a design pattern that focuses on efficient memory management and performance optimization when handling data in fast-paced industries, such as airlines, retail, quick-service restaurants, and other time‑sensitive environments. The implementation process of the time-based replication design pattern with a TTL Eviction strategy varies based on your approach:
  • Big Peer Management (Recommended) — Through regular interval HTTP API calls, the Big Peer cloud deployment delegates all TTL-based evictions mesh-wide. By opting for the Big Peer approach, you ensure that documents sync with the Big Peer before eviction to prevent any potential data loss. (See Big-Peer-Based Evictions)
  • Small Peer Local Management — Each Small Peer device manages its own TTL-based evictions. This is suitable if you either are not using a Big Peer, or if you are confident operationally that devices will never be offline longer than the eviction process will run. (See Small-Peer-Based Evictions)
A common implementation is the time-based replication design pattern, in which you use a Time‑to‑Live (TTL) Eviction strategy to set document retention logic; for example, logic specifying that only data written to the local Ditto store within the past seven days will persist on a local device.

​
Example Scenario

Imagine you’re developing a meal ordering app that flight attendants use to record passenger meal orders on their devices. To ensure that your app remains highly performant throughout its life cycle while also guaranteeing that flight attendants using the app always work with the most recent and relevant information, you specify a TTL of 72 hours so that only documents written to the local Ditto store within the past 72 hours persist on the Small Peer device. 
{
  "createdAt": "2022-09-17T20:00:46.945Z",
  "expiresAt": "2022-09-20T20:00:46.945Z",
  "flightNo": "DIT101"
}

​
Flag-based Eviction

As Ditto’s recommended approach, by opting to use the Big Peer to manage TTL-based evictions not only do you ensure that locally stored documents sync with the Big Peer before eviction, but you also streamline the eviction process by reducing the frequency of needing to perform subscription updates on Small Peers to align with the latest TTL queries. To Implement TTL-based evictions using the Big Peer make a POST request to the /api/v4/store/write endpoint:
1

Using a top-level args variable, declare your TTL.
2

Include the necessary authorization headers.
3

In your request body, indicate which documents to target for eviction.
4

Configure your HTTP server to run the TTL-based eviction operation at regular intervals using HTTP API calls.For example, set it to run daily, removing all documents older than 24 hours from the current day.
5

Make sure the Small Peer devices are evicting and subscribing to the right documents.

​
Specifying Targets for Purge

  1. Pass the update command.
  2. Specify the target collection; for instance, the “orders” collection.
  3. Using a query, define time‑based eviction criteria to select documents where the createdAt time is less than or equal to $TTL.
  4. Ensure only eligible documents, those not previously evicted, are considered by setting evictionFlag in your query to true.
cURL
curl -X POST 'https://{app_id}.cloud.ditto.live/store/execute' \
  --header 'Authorization: Bearer $API_KEY' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "statement": "SELECT * from orders WHERE createdAt <= :TTL && evictionFlag == false",
    "args": TTL
  }
}'

​
Configuring Small Peers

Once you’ve configured for TTL-based eviction management using the Big Peer, configure Small Peers to make sure their devices are evicting and subscribing to the right documents:
1

Using the Find method, evict all documents that have the evictionFlag set to true.
2

Using the EVICT method, execute the eviction process on the documents that match your time-based eviction criteria.
pseudocode
ditto.store.execute("EVICT FROM orders WHERE evictionFlag = true")
3

Once eviction is complete, using the Find method, subscribe to all documents that have the evictionFlag set to false.
4

Using the subscription method, execute the sync subscription on the documents that match your query criteria.Make sure to remove the timestamp range from your subscription query.This is because, by using the Big Peer for management, Small Peers no longer need to perform regular subscription updates to align their time-based eviction criteria in their queries.
pseudocode
ditto.store.registerSubscription("SELECT * from orders WHERE createdAt > locationId == 1234 && evictionFlag == false")

​
Time-Based Eviction

In most cases, choosing to use the Big Peer for TTL-based eviction management is ideal; however, in select scenarios where concerns like data availability, integrity, privacy, and security require local management, opting for the Small Peer local management approach may be the better choice.
1

Before invoking the Evict method to purge documents older than the TTL, cancel any active subscriptions that may be impacted by the eviction. (See Canceling Affected Subscriptions)
2

By stopping any subscriptions affected by the evict update, you ensure that Ditto does not attempt to purge documents actively syncing across the mesh.
3

Once you ensure that any ongoing subscriptions affected by the eviction update are canceled, run the eviction process. (See Invoking Eviction Updates)
4

Once the eviction process is complete, set up a new subscription with your updated time‑based eviction criteria. (See Creating New Subscriptions)

​
Cancel Affected Subscriptions

To maintain data integrity, you must cancel any active subscription that the eviction update may impact before executing your eviction process. Otherwise, an undesirable loop may result in which Ditto continuously evicts documents from the local Ditto store and then automatically re‑syncs them, leading to increased memory usage.
pseudocode
subscription.cancel()

​
Invoke Eviction Updates

1

Using a top-level args variable, declare your TTL.
2

Using the Find method, define your time‑based eviction criteria.
3

Using the Evict method, execute the eviction process on documents matching your criteria.Notice that the query used for eviction is the opposite of the query used for subscription, enclosed as parameters in the Find method; that is, they do not select the same documents.By using opposite queries, you prevent potential conflict by ensuring that the queries you use for eviction and subscription are distinct and do not overlap.
pseudocode
ditto.store.execute("SELECT * from orders WHERE createdAt <= :ttl", args: [ "ttl": getTimeToLive() ])

​
Create New Subscriptions

To update a subscription with your updated time‑based eviction criteria, do the following:
You must declare your subscription object from the top-most scope of your app to ensure access throughout the life cycle of your app.Otherwise, you cannot modify or cancel your subscription from any part of your code, resulting in difficulty and potential errors when managing the subscription’s lifecycle.
1

Specify the name of the collection to watch; for example, the "flights" collection.
2

Using the Find method, define the query that retrieves documents where the "createdAt" field is greater than the ttl.
3

Using the Subscribe method, initiate your subscription.
pseudocode
ditto.store.registerSubscription("SELECT * from orders WHERE createdAt > :ttl", args: [ "ttl": getTimeToLive() ])