Data-Handling Essentials

​

Introduction

In Ditto, there’s a clear distinction between traditional CREATE, READ, UPDATE, and DELETE (CRUD) database operations and data sync.

​

CRUD

For CRUD operations, you interact with data stored locally in the Ditto store by invoking a single operation on the store namespace. Once executed, Ditto returns a response object encapsulating the results.&

The following table provides a high-level overview of the different ways you can perform CRUD in Ditto:

​

Sync

For syncing, you set up remote listeners, or sync subscriptions, to monitor changes to the data you’re interested in. When data changes, Ditto automatically syncs the delta changes to your local Ditto store.

​

Observe

In addition to setting up sync subscriptions to monitor mesh-wide data changes, you can establish local listeners known as store observers to monitor changes in the Ditto store operating locally.

A store observer is a DQL query that runs continuously and, once Ditto detects relevant changes, asynchronously triggers the callback function you defined when you set up your store observer. For instance, when the end user updates their profile, display the profile changes to the end user in realtime.

​

Documents and Collections

Ditto stores data in structured JSON-like document objects. Each document consists of human‑readable fields that identify and represent the information the document stores.&

The following snippet provides an example of a basic JSON-encoded document object:

{

  "_id": "abc123",

  "make": "Hyundai",

  "year": 2018,

  "color": "black"

}

A single document consists of one or more fields* *that self‑describe the data it encodes; each field is associated with a value:

A grouping of related documents is referred to as a collection. Think of a collection* as a table in a relational database table — but with greater flexibility in terms of the data it can hold — and *the documents in a collection like table rows:

In the following structure, the location field property is logically grouped with details about the car, such as its make, year, and color.&

It contains nested fields. also referred to as subfields,* *representing both the coordinates and the address where the car is located:

​

Dedicated Query Language

Ditto Query Language (DQL) is the platform’s dedicated query language for defining criteria for local database operations and data sync with remote peers. For instance, you use DQL to perform various filter operations like traditional CREATE, READ, UPDATE, and DELETE (CRUD) and filter data for sync subscriptions. 

DQL is a familiar SQL‑like syntax designed specifically for Ditto’s edge-sync features that enable the platform’s offline-first capabilities. DQL offers features like:

• Reusable statements
• Clear syntax
• Advanced querying capabilities

​

Query Statements Overview

The following table provides an overview of the types of statements you’ll write in DQL and enclose within your API method operations to specify data selection criteria:

• ditto.store.execute — Interact with data stored locally within the Small Peer device running your app. (See ditto.store.execute)
• ditto.store.registerObserver — Set up store listeners to monitor changes to data in the local Ditto store based on specified query criteria. (See ditto.store.registerObserver)
• ditto.sync.registerSubscription — Set up subscriptions for syncing data from other peers based on their associated queries. (See ditto.sync.registerSubscription)
• ditto.presence — Enable end-user functionality like mesh network monitoring, management, and transport optimization. (See Presence Operations)

​

Data Types and Scalar Types

When writing queries with DQL, data formatting is categorized into two buckets:

​

Specifying Data Types

To use a DQL type other than a REGISTER — the default data type in Ditto — you must explicitly specify the type in your query; otherwise, Ditto defaults to the REGISTER type as follows.&

SELECT * FROM COLLECTION cars

WHERE features.trim = 'Standard'

Here is an example illustrating the same SELECT statement query explicitly expressed as a MAP structure. It specifies retrieval of the MAP structure storing the"features" collection with a "trim" field property set to a value of "standard":

SELECT * FROM COLLECTION cars (features MAP)

WHERE features.trim = 'Standard'

​

Conflict Resolution

As the foundation of how Ditto exposes and models data, these data types leverage conflict-free replicated data type (CRDT) technology to ensure that no data inconsistencies occur as a result of concurrent modifications; that is, simultaneous edits made to the same data types in multiples local Ditto stores.&

All data types —REGISTER, MAP, and ATTACHMENT— adhere to the causal consistency model when resolving concurrency conflicts.&

The causal consistency model is a guarantee that if there is an operation that must happen before another operation — for example, events A and B, where B is a result of A — all peers agree upon and observe the same sequential order of these operations; as in, A always executes before B.

​

Merge Strategies

In Ditto’s implementation, conflicts are automatically resolved, merged, and synced across peers without the need for coordination or validation from a centralized authority.&

Within this consistency model, there are two principles for guiding conflict resolution at merge:

• Last-write-wins merge strategy
• Add-wins merge strategy

The following table provides a quick overview of the data types you can use to write queries, along with their merge strategies, a brief description, and a common usage scenario:

​

Representing Complex Datasets

When you want to embed a hierarchical structure to represent complex parent‑children data structures within a document in DQL, you have the option to nest either of the following:

• JSON blob functioning as a REGISTER
• MAP

The decision between the two depends on your specific use case.

For instance, to represent embedded values with dependencies, such as a GPS coordinate along with its corresponding address, structure your data in a JSON object, as follows:&

"location": {

  "coordinates": [-122.0308, 37.3318],

  "address": "123 Main St, San Francisco, CA 94105"

}

This is because, unlike a MAP, a JSON object functions as a single unit. Managing both the coordinate and address as a cohesive unit ensures that any changes made to one automatically update the other.

To represent embedded values with no dependencies; that is, you want the flexibility to update each key-value pair independently, and structure your data as separate REGISTER fields in a MAP:

"features": {

  "trim": "Standard"

  "speakers" : 5

}

To represent highly complex data structures in which you need to establish additional hierarchies, embed a MAP within another MAP, as follows:&

"owner": {

      "name": "John Larson",

      "information": {

          "address": "2147 Chestnut St, Oakland, CA 94607",

          "phone": "555-555-5555",

          "updated": "June 2, 2023"

         }

     }

​

Distinctions by Type

The following graphic and corresponding table aim to demonstrate the distinct capabilities and versatility of each DQL type.&

​

MAP Data Type Conflicts

An issue unique to MAP data types is the possibility for two offline peers to create a new document, in which one peer represents the field as an object (MAP), while the other peer represents the field as an array.

​

Divergent Types Preventing Merge

The following snippets illustrate a scenario of a type-level conflict unique to MAP types.&

Peer A creates the following new document:

{

  "name":"Bob Jones",

  "address": {

    "street":"Long Road",

    "house number":10298,

    "zip":"90210"

  }

}

While at the same time Peer B creates the following new document:

{

  "name": "Bob Jones",

  "address":[

    10298,

    "Long Road",

    "90210"

  ]

}

​

Managing Conflicts: Update History

​

Store Operations

There are two ways of interacting with the Ditto store operating locally on end-user devices:

• Perform a one-time execution of a create, read, update, delete (CRUD) operation against the store namespace.

• Asynchronously monitor local changes by setting up observers against the store namespace.

​

ditto.store.execute

Interact with data stored locally within the Small Peer device running your app.

On the right is a graphic illustrating CRUD operations on a Small Peer end‑user device.&

With these one-time operations, you perform a single action, such as modifying or retrieving data, on the Ditto store at a given time.&

Once invoked, you must wait for the operation to complete before continuing to the next action in your code.

For example, the following single execution query, once called, uses the execute API method and a local SELECT query to search within the local Small Peer store for data in a dataset named cars:&

​

Creating

To create a new document in your local Ditto store, call INSERT. Following is an example of how to perform an INSERT operation using Ditto’s SDK.

​

Reading

Search for documents within your local Ditto store by calling the execute API method on a SELECT query as follows:

Establish a listener, known as a store observer, to watch local changes so you can respond immediately; for example, to observe updates to end-user profiles.

Here is how you set up a store observer in Ditto:

​

Updating

Modify fields within one or more documents in your local Ditto store.

For example, executing an UPDATE operation within the cars collection, changing the color to 'blue' and the mileage to 3001 in documents where the _id field is '123', as follows:

​

Deleting

Call the Evict method to clear one or more documents from the local Ditto store. Once invoked, the documents are no longer accessible by local queries, however, they remain accessible from other peers connected in the mesh.&

The following snippet shows how to write a basic EVICT operation to purge the document with an _id field of '123' from the local Ditto store:&

​

ditto.store.registerObserver

Set up store observers to monitor changes to data in the local Ditto store based on specified query criteria.

​

Sync Operations

Establish sync subscriptions to monitor changes to the data you’re interested in. The queries defining your data criteria run on all remote stores subscribed to the same sync subscription.&

When you subscribe to a collection, the subscription query is distributed to all relevant stores in the network. Each store executes the query locally and sends updates to the subscriber whenever changes match the subscription criteria.

​

ditto.sync.registerSubscription

Following is a graphic illustrating how to set up a sync subscription to monitor changes to the data you’re interested in.&

First, you initiate a subscription request from your local Ditto store to the remote peer.

Then, when the data you’re subscribed to changes on the remote peer, such as updates, deletions, or insertions, the remote peer automatically syncs the delta changes to your local Ditto store over the mesh.

Combining store observers with your sync subscriptions ensures that you receive updates from remote stores, enabling you to quickly respond in your app:

​

Presence Operations

As Ditto establishes connections and forms a mesh network using all available network transports on a device, each peer discovered in the mesh creates a presence graph through advertising and forming connections. The presence graph is a data structure representing the current state of the mesh from a specific peer’s point of view.&

With Ditto’s Presence API, you can implement the following end-user functionality:

• Intercept and decide to accept or reject incoming connection requests.
• Retrieve and view the presence graph providing near realtime status of connected peers and network resources.
• Retrieve and view your local device’s peer key or the peer key identifying a specific remote peer device.
• Input and view personal information about yourself, such as name and role.
• View the personal information set by remote peers.

​

Patterns

This topic provides an overview of lazy-load retrieval, memory management, ping-pong effect, and utilizing a MAP to sync concurrent changes.&

Understanding these patterns will help you make informed design decisions in your app.&

​

Lazy-Load Retrieval

To improve performance, instead of storing a file that encodes large amounts of binary data within a document, consider storing a reference to it in a separate, explicitly fetched object (token) known as an ATTACHMENT.&

With the ATTACHMENT data type, you can implement lazy loading. Lazy loading is when you delay retrieval until necessary rather than aggressively fetching the data in anticipation of hypothetical future use. This “on-demand” retrieval pattern enhances performance by optimizing resource usage.&

​

Memory Management

Data storage management is essential for preventing unnecessary resource usage, which affects sync performance, battery life, and overall end-user experience.&

Eviction is important for use cases like cabin crew apps where data from the last flight is not needed on the next flight.

​

Ping-Pong Effect

The best approach to handle conflicts that result from two peers making concurrent offline edits and then later rejoining online depends on your specific requirements and use case.

Following is an overview of best approaches for handling concurrency conflicts:

• Resolving concurrency conflicts — If you want to give priority to the “latest” change, use a register. Ditto’s register type use Last-Write-Wins semantics so the value written last always becomes the current value.
• Auditing concurrency conflicts — If you want to keep track of the changes made by different peers over time, use the map type to model your list of operations. Each write operation is independently tracked as a field-value pair, with the field representing the unique identifier and the value storing only the specific changes made by a given peer.
• Prompting end users to choose — If you want your end users to resolve concurrency conflicts instead of Ditto, use the map type inside of your document and prompt end users to select the value to replicate.

​

Using a MAP for Concurrent Updates

Imagine a scenario in which two Ditto stores, peer A and peer B, have the following document:

{

  "_id": "abc123",

  "color": "red",

  "make": "Toyota",

  "mileage": 160000,

  "inspections": "<very large map>"

}

Peer A calls the Upsert method to change the field-value color:red to color:blue:

At the same time, peer B calls the Update method to change the value of the mileage field:&

When the changes replicate across the distributed peers, both changes merge resulting in both peer A and peer B Ditto stores having a mileage increment of 200 and the color change to blue:

​

Antipatterns

Following is a table summarizing the risk associated with improperly implemented patterns:

• Potential resource contention from simultaneous requests | | Memory management | Memory leaks that lead to increased memory consumption over time | | Ping-pong effect | * Excessive network traffic or processing overhead
  * Reduced system responsiveness if components engage in a cycle of redundant operations | | Using a MAP for sync | Race conditions causing data inconsistency |