MAP
An alternative approach to using a JSON object as a REGISTER to represent and organize data in a hierarchical structure within a document is to use a MAP.
Similar to a JSON object, the MAP data type in DQL forms a tree-like parent-child relationship within a document's dataset.
However, unlike a JSON object which is limited to scalar subtypes, functions as a single value, and enforces a last-write-wins merge strategy, a MAP allows:
- Nesting of various DQL data types:REGISTER, MAP, and ATTACHMENT
- Independent modification of each key-value pair
- Utilization of an add-wins merge strategy for conflict resolution. With the add-wins approach, all conflicting changes are retained rather than converging on only one
The response structure of a MAP is a JSON-like object; however, each field in a MAP has a corresponding data type. This means that each field within a MAP has an independent merge definition the same as fields on the top-level document.
The MAP type is useful when you want to create a list of non-conflicting items and update items over time within a document.
The following snippet demonstrates a Ditto document with an embedded MAP my_map:
The MAP type uses the add-wins approach to resolving concurrency conflicts.
So instead of choosing a single definitive value to merge and distribute across peers like the last-write-wins strategy utilized by the REGISTER and ATTACHMENT data types, choose them all — even if the data is already contained within the MAP object.
The REGISTER is the default data type in Ditto; therefore, there is no need for explicit declaration — it's implicitly assumed.
However, to use a MAP (and ATTACHMENT) data type, you must explicitly declare it within your DQL statement using Type Definition syntax. A Type Definition is a description that informs Ditto that the given set of key-value pairs embedded in the document as a nested structure is of the non-REGISTER type and has specific characteristics in how it handles data.
To declare a data type a MAP, you use the COLLECTION identifier prefix, followed by the collection name enclosed in parentheses, along with a type definition. All done inline as follows:
For more information and how-to instructions, see Ditto Query Language > Types and Definitions.
There are two ways to create a MAP structure:
- In an INSERT operation — Create a new document and nest it with this set of fields. (See Inserting to Create a MAP)
- In an UPDATE operation — If the MAP does not already exist, create it. (See Updating to Create a MAP)
When you need to represent a highly-complex data structure in a MAP, consider embedding another MAP. Embedding a MAP within a MAP establishes an additional hierarchy. There are The decision to use deeply embedded MAPS in a single document or opt for a flat model instead depends on your specific requirements, relationships between data, and tolerance for certain tradeoffs.
The flat model is a simple, non‑embedded structure in which you spread your data across multiple separate documents.
When inserting a new document in Ditto, you can define a field as a MAP and include the structure of key-value pairs nested within it — a two-in-one approach. For example:
Another method to creating a MAP within a document is to perform an UPDATE operation.
In the UPDATE approach, you set the structure of key-value pairs and specify the document ID for storage.
For example, this statement creates the MAP structure — unless it already exists — setting a single key-value pair of color is red within the properties MAP for the document with the ID 123 in the cars collection:
For more information and how-to instructions, see Ditto Query Language > CREATE.
MAPS can be updated with specific DQL arrow operator syntax -> (...). This syntax allows for multiple child fields to be edited on a single MAP.
For more information and how-to instructions, see Ditto Query Language > Types and Definitions > MAP Operations.
If you need to represent and organize highly complex data in a hierarchical structure, consider embedding a MAP within another MAP to establish a parent-child relationship within a document as follows:
The MAP data type can be marked as deleted using the tombstone() functional operation.
When a MAP is tombstoned, a metadata property is internally set on the field and all the data within the MAP is removed. This is done by iterating through all the fields in the MAP and calling tombstone on those fields. For REGISTER or ATTACHMENT data types within the MAP this will result in their values also being deleted.
For more information and how-to instructions, see Ditto Query Language > Types and Definitions > Tombstone Operation.
The MAP type has an add-wins merge behavior. This means that the adding or updating of a field always wins over the removal (tombstone) of the same field when performing a merge across two peers.
If multiple fields are being added across Small Peer devices, the result will be a MAP with all added fields. Because each field in a MAP has its own data type, the merge strategy for that field will depend on the data type of that field.
In the following example, USER 1 inserts item1 and USER 2 inserts item2 to an items MAP:
The resulting document contains an items MAP with both item1 and item2:
The nested JSON object, functioning as a REGISTER, closely resembles a nested MAP structure within a document. However, their merge strategies and supported types for nested values differ significantly.
Understanding these distinctions is crucial when selecting the appropriate data type for expressing your data structure.
For example, consider a scenario in which you have a document with a properties field of the REGISTER data type storing information about a car in an embedded JSON object. You want to change only the color field for a car identified by document ID 123 to blue.
Since a REGISTER acts as a single object, to update just the color, you'd need to update the entire object by providing the entire set of nested key-value pairs, including the modified color , as follows:
However, if the properties field was of the MAP data type instead, to update just the color without altering the other nested key-value pairs in the set, you would simply set the properties.color field to blue as follows:
If a tombstone and modification (or INSERT) occur at the same time across two peers, the result is dependent on the casual order of changes.
Modifications to the value of fields within a MAP do not count as modifications to the MAP itself.
Following is a truth table showing "if a MAP is deleted" based on "if the peers are connected" and "when the tombstone operation occurs relative to the modification."
PEER Status | Tombstone First | Tombstone Last |
Connected | FALSE | TRUE |
Disconnected | FALSE | FALSE |
Syncing large documents can significantly impact sync performance.
Caution is advised when handling a deeply embedded document or a very large document. If you have a deeply embedded or large document object, consider using an ATTACHMENT instead of a regular document.
When internet connection is limited or when using only Bluetooth to sync across connected peers, the process of syncing documents that contain large amounts of data is slow. For instance, when using only Bluetooth Low Energy (LE) to sync a document of a typical size, the rate of replication is 20 KB per second maximum. Given this, a document of 250 KB or larger may require 10 seconds or more to sync for the first time between Small Peer devices.
A slow replication rate results in a loading spinner being displayed to your end users until the sync process fully completes. This is due to the callback being unable to render the returned data; the end-to-end sync process requires that documents be broken down into smaller parts before being synced across the mesh, and then, until the client receives all of the smaller parts and reconstructs them, the document is not returned.
Instead of using a single document to encode all of your large dataset, use a series of smaller documents.
Note that if a document exceeds 250 KB in size, a stdout warning prints, and any documents larger than 5 MB will not sync to other peers.