Sync Overview
Data consistency and integrity is the foundation of any data management system, particularly in distributed systems such as Ditto.
This article offers a brief overview of how interconnected peers communicate and collaborate quickly, continuously, and reliably across the system, even in the presence of network failures.
For comprehensive information, see the Platform Manual:
Data in Ditto is delta encoded. That is, when peer-to-peer syncing, instead of transmitting entire datasets, only changes (delta) are retrieved, sent, or locally stored.
The key advantages to Ditto's delta-based approach include:
- Highly frequent and efficient data transmission — Avoiding redundant data transmission and reducing bandwidth consumption optimizes network performance.
- Consistency without centralized coordination — Distributed peers do not need to reach consensus to converge and merge a single value without conflict.
Data synchronization, or "sync" for short, refers broadly to the practice of ensuring that peers always have the most up-to-date and accurate information.
However, data sync goes beyond basic updates and involves sophisticated techniques like partitioning and replication:
- Partitioning is a strategy that involves dividing the data into smaller, more manageable pieces called partitions.
- These partitions are then asynchronously copied across connected Small Peers. That is, the copies of the data, or replicas, are distributed across separate storage locations running locally within an end-user's physical environment, such as a mobile device.
Partitioning and replication are central to ensuring rapid and dependable data access for your end users.
Ditto incorporates a multiplexer inside the peer-to-peer mesh network to facilitate the data replication process.
Developed by Ditto, the multiplexer is an intelligent replication machine that seamlessly switches between active transport types as needed, without duplicating data.
In addition, the multiplexer breaks data packets down into small fragments and then, once received on the other side, reassembles them.
When the transports collaborate in parallel and the multiplexer automatically switches between them to establish the most optimal connection, these diverse transport types collectively form what is known as the rainbow connection — each color of the rainbow symbolizes a different transport type:
In Ditto's decentralized architecture, data propagates asynchronously among offline peers, eliminating the need for a centralized authority like a central server to validate and merge database operations.
This decentralized model, however, presents the possibility of concurrency conflicts. A concurrency conflict happens when multiple end users make changes to the same data items at the same time. When merging these changes, any inconsistencies collide.
To resolve these concurrency conflicts, as well as enable optimizations like delta-based replication, Ditto harnesses the power of conflict-free replicated data type (CRDT) technology. CRDT is an advanced class of data type designed to manage and replicate data changes in a way that allow multiple distributed peers to make updates concurrently without the need to reach consensus to form a single meaningful value for merge.
For an overview of the different CRDTs that Ditto offers, see Data Structures and Types.
Ditto’s peer-to-peer network architecture utilizes the following diverse set of communication transport technologies and communication channels to achieve seamless, offline-first connections between devices:
By leveraging a diverse set of technologies, Ditto supports a variety of use cases, devices, and environments.
By default, peers use all available transports available to their device. For instructions on how to enable and disable transports, see the Platform Manual > Transports > Configuring Transports.
Ditto utilizes both traditional Wi-Fi and Bluetooth Low Energy (LE) to maintain a continuous mesh network of Transmission Control Protocol (TCP) connections.
Bluetooth LE technology forms low-powered and high-distance connections between devices, making it highly performant in offline scenarios when replicating small amounts of data.
In the event of a Wi-Fi network disruption, such as a router getting disconnected, Ditto does not automatically switch transports to establish an independent Bluetooth LE connection as a fallback.
Instead, Ditto actively works to maintain the previously established Wi-Fi and Bluetooth LE connections.
The following video provides a brief visual overview of how Ditto uses Bluetooth LE to form low-powered high-distance connections between devices.
The distances and bandwidth limitations and capacities illustrated in the video are approximations and vary depending on the Bluetooth® firmware installed on the device.
For detailed information, see Platform Manual > Bluetooth Low Energy.
Available on most devices, standard peer-to-peer Wi-Fi enables direct, point-to-point connections between peers without requiring traditional network infrastructure like a centralized server, router, or access point.
In addition to standard peer-to-peer Wi-Fi, Ditto incorporates a customized multiplexer to increase frequency, speed, and efficiency of data transmission between peers connected in the mesh network.
The following table provides an overview of the Wi-Fi technologies Ditto supports:
Platform | Ditto-Supported Technology |
Apple | Apple Wireless Direct Link (AWDL) |
Android | Wi-Fi Aware |
Windows | Wi-Fi Aware |
The following video illustrates Ditto’s sophisticated and decentralized approach to peer-to-peer Wi-Fi connections between distributed peers connected in the mesh network:
The maximum distance of a connection can vary depending on the specific peer-to-peer transport type being used.
For detailed information, see Platform Manual > Peer-to-Peer Wi-Fi.
When devices are connected over the same Wi-Fi access point or through other means like an Ethernet cable, they can take advantage of a local area network (LAN), if available, to communicate directly with each other without requiring internet access.
A LAN is an interconnected network made up of devices that are physically near each other, such as a home, enterprise, or college campus.
For detailed information, see Platform Manual > Local Area Network.
Peers use WebSocket connections to enable realtime, bidirectional sync between Small Peers and the optional cloud deployment, the Big Peer.
Before you can establish a connection with the Big Peer using a WebSocket, you need to specify an IP/PORT. For instructions, see Platform Manual > Configuring Transports.
The defining attributes of a WebSocket include:
- Requires an internet connection to establish a peer-to-peer communication.
- Unlike the other peer-to-peer transport types, does not automatically discover other peers physically nearby.
- WebSocket connections can be made over a Transmission Control Protocol (TCP) or Local Area Network (LAN).
For detailed information, see Platform Manual > WebSockets.
