JavaScript | Web and Node.js | React Native
4.11 - DQL & Legacy Compatibility
With 4.11 we are excited to introduce legacy compatibility for DQL, which allows developers to use the new DQL APIs while maintaining compatibility with existing code and data. This capability, called STRICT MODE, is designed to ease the transition to the new APIs and ensure that existing applications continue to function as expected.
DQL Strict Mode
DQL Strict Mode provides control over how DQL statements are evaluated and how data types are inferred. When disabled, it offers more flexibility in working with documents without requiring explicit collection definitions.
DQL Strict ModeWith strict mode disabled (DQL_STRICT_MODE=false), Ditto is more flexible and will infer the CRDT type based on the document’s shape:
- Objects are treated as CRDT maps
- Scalars and arrays are treated as Registers
- Counters and attachments are inferred from operations
- Collection definitions are no longer required
Setting DQL Strict Mode
DQL Differ
DQL 4.11 introduces a new Differ API for calculating diffs between query results.
This API can be used with a store observer to understand how its results change
over successive callbacks. The Differ API provides a way to track changes in
the data and can be used to optimize data synchronization and conflict resolution.
DQL Transactions
DQL 4.11 introduces support for transactions. Transactions can be used to group multiple DQL statements together, ensuring that either all of the statements are executed successfully or none of them are. This is particularly useful in scenarios where multiple updates or inserts need to be made to the database, and you want to ensure that either all of them are applied or none of them are.
Transactions API documentation.
Advanced DQL Features
DQL 4.11 introduces numerous advanced querying and manipulation features:
| Feature | Description | Example |
|---|---|---|
UNSET clause | Remove fields from documents | UPDATE orders UNSET items.abc WHERE _id = 'my-id' |
| CAST function | Explicitly convert between data types | SELECT * FROM products WHERE CAST(price AS INT) > 10 |
| USE IDS clause | Direct document retrieval by ID | SELECT * FROM products USE IDS ['123', '456'] |
| SIMILAR TO | Pattern matching expressions | SELECT * FROM products WHERE name SIMILAR TO '%phone%' |
| Runtime expressions | Use expressions in object/array construction | UPDATE products SET metadata = { 'updated_at': current_timestamp() } |
| Counters | Support for legacy compatible counter type | UPDATE products APPLY in_stock PN_INCREMENT BY 1.0 |
In 4.11, functional operations are denoted with an arrow (->) are deprecated.
Tombstone Reaper
The tombstone reaper is now enabled by default for all SDKs. This feature helps clean up deleted document markers (tombstones) to improve database performance and reduce storage requirements.
The reaper:
- Automatically removes tombstones after a configurable time-to-live (TTL)
- Uses read transactions for efficiency
- Includes random scheduling variations to prevent performance problems
- Tracks the number of tombstones evicted for monitoring
- Remembers previous reap times across restarts
This improves overall system performance by reducing the long-term accumulation of
tombstones when calling DELETE to remove documents.
4.11.0 Common Changelog
Added
Added
- Introduced
DQL_STRICT_MODEas a system parameter to opt-out of requiring collection definitions (#16573) - Added transports_ble_adapter_mac system parameter to allow configuring bluetooth adapter MAC address (#16159)
- Small peers now log the result of OnlinePlayground authentication, including if the supplied shared token was incorrect (#16301)
- A cumulative running total of the number of tombstones ever evicted by the reaper to Small Peer Info (#16424)
- Added system parameters to fine-tune Ditto behavior when it encounters a locked persistence directory (#CORE-479)
- New DQL CAST function to explicitly cast between types (#QE-108)
- Conditional functions for unknowns (#QE-111)
- DQL DATE processing functions (#QE-112)
- New DQL clause USE IDS for direct document retrieval by id (#QE-114)
- Additional string functions (#QE-124)
- SIMILAR TO filter expression (#QE-125)
- PN_COUNTER support (APPLY clause in UPDATE statement) (#QE-127)
- UNSET clause to the UPDATE statement (#QE-128)
- MERGE as an alias for UPDATE in the ON ID CONFLICT clause of an INSERT statement (#QE-132)
- The ability to use runtime-evaluated expressions in object constructs in DQL statements (#QE-133)
- The ability to use runtime-evaluated expressions in array constructs in DQL statements (#QE-134)
- Static expressions in mutators (#QE-139)
- Additional object manipulation DQL scalar functions (#QE-159)
- EXPLAIN for query plan (#QE-105)
- More detailed messages about network problems are logged when authentication fails (#SEC-124)
Changed
Changed
- Use the modern rustls-platform-verifier TLS crate on all platforms rather than rustls-native-certs (#15956)
- The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer (#16237)
- The tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously (#16286)
- The tombstone reaper to be enabled by default for all SDKs (#16350)
- INSERT to now accept either DOCUMENTS or VALUES as the keyword introducing the list of values to insert (#QE-140)
- Date functions accept “local” as a timezone & use the executing peer’s local timezone (#QE-168)
- Unified dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)
Fixed
Fixed
- Fixed issue where user_collection_sync_scopes could not be updated after initial set (#16299)
- Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)
- Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)
Removed
Removed
- Old multihop networking stack and system parameter
network_use_nextgen_multihop_stack(#16370) - TLS certificate compression, which led to incompatibilities (#SDKS-1046)
Performance
Performance
- The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database (#16338)
- Improved speed and efficiency of the tombstone reaping process (#16535)
4.11.0 JavaScript & React Native Specific Changes
Fixed: Creating multiple Ditto instances with the same persistenceDirectory parameter now throws a DittoError with code store/persistence-directory-locked instead of only logging an error. This change prevents constructing a non-functional instance as Ditto does require separate configurations for this parameter when multiple instances are instantiated. (#12336)
Added: Error code store/persistence-directory-locked to the DittoError class. (#12336)
Added: property SmallPeerInfo.syncScope to replace the now deprecated methods SmallPeerInfo.getSyncScope() and Ditto.smallPeerInfo.setSyncScope(). (#15579)
Deprecated: Methods SmallPeerInfo.getSyncScope() and SmallPeerInfo.setSyncScope(), use the SmallPeerInfo.syncScope property instead. (#15579)
Added: Method transaction() on Store allowing a DQL transaction to be performed. (#15800)
Added: Transaction representing an active DQL transaction. (#15800)
Added: TransactionInfo encapsulating information about a transaction. (#15800)
Added: TransactionCompletionAction representing an action to be taken upon completing a transaction: Commit or Rollback. (#15800)
Added: An opt-in Android foreground service which allows Ditto to sync in the background. (#16097)
Added: ditto.observeTransportConditions() API for React Native that will allow developers to be aware transports conditions changes, like shutting down bluetooth. (#16518)
Added: class Diff to represent diffs produced by the Differ. (#16723)
Added: class Differ, which calculates diffs between query results. This can be used with a store observer to understand how its results change over successive callbacks. (#16723)
Added: More detailed messages about network problems are logged when authentication fails (#SEC-124)
Changed: Unify dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)
JavaScript | Web and Node.js | React Native
4.11 - DQL & Legacy Compatibility
With 4.11 we are excited to introduce legacy compatibility for DQL, which allows developers to use the new DQL APIs while maintaining compatibility with existing code and data. This capability, called STRICT MODE, is designed to ease the transition to the new APIs and ensure that existing applications continue to function as expected.
DQL Strict Mode
DQL Strict Mode provides control over how DQL statements are evaluated and how data types are inferred. When disabled, it offers more flexibility in working with documents without requiring explicit collection definitions.
DQL Strict ModeWith strict mode disabled (DQL_STRICT_MODE=false), Ditto is more flexible and will infer the CRDT type based on the document’s shape:
- Objects are treated as CRDT maps
- Scalars and arrays are treated as Registers
- Counters and attachments are inferred from operations
- Collection definitions are no longer required
Setting DQL Strict Mode
DQL Differ
DQL 4.11 introduces a new Differ API for calculating diffs between query results.
This API can be used with a store observer to understand how its results change
over successive callbacks. The Differ API provides a way to track changes in
the data and can be used to optimize data synchronization and conflict resolution.
DQL Transactions
DQL 4.11 introduces support for transactions. Transactions can be used to group multiple DQL statements together, ensuring that either all of the statements are executed successfully or none of them are. This is particularly useful in scenarios where multiple updates or inserts need to be made to the database, and you want to ensure that either all of them are applied or none of them are.
Transactions API documentation.
Advanced DQL Features
DQL 4.11 introduces numerous advanced querying and manipulation features:
| Feature | Description | Example |
|---|---|---|
UNSET clause | Remove fields from documents | UPDATE orders UNSET items.abc WHERE _id = 'my-id' |
| CAST function | Explicitly convert between data types | SELECT * FROM products WHERE CAST(price AS INT) > 10 |
| USE IDS clause | Direct document retrieval by ID | SELECT * FROM products USE IDS ['123', '456'] |
| SIMILAR TO | Pattern matching expressions | SELECT * FROM products WHERE name SIMILAR TO '%phone%' |
| Runtime expressions | Use expressions in object/array construction | UPDATE products SET metadata = { 'updated_at': current_timestamp() } |
| Counters | Support for legacy compatible counter type | UPDATE products APPLY in_stock PN_INCREMENT BY 1.0 |
In 4.11, functional operations are denoted with an arrow (->) are deprecated.
Tombstone Reaper
The tombstone reaper is now enabled by default for all SDKs. This feature helps clean up deleted document markers (tombstones) to improve database performance and reduce storage requirements.
The reaper:
- Automatically removes tombstones after a configurable time-to-live (TTL)
- Uses read transactions for efficiency
- Includes random scheduling variations to prevent performance problems
- Tracks the number of tombstones evicted for monitoring
- Remembers previous reap times across restarts
This improves overall system performance by reducing the long-term accumulation of
tombstones when calling DELETE to remove documents.
4.11.0 Common Changelog
Added
Added
- Introduced
DQL_STRICT_MODEas a system parameter to opt-out of requiring collection definitions (#16573) - Added transports_ble_adapter_mac system parameter to allow configuring bluetooth adapter MAC address (#16159)
- Small peers now log the result of OnlinePlayground authentication, including if the supplied shared token was incorrect (#16301)
- A cumulative running total of the number of tombstones ever evicted by the reaper to Small Peer Info (#16424)
- Added system parameters to fine-tune Ditto behavior when it encounters a locked persistence directory (#CORE-479)
- New DQL CAST function to explicitly cast between types (#QE-108)
- Conditional functions for unknowns (#QE-111)
- DQL DATE processing functions (#QE-112)
- New DQL clause USE IDS for direct document retrieval by id (#QE-114)
- Additional string functions (#QE-124)
- SIMILAR TO filter expression (#QE-125)
- PN_COUNTER support (APPLY clause in UPDATE statement) (#QE-127)
- UNSET clause to the UPDATE statement (#QE-128)
- MERGE as an alias for UPDATE in the ON ID CONFLICT clause of an INSERT statement (#QE-132)
- The ability to use runtime-evaluated expressions in object constructs in DQL statements (#QE-133)
- The ability to use runtime-evaluated expressions in array constructs in DQL statements (#QE-134)
- Static expressions in mutators (#QE-139)
- Additional object manipulation DQL scalar functions (#QE-159)
- EXPLAIN for query plan (#QE-105)
- More detailed messages about network problems are logged when authentication fails (#SEC-124)
Changed
Changed
- Use the modern rustls-platform-verifier TLS crate on all platforms rather than rustls-native-certs (#15956)
- The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer (#16237)
- The tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously (#16286)
- The tombstone reaper to be enabled by default for all SDKs (#16350)
- INSERT to now accept either DOCUMENTS or VALUES as the keyword introducing the list of values to insert (#QE-140)
- Date functions accept “local” as a timezone & use the executing peer’s local timezone (#QE-168)
- Unified dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)
Fixed
Fixed
- Fixed issue where user_collection_sync_scopes could not be updated after initial set (#16299)
- Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)
- Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)
Removed
Removed
- Old multihop networking stack and system parameter
network_use_nextgen_multihop_stack(#16370) - TLS certificate compression, which led to incompatibilities (#SDKS-1046)
Performance
Performance
- The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database (#16338)
- Improved speed and efficiency of the tombstone reaping process (#16535)
4.11.0 JavaScript & React Native Specific Changes
Fixed: Creating multiple Ditto instances with the same persistenceDirectory parameter now throws a DittoError with code store/persistence-directory-locked instead of only logging an error. This change prevents constructing a non-functional instance as Ditto does require separate configurations for this parameter when multiple instances are instantiated. (#12336)
Added: Error code store/persistence-directory-locked to the DittoError class. (#12336)
Added: property SmallPeerInfo.syncScope to replace the now deprecated methods SmallPeerInfo.getSyncScope() and Ditto.smallPeerInfo.setSyncScope(). (#15579)
Deprecated: Methods SmallPeerInfo.getSyncScope() and SmallPeerInfo.setSyncScope(), use the SmallPeerInfo.syncScope property instead. (#15579)
Added: Method transaction() on Store allowing a DQL transaction to be performed. (#15800)
Added: Transaction representing an active DQL transaction. (#15800)
Added: TransactionInfo encapsulating information about a transaction. (#15800)
Added: TransactionCompletionAction representing an action to be taken upon completing a transaction: Commit or Rollback. (#15800)
Added: An opt-in Android foreground service which allows Ditto to sync in the background. (#16097)
Added: ditto.observeTransportConditions() API for React Native that will allow developers to be aware transports conditions changes, like shutting down bluetooth. (#16518)
Added: class Diff to represent diffs produced by the Differ. (#16723)
Added: class Differ, which calculates diffs between query results. This can be used with a store observer to understand how its results change over successive callbacks. (#16723)
Added: More detailed messages about network problems are logged when authentication fails (#SEC-124)
Changed: Unify dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)