​

4.11.1 Common Changes

No Specific Changes

​

4.11 - DQL & Legacy Compatibility

With 4.11 we are excited to introduce legacy compatibility for DQL, which allows developers to use the new DQL APIs while maintaining compatibility with existing code and data. This capability, called STRICT MODE, is designed to ease the transition to the new APIs and ensure that existing applications continue to function as expected.

​

DQL Strict Mode

DQL Strict Mode provides control over how DQL statements are evaluated and how data types are inferred. When disabled, it offers more flexibility in working with documents without requiring explicit collection definitions.

With strict mode disabled (DQL_STRICT_MODE=false), Ditto is more flexible and will infer the CRDT type based on the document’s shape:

• Objects are treated as CRDT maps
• Scalars and arrays are treated as Registers
• Counters and attachments are inferred from operations
• Collection definitions are no longer required

Setting DQL Strict Mode

ALTER SYSTEM SET DQL_STRICT_MODE = false

​

DQL Differ

DQL 4.11 introduces a new Differ API for calculating diffs between query results. This API can be used with a store observer to understand how its results change over successive callbacks. The Differ API provides a way to track changes in the data and can be used to optimize data synchronization and conflict resolution.

Differ API documentation.

​

DQL Transactions

DQL 4.11 introduces support for transactions. Transactions can be used to group multiple DQL statements together, ensuring that either all of the statements are executed successfully or none of them are. This is particularly useful in scenarios where multiple updates or inserts need to be made to the database, and you want to ensure that either all of them are applied or none of them are.

Transactions API documentation.

​

Advanced DQL Features

DQL 4.11 introduces numerous advanced querying and manipulation features:

​

Tombstone Reaper

The tombstone reaper is now enabled by default for all SDKs. This feature helps clean up deleted document markers (tombstones) to improve database performance and reduce storage requirements.

The reaper:

• Automatically removes tombstones after a configurable time-to-live (TTL)
• Uses read transactions for efficiency
• Includes random scheduling variations to prevent performance problems
• Tracks the number of tombstones evicted for monitoring
• Remembers previous reap times across restarts

This improves overall system performance by reducing the long-term accumulation of tombstones when calling DELETE to remove documents.

​

4.11.0 Common Changelog

​

Added

• Introduced DQL_STRICT_MODE as a system parameter to opt-out of requiring collection definitions (#16573)
• Added transports_ble_adapter_mac system parameter to allow configuring bluetooth adapter MAC address (#16159)
• Small peers now log the result of OnlinePlayground authentication, including if the supplied shared token was incorrect (#16301)
• A cumulative running total of the number of tombstones ever evicted by the reaper to Small Peer Info (#16424)
• Added system parameters to fine-tune Ditto behavior when it encounters a locked persistence directory (#CORE-479)
• New DQL CAST function to explicitly cast between types (#QE-108)
• Conditional functions for unknowns (#QE-111)
• DQL DATE processing functions (#QE-112)
• New DQL clause USE IDS for direct document retrieval by id (#QE-114)
• Additional string functions (#QE-124)
• SIMILAR TO filter expression (#QE-125)
• PN_COUNTER support (APPLY clause in UPDATE statement) (#QE-127)
• UNSET clause to the UPDATE statement (#QE-128)
• MERGE as an alias for UPDATE in the ON ID CONFLICT clause of an INSERT statement (#QE-132)
• The ability to use runtime-evaluated expressions in object constructs in DQL statements (#QE-133)
• The ability to use runtime-evaluated expressions in array constructs in DQL statements (#QE-134)
• Static expressions in mutators (#QE-139)
• Additional object manipulation DQL scalar functions (#QE-159)
• EXPLAIN for query plan (#QE-105)
• More detailed messages about network problems are logged when authentication fails (#SEC-124)

​

Changed

• Use the modern rustls-platform-verifier TLS crate on all platforms rather than rustls-native-certs (#15956)
• The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer (#16237)
• The tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously (#16286)
• The tombstone reaper to be enabled by default for all SDKs (#16350)
• INSERT to now accept either DOCUMENTS or VALUES as the keyword introducing the list of values to insert (#QE-140)
• Date functions accept “local” as a timezone & use the executing peer’s local timezone (#QE-168)
• Unified dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)

​

Fixed

• Fixed issue where user_collection_sync_scopes could not be updated after initial set (#16299)
• Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)
• Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)

​

Removed

• Old multihop networking stack and system parameter network_use_nextgen_multihop_stack (#16370)
• TLS certificate compression, which led to incompatibilities (#SDKS-1046)

​

Performance

• The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database (#16338)
• Improved speed and efficiency of the tombstone reaping process (#16535)

​

4.11.0 JavaScript & React Native Specific Changes

Fixed: Creating multiple Ditto instances with the same persistenceDirectory parameter now throws a DittoError with code store/persistence-directory-locked instead of only logging an error. This change prevents constructing a non-functional instance as Ditto does require separate configurations for this parameter when multiple instances are instantiated. (#12336)

Added: Error code store/persistence-directory-locked to the DittoError class. (#12336)

Added: property SmallPeerInfo.syncScope to replace the now deprecated methods SmallPeerInfo.getSyncScope() and Ditto.smallPeerInfo.setSyncScope(). (#15579)

Deprecated: Methods SmallPeerInfo.getSyncScope() and SmallPeerInfo.setSyncScope(), use the SmallPeerInfo.syncScope property instead. (#15579)

Added: Method transaction() on Store allowing a DQL transaction to be performed. (#15800)

Added: Transaction representing an active DQL transaction. (#15800)

Added: TransactionInfo encapsulating information about a transaction. (#15800)

Added: TransactionCompletionAction representing an action to be taken upon completing a transaction: Commit or Rollback. (#15800)

Added: An opt-in Android foreground service which allows Ditto to sync in the background. (#16097)

Added: ditto.observeTransportConditions() API for React Native that will allow developers to be aware transports conditions changes, like shutting down bluetooth. (#16518)

Added: class Diff to represent diffs produced by the Differ. (#16723)

Added: class Differ, which calculates diffs between query results. This can be used with a store observer to understand how its results change over successive callbacks. (#16723)

Added: More detailed messages about network problems are logged when authentication fails (#SEC-124)

Changed: Unify dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)

​

4.10.5 JavaScript & React Native Specific Changes

Fixed: Windows devices failed to connect to Android on LAN (#17199)

Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)

​

4.10.5 Common Changes

No Specific Changes

​

4.10.2 Common Changes

Removed: zlib certificate compression for small peer TLS 1.3 session negotiation, which caused incompatibilities (#1005)

​

4.10.1 Common Changes

Changed: The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer. (#16237)

Changed: the tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously. (#16286)

Fixed: Unable to update user_collection_sync_scopes after initial set bug (#16299)

Performance: The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database. (#16338)

Changed: The tombstone reaper to be enabled by default for all SDKs. (#16350)

​

4.10.0 New Capabilities

​

Support for React Native Expo CNG

Ditto’s React Native SDK now supports Expo for iOS & Android native development.

​

Small Peer User Collection Sync Scopes

User Collection Sync Scopes let you control how data from each user collection is shared with other connected Peers. This is useful when you want to fine-tune which collections do not sync, which collections sync with only Ditto Server and which collections sync only with connected Small Peer devices.

Sync scopes can be used to control how documents sync, reduce unnecessary data transfer, and improve performance.

​

Details

DQL Syntax for setting a Sync Scope

ALTER SYSTEM SET USER_COLLECTION_SYNC_SCOPES = {
    collection_1: "BigPeerOnly",
    collection_2: "SmallPeersOnly",
    collection_3: "LocalPeerOnly"
}

Example Setting A Sync Scope in the SDKs

In the example below we’ll set the collection local_mesh_orders to only sync with other small peers in the mesh and not with Ditto Serverusing the SmallPeersOnly sync scope.

var syncScopes = [
"local_mesh_orders": "SmallPeersOnly"
];
await ditto.store.execute(
    query: "ALTER SYSTEM SET USER_COLLECTION_SYNC_SCOPES = :syncScopes",
    arguments: ["syncScopes": syncScopes]);

​

Store Observers Run Multi-Threaded By Default

Ditto store observers (e.g ditto.store.registerObserver(...)) will now run multi-threaded on all platforms (except for in Web Browsers) by default. Prior to 4.10.0 all store observers would run on the same thread which could lead to performance issues for applications with multiple store observers. Users may see performance improvements on applications using more than one store observer, especially during high-volume usage.

Ditto guarantees order sequencing for a single observer. Because observers now run in parallel there is not a guarantee for order sequencing between observers. This behavior change should not have an impact on your application. If you have concerns or questions about this behavior change reach out to Ditto’s Customer Support.

To reset Ditto to only use a single thread for all Ditto observers use the following system parameter. For instruction on how to set a system parameter in your specific SDK see SDK>Customizing System Settings

ALTER SYSTEM SET STORE_OBSERVERS_NUM_THREADS = 1

​

DQL Support for LIKE operator

The LIKE operator can now be used for pattern matching of string values.

Example

SELECT * FROM my_collection
WHERE some_field LIKE 'abc%'

​

DQL New Objects operators

DQL now supports two new methods for querying over the keys and values in an object.

• object_keys(object): Returns an array with all the keys in the given object.
• object_values(object): Returns an array with all the values in object.

​

DQL LIMIT/ OFFSET / ORDER BY support for mutation operators

Users can now use LIMIT, OFFSET and ORDER BY in EVICT and UPDATE statements.

DQL Example

EVICT FROM users
WHERE status = 'inactive'
LIMIT 10;

​

4.10.0 JavaScript & React Native Specific Changelog

• Add support for React Native Expo CNG
• Repeated reads of a document using an ID-specific transaction cursor of the query builder API now reflect changes made in the same transaction. (SDKS-245)
• Errors regarding missing dittoffi_query_result_item_free() and ditto_clear_presence_callback() functions in React Native. (#SDKS-555)
• Interface IdentityOnlinePlayground to add property customAuthURL and allow authenticating with a custom authentication service instead of the Ditto cloud. (#15904)

​

4.10.0 Common Changelog

​

Added

• In DQL, an element of an array can be accessed by expression. (feat: Disallow double-quoted identifiers inside of new operators #12529)

• DQL field and element selection (DQL: Support field and element selection. #13920)

• Full support for DQL virtual collections, and the system:dual virtual collection (parser changes for virtual collections #14954)

• On Android, logs now indicate if a missing permission is not declared in an app’s AndroidManifest.xml. (Clarify Android missing permission messages based on manifest presence #15094)

• LIKE DQL operator. (#QE-104)

• Added object_keys and object_values scalar DQL functions. (#QE-73)

• A new system parameter network_use_nextgen_multihop_stack to switch between our existing and experimental next generation network stacks (#15362)

• Enabled zlib certificate compression (RFC8879) for small peer TLS 1.3 session negotiation ([security] enable TLS certificate compression #15801)

• A system parameter to selectively disable the BLE peripheral (server) transport. (New System Parameter: transports_ble_peripheral_is_enabled #15802)

• A STORE_OBSERVERS_NUM_THREADS system parameter to tweak the number of threads used by our store observer runtime to dispatch the sdk callback invocations. (Allow different live queries to be called in parallel #15814)

• The ability to configure a TTL for tombstones in the small peer store, which can be cleaned up via periodic sweeps from the new “reaper” module. (Finish porting the prototype tombstone reaping implementation into main #15886)

• Support for the new DQL DELETE statement on small peers. (Stabilize the DQL DELETE statement on SP #15899)

• A new Small Peer Info section named “reaper” and a field named “last_reap_time” that indicates the last time the tombstone reaper successfully completed its cleanup of expired tombstones. (Add small-peer-info field to track last completed reaper timestamp #16000)

• Add support for LIMIT, OFFSET and ORDER BY for mutation statements iin DQL (feat(query/dql): Support ORDER BY, LIMIT, OFFSET in mutation statements #16020)

• Added support for permanent system wide document deletes with the DELETE keyword in DQL for all Small Peer SDKs. This change requires a coordinated release with Ditto Server to ensure data is completely removed from the system. The Ditto Server change will be rolling out over the coming month and Ditto will be making a larger announcement upon release. For usage on deletes prior to the larger announcement see the following documents. Note that these documents are only accessible via their direct links and will not appear in the navigation bar just yet.

  • DQL DELETE Syntax
  • Removing Documents with Deletes

  For support on using deletes in your application reach out to Ditto Customer Support to get started.

​

Fixed

• A rare scenario where stopping sync with in-progress TCP connections could result in a crash. (#TRANS-303)
• If Ditto’s storage is cloned to another device (not recommended), clones which connect to each other will automatically create a new Peer Key and reauthenticate so that sync can proceed (Trigger reset peer identity when another peer clearly has the same Peer Key, add integration test #14653)
• On Android, missing system permissions will no longer crash Ditto. (Check for normal permissions in DittoSyncPermissions #15233)
• Fixed initialisation failing for the Linux Bluetooth transport due to invalid dbus messages (Linux BLE baseline bugfixes #15717)
• Fix OldQL and DQL comparison between large floats and integers returning an arbitrary result (Fix OldQL and DQL comparison between large floats and integers #15949)
• Reduced duplicate notifications for authentication state changes. (auth client: validity_update only on changes #15981)

​

Changed

• The link_enabled system parameter has been renamed to network_enable_multihop ([network] Add configurability for switching between legacy & next gen networking stacks #15362)
• Improve authentication flow over bad connections (Auth client/server improvements around challenge tokens #15430)
• Store observers are now invoked in parallel by default. (STORE_OBSERVERS_NUM_THREADS to default to 0 = num_cpus #16064)
• Synchronization protocol enhanced to reduce re-transmission of document data and metadata after eviction. (#DS-188)

​

4.9.4 Common Changes

Added: new system parameter which defines the number of threads used by the store observer runtime to dispatch the sdk callback invocations. For more information reach out to Ditto’s Support (#15814)

​

4.9.3 JavaScript & React Native Specific Changes

Fixed: renamed a misspelled property in the Peer type to the correct spelling dittoSdkVersion from the previous, incorrect spelling dittoSDKVersion. (#SDKS-176)

​

4.9.3 Common Changes

Changed: Added more actionable information to common connection handshake issues. (#15298)

Fixed: Fixed: Some Android devices could fail to export logs to the portal due to a file permissions error (#15582)

No JavaScript Specific Changes

​

4.9.1 JavaScript & React Native Specific Changes

Changed: Enhanced the efficiency of our memory management processes on React Native. (#15126)

Changed: Optimized JSI runtime object management for greater efficiency and responsiveness on React Native. (#15170)

​

4.9.1 Common Changes

Fixed: iOS 18 crashes related to Bluetooth LE are now resolved. (#15351)

Fixed: Disk usage stats are now reported in small peer info on android devices

​

4.9.0 JavaScript Release Notes

Added: login() method to Authenticator which enables access to userInfo provided by the configured authentication webhook. (#12354)

Added: new methods PendingIDSpecificOperation.updateV2(), and WriteTransactionPendingIDSpecificOperation.updateV2() to replace the deprecated update() methods. These methods do not throw when the document with the given ID can not be found, and call the given closure with null instead. Additionally, these methods throw when given an unsupported async closure argument. (#14504)

Added: (React Native) React Native apps can now also set a logging callback using Logger.setCustomLogCallback(). This API was previously not available in React Native environments. (#14658)

Added: CFBundleShortVersionString inside the React Native framework. (#14845)

Changed: Error messages and documentation for APIs not supported in React Native environments. Some parts of the SDK’s API, in particular the original query builder APIs, are not supported and throw errors when called from a React Native app. In these cases, error message wording and documentation has been improved to make it easier to find the right replacement APIs to use instead. (#14612)

Changed**: (React Native)** Transitioned from static to shared libraries to improve load times and reduce overall application size. This change requires updates to build configurations to ensure dynamic linking is correctly handled. See our integration guide for detailed instructions on updating your project settings. (#15018)

Fixed: Passing an unsupported async closure to any cursor operation’s update() method now logs a warning. This includes PendingIDSpecificOperation.update(), PendingCursorOperation.update(), WriteTransactionPendingIDSpecificOperation.update(), and WriteTransactionPendingCursorOperation.update(). (#14504)

Fixed**: (React Native)** Sporadic Android BLE crash when syncing with other peers. (#15018)

Deprecated: Method loginWithToken() from Authenticator. Use the newly added login() method instead. (#12354)

Deprecated: PendingIDSpecificOperation.update(), and WriteTransactionPendingIDSpecificOperation.update() whose behavior did not match their documentation. Use each class’s new updateV2() method instead. (#14504)

Removed:

​

4.9.0 Logging Clarity Improvements

This release comes with a number of logging clarity improvements. The goal of these changes are to make Ditto logs more consumable and actionable for end users. While there are wide ranging changes one of the principal goals of this work is to make our INFO level logs contain the right information to understand the state and health of the Ditto SDK.

• More information is included when physical connections start and end (such as the type of connection and the remote peer’s key)

• Devices that try to connect to themselves now say “device has connected to itself” instead of an error about the remote site ID being the same as the local peer

• Authentication client logs have been made more human readable. In particular:

• Recoverable errors are logged together with information about the recovery actions that are being undertaken

• Several abbreviations have been replaced with human-readable descriptions

• Errors are printed using their descriptions rather than by dumping debug representations

• Abbreviated jargon has been replaced with clearer wording in various log lines

• “phy ended” has become “physical connection ended”

• “error getting char” has become “required Ditto BLE characteristic not found”

• “error getting radio state” has become “error getting Windows Bluetooth radio state”

• Peer events that represent a transport starting or stopping are now logged with the message “transport started”/”transport ended” rather than as a raw peer_event=… log

• Reduced the noisiness of “starting” log lines around Ditto initialization time

• Noise from long peer keys in logs has been reduced in some places by replacing them with their truncated form

• Noise from duplicated peer ID/peer key information in replication logs has been reduced by consolidating the information

• “starting compression and export” is now followed by a “completed export” log at the same level

• Successful SQLite WAL recovery logs were further lowered from INFO to DEBUG

​

4.9.0 Common Changes

Added: regexp_like function in DQL. (#12129)

Added: peer_metadata and identity_service_metadata to the small peer info document (#12477)

Added: WebSocket connections now send X-DITTO-PEERPUBKEY and X-DITTO-ROUTING-HINT headers as part of the client connect handshake on all platforms except wasm32 (#13171)

Added: Small Peer Info now includes information about the current logger configuration (#14317)

Added: Permission rules can be written in DQL. (#14563)

Added: Ditto logs clearly indicate if a device has established a connection to itself. (#14634)

Added: the ability for DQL select statement to produce arbitrary projection lists. (#7979)

Added: local subscription info to small peer info (#9412)

Added: “device_disk_usage” information to small peer info (#9784)

Added: “store” information to small peer info (#9785)

Changed: Internal sync metadata format changed. Peers that downgrade to a previous SDK version after upgrading may require full resynchronization. (#14799)

Changed: Improved (de)serialization performance and reduced its contribution to the library size (#11545)

Changed: In updated replication protocol, verification of a received update file no longer requires both peers to use the same approach to subscription hashing. (#12672)

Changed: Experimental multihop connectivity is more resilient to mesh disruptions (#13648)

Changed: Synchronization protocol enhanced to reduce the amount of redundant data sent from small peers to big peers. (#13737)

Changed: Failed certificate verification due to a lagging device clock now results in a clearer error message (#14035)

Changed: Multicast peer-to-peer transport is no longer deprecated to improve support for environments where mDNS is unavailable (#14188)

Changed: Replication no longer redundantly re-transmits unchanged subscriptions after eviction. (#14290)

Changed: truncated peer keys are now logged in lieu of site IDs. (#14362)

Changed: improved logging in the authentication code to provide more useful error context. (#15038)

Changed: made the small peer “log request” feature much more responsive (#9777)

Fixed: Windows P2P LAN now can connect to IPv6 peers (#11468)

Fixed: In certain rare cases, resetting the same replication session multiple times could cause failure to converge. (#12616)

Fixed: Internal connection TLS caches are now bounded in size. This prevents a long living peer from holding onto more memory than required. (#13801)

Fixed: small peer info “metadata” is now persisted across app launches. (#13987)

Fixed: Bluetooth peers are handled better when disconnections occur, fixing a possible loop where a device might repeatedly try and fail to connect. (#14018)

Fixed: When replication sessions time out, write transactions on the document store may be blocked for a long period. (#14148)

Fixed: Multicast peer-to-peer transport (non-MDNS) is no longer influenced by TCP listen configuration in DittoTransportConfig (#14187)

Fixed: an issue preventing attachment transfers under high document sync load is now resolved. (#14189)

Fixed: A race condition that could cause attachments to unexpectedly crash under heavy load. (#14342)

Fixed: device name no longer incorrectly defaults to “ditto” in small peer info, when a named device first starts syncing (#14674)

Fixed: presence observers are now less-likely to fire when there is no observable change to presence data. (#14807)

Fixed: Logs report blocked_by=“no blocking transaction” when blocking transaction is known (#14966)

Fixed: In-band authentication data is now cleared on logout. (#15039)

Fixed: switching appIDs now invalidates cached authentication material correctly. (#15040)

Fixed: Transport conditions are now reported for a wider range of failures when starting Bluetooth and mDNS transports. (#8729)

Fixed: documentation for the stopSync() method to clarify that the Ditto store can be used locally after stopping sync. (#8849)

Fixed: Auth Client would hang when cached X.509 was expired. (#15110)

Deprecated:

Removed:

​

4.8.4 Common Release Notes

Fixed: Some Android devices could fail to export logs to the portal due to a file permissions error (#15582)

​

4.8.3 React Native Specific Changes

Changed: Error messages and documentation for APIs not supported in React Native environments. Some parts of the SDK’s API, in particular the original query builder APIs, are not supported and throw errors when called from a React Native app. In these cases, error message wording and documentation has been improved to make it easier to find the right replacement APIs to use instead. (#14612)

​

4.8.3 Common Release Notes

Fixed: iOS 18 crashes related to Bluetooth LE are now resolved. (#15351)

Changed: Added more actionable information to common connection handshake issues. (#15298)

​

4.8.2 JavaScript Specific Changes

Fixed: Missing API reference for class UpdateResult. (#14611)

​

4.8.2 Common Changes

Fixed: Setting the minimum log level to WARN or ERROR will now correctly filter out very early logger initialization messages. (#14451)

Fixed: A bug that could prevent an attachment created with the same data on multiple peers from syncing. (#14481)

​

4.8.1 JavaScript Release Notes

Fixed: Prevent race condition that can cause attachment fetchers to fail being cleaned up. (#13961)

Fixed: a memory leak affecting attachments downloaded using the legacy query builder API. (#13819)

Fixed: iOS devices may fail to sync on LAN when the app is backgrounded then foregrounded. (#14150)

​

4.8.1 Common Changes

Added: The presence_use_multihop system parameter can be used to disable replicating presence information to other peers in the mesh, reducing network overhead. Use ONLY with guidance from the Ditto Customer Experience team. (#14128)

Changed: When failing to load document data sync (replication) metadata, Ditto now attempts to clear and regenerate the metadata. This is a behavior change where previously Ditto would fail to start. This serves to prevent problems after SDK version downgrades. (#13111)

Fixed: A rare issue where attachments may not be transferred between two peers.(#14158)

Fixed: On-disk logs are more robust in situations with unexpected Ditto shutdown. (#13723)

The version 4.8.0 SDK release update brings a substantial number of improvements and new features designed to enhance your development experience and provide even more powerful tools for building robust, real-time applications.

​

4.8.0 JavaScript Release Notes

Added: Method exportToFile() to Logger, which exports collected logs to a compressed and JSON-encoded file on the local file system. Logs returned through this method are at DEBUG log level. This new API can be used to do forensic log gathering on a device. Logs are always collected and restricted to a fixed specific size limit to ensure they don’t grow endlessly. Once the size limit has reached new logs replace the oldest logs. #11961

Added: Properties peerKeyString1 and peerKeyString2 on DittoConnection replacing the deprecated properties peer1 and peer2  #12799

Added: Documentation for startSync() and disableSyncWithV3() to include a recommendation for performance improvement. #13098

Performance of initial sync when bootstrapping a new peer can be improved by calling disableSyncWithV3() before calling startSync(). Only do this when all peers in the mesh are known to be running Ditto v4 or higher.

Added: Documentation for property peerMetadata on DittoPeer to add information about how the property behaves over the lifecycle of the DittoPresenceGraph #13479

Changed: DQL methods now use generics for improved type safety and better IDE support. This change affects the Ditto Store’s methods execute(), registerObserver(), registerObserverWithNextSignal(), its observers field, as well as the related types QueryResult, QueryResultItem, StoreObserver, StoreObservationHandler, and StoreObservationHandlerWithSignalNext. In the Sync class, the method registerSubscription() and the subscriptions field are updated, as well as the related SyncSubscription class. Type generics are added in a backwards-compatible manner by providing default types with wide bounds.

Changed:  Ditto.runGarbageCollection() is now an async method, which allows awaiting the completion of manual garbage collection. #8743

Fixed: Errors that occur while running manual garbage collection with Ditto.runGarbageCollection() are now thrown. #8743

Fixed: Ditto logger is now initialized when first constructing a Ditto instance, when it used to be initialized on calling init() in browser environments and on import of the SDK in other environments. This fixes an issue where it was not possible to set the logger’s verbosity level before logging began. #12719

Deprecated: Properties peer1 and peer2 on DittoConnection, please use peerKeyString1 and peerKeyString2 instead #12799

​

4.8.0 Common Changes

Changed: The Logging output format for Ditto SDKs has been changed in 4.8.0 as part of our process to improve readablility and observability across the Ditto platform. Customers should validate the new data format in their existing data ingestion pipelines.

New Logging Schema:

YYYY-MM-DDTHH:MM:SS.ssssss LEVEL TARGET: MESSAGE {field1=value1, field2=value2, …}

• Timestamp (YYYY-MM-DDTHH:MM.ssssss):

• Format: ISO 8601 format (e.g., 2024-08-16T14:23:05.123456Z).

• Description: The exact time when the log entry was generated.

• Log Level (LEVEL):

• Format: One of the following levels— DEBUG, INFO, WARNING, ERROR.

• Description: Indicates the severity or importance of the log entry.

• Target (TARGET):

• Format: A string, representing the internal module/area

• Description: The source of the log entry, often corresponding to the internal module where the error was called. (e.g. ditto_init or peer_event_listener)

• Message (MESSAGE):

• Format: A human-readable string.

• Description: The main content of the log entry.

• Structured Fields ({field1=value1, field2=value2, …}):

• Format: A list of key-value pairs enclosed in curly braces.

• Description: Any additional structured data associated with the log entry. These fields can include custom data that you attach to the log entry using the tracing::info!(field1 = value1, “Message”) syntax.

4.7 and earlier Logging Format Example

[INFO] 2024-08-02T10:25:39.981Z: AuthClient: using cached SiteId; site_id = 16248337916605943930

4.8.0 and later Logging Format Example

2024-08-02T11:32:25.905562+01:00 INFO ditto_init:new_auth_client: ditto_auth: using cached SiteId site_id=16248337916605943930

Added: DQL support for string functions byte_length and char_length #12147

Added: DQL support for inline object and array literal creation #12312

Added: object_length and deserialize_json functions in DQL #12605

Added: DQL INSERT statements support inlining the construction of an object #12761

Added: Remote query capability allows Big Peer to execute read-only queries on Small Peers. Usagne currently internal to Ditto Customer Experience team members #12268

Added: Diagnostics are now provided when collecting Small Peer Info on-disk logs #12080

Added: output version, sdk and build information on startup for easier debugging #12845

Added: Introduced new internal capabilities ensure a large data payload (e.g. a large attachment file) cannot use all the bandwidth and prevent other data from being synced.

Added: mDNS transport can now be configured with a static port for the TCP server using the tcp_server_bind_mdns_server_port system parameter #13442

Changed: New logging format that includes detailed information #10794

Changed: The Ditto logger is now more verbose by default, with a minimum log level of INFO instead of WARNING #11141

Changed: small peer info collection is now enabled by default and its default sync scope is now BigPeerOnly #12709 (Device dashboard will populate information automatically)

Changed: (Experimental) Multihop connections are formed more efficiently, e.g., for Ditto Bus. This is an incompatible protocol change from previous versions #12108, #12623

Changed: Improve performance by avoiding deserializing the entire Document if only part of it is required #10887

Changed: Attachments now sync independently of documents, improving rainbow connection efficiency #11156

Changed: Updating several depdencies for security patching (Patch multiple old packages #12087)

Changed: How network OS capabilities are internally linked into the library to allow for more relaiable and reusable internal logic #12362

Changed: API documentation for execute method to include information about its interaction with sync subscriptions #12365

Changed: Default to creating backup Bluetooth connections when there is an existing WiFi connection (Default mesh_chooser_avoid_redundant_bluetooth to false #13398)

Changed: console logging will now output to STDERR instead of STDOUT ([logs] logging now outputs to STDERR instead of STDOUT #13477)

Fixed: Bytes and Arrays not being always comparable in the legacy query language #10887

Fixed: Legacy query language casting counters and maps with tombstones to the wrong boolean value #10887

Fixed: Lowered the log level of a few SQLite3 notices that were being unnecessarily logged as warnings #12032

Fixed: A crash that can occur if a DQL query contains a multi-byte character #12324

Fixed: An issue with the transports multiplexer that could lead to a hung process. This occured when dealing with large messages spanning multiple chunks (>64KB)  #12428

Fixed: A rare edge case that could slow down data sync betten peers #12527

Fixed: Frequent subscription changes along with eviction calls could cause failure to converge on data leading to inconsistent data state #11431

Fixed: Memory leak in third party library used for peer-internal messaging #12895

Fixed: Transport inactive connection tracking no longer stores identifiers indefinitely. This could lead to a slow growth in memory consumption for long lived applications (~ 1MB/day on transports with high rate, unreliable discovery - such as BLE) #13147

Fixed: Unexpected failures to obtain a certificate for peer-to-peer sync are now logged at ERROR level instead of WARNING #13323

Removed: (Experimental) Ditto no longer supports the experimental Query Overlap Groups feature and support has been removed. This experimental feature has been removed after running a determining a lack of feasability in production scenarios. The team is iterating and plan on interoducing improved capabilitites. #13075

​

ADDED: Support for specifying the mDNS connection ports for Small Peer SDK instances

Allows you to specify a specifc port that can will be used for mDNS announcements and connections on a given device. This can be used for connect with peers across a VLAN while scanning for open ports.

Not setting the config will result in the current default behavior of a random port being assigned.

Setting the config

The config can be set using DQL ALTER SYSTEM and the tcp_server_bind_mdns_server_port config.

ALTER SYSTEM SET tcp_server_bind_mdns_server_port TO 1234

let result = await ditto.store.execute(query: "ALTER SYSTEM SET tcp_server_bind_mdns_server_port TO 1234")

This command should be run using the ditto.store.execute(…) command within your application.

​

FIXED: A finalizer issue that was causing crashes when closing sync subscriptions (#13849, #13732)

Resolves a race condition that could be hit when closing a sync subscription or stopping sync that caused Ditto to crash.

​

IMPROVED: (Linux) Increased mDNS multicast TTL from 1 to 255 to allow for cross-VLAN announcements (#13590)

To support scenarios with cross-VLAN device discovery we increated the TTL to ensure message have enough time to reach their destination and get acknowledged.

​

FIXED: An issue causing a race condition with a Live Query callback. (#13180)

This fix addresses the issue causing a Live Query callback to receive a delta update, known as a diff, before the initial document.

​

FIXED: A compatibility issue between documents created using different CRDT versions. (#13236)

Previously, there was an issue in which two documents on different internal versions could fail to integrate changes created using an older CRDT version.

​

IMPROVED: Logging for batch write transactions. (#13339, #13339)

Due to a race condition caused by the logging system mistakenly identifying the previous transaction as blocking the current one, the logger would inaccurately indicate the current transaction as being blocked by the previous one from an unknown originator.

​

FIXED: An issue in the Small Peer datastore causing slow performance or process termination. (#13395)

This issue was due to an aging dependency with a race condition, which caused thread tasks to get stuck and transactions to be blocked for extended periods.

​

IMPROVED: The Ditto SDKs compilation performance. (#13174)

We’ve optimized compilation by reducing the size of each binary slice of the Ditto SDKs by 19% and fixing incompatibility with PLCrashReporter.

​

ADDED: mesh_chooser_avoid_redundant_bluetooth system parameter. (#12984)

This new system parameter allows greater control over the use of Bluetooth Low Energy (LE) connections in Ditto’s peer-to-peer mesh network. When set to false, Ditto creates Bluetooth LE connections alongside other peer-to-peer connections, serving as a fallback if the primary transport, such as Apple Wireless Direct Link (AWDL) or LAN, fails.

However, before you enable the mesh_chooser_avoid_redundant_bluetooth parameter, consider the following technical tradeoffs:

• Benefits:

  • Provides additional fallback to maintain connection stability when primary connections cease to function.

  • Reduces time to establish new connections since, by having multiple fallback options, Ditto can more quickly re-establish connections.

• Drawbacks:

  • Increases network traffic since using multiple transports results in additional network overhead

  • Using Bluetooth LE alongside other transports increases resource consumption. In the worst case, enabling this setting can result in slower sync performance and battery drain.

To enable the mesh_chooser_avoid_redundant_bluetooth parameter, use the following DQL query: ALTER SYSTEM SET mesh_chooser_avoid_redundant_bluetooth = false

​

CHANGED: The AuthClient by improving the log message for certificate issues returned from the Big Peer Authentication Service. (#13138)

​

FIXED: A performance-related issue affecting bulk eviction operations for documents with attachments. (#12331)

​

ADDED: Improved data compression in the handshake protocol when establishing connections to other peers. (#11264)

This allows peers with large permission sets to establish connections on low bandwidth transports like Bluetooth LE more quickly.

​

FIXED: The HTTP protocol by adding content-length: 0 header compliance in empty POST requests. (#12346)

Before, if you invoked an empty POST request, the cloud load balancer would reject your request with a411 - Length Required error message.

Now you must explicitly state that the body of your POST request is empty by setting content-length: 0 in the header of your HTTP API call.

​

FIXED: Handling of malformed subscription queries in the legacy query builder API by adding validation logic early to ensure invalid queries are identified early and logged. (#12463)

There was an issue where invoking subscribe() on an invalid query caused sync to be silently disabled.

Invalid queries in the subscribe() method are now identified, disabled, and logged as errors.

​

FIXED: Subscription assumption logic so all acknowledgments are processed, regardless of subscription changes. (#12541)

In scenarios where subscriptions rapidly changed back and forth while documents were being concurrently altered, documents would diverge at merge.

We’ve removed subscription logic ignoring acknowledgements based on assumed subscription mismatch and introduced salting functionality to ensure each document is unique for each occurrence of a subscription. This helps differentiate between new and reverted subscription states.

​

FIXED: Sync inefficiency where changing a subscription resulted in unnecessary syncing of redundant update files. (#12967)

To eliminate redundant data from being synced unnecessarily, regardless of whether the remote peer previously acknowledged them, we’ve added validation checks ensuring that only unacknowledged data is synced.

This fix reduces sync update file sizes, leading to more performant session sync updates, particularly in apps consisting of large, stable subscriptions and infrequent document changes.

​

FIXED: Compatibility issues preventing the React Native SDK from working correctly with older versions of Ditto. (#12641)

This fix ensures the Ditto SDK is compatible with versions of React Native lower than 0.72.

​

FIXED: Blank deviceName property on Ditto instances in React Native apps. (#12661)

​

Fixed: Misleading errors related to attachment fetch operations. Now, associated error logs are more accurate and clear. (#12409)

​

Fixed: A specific issue where incorrect sync metadata rollback caused problems with data consistency at merge. (#12422)

​

Fixed: Potential for a crash when closing a Ditto instance immediately following operations such as setPeerMetadata or setPeerMetadataJSON. (#12351)

​

Fixed: Potential crash when closing a Ditto instance after calling Ditto.presence.observe() to register a presence observer. (#12388)

​

Fixed: Premature triggering of asynchronous callback handlers created using observeLocal(). (#12388)

Before this release, there were issues with the timing of callback triggers where the next callback in a live query could trigger before the asynchronous callback handler function had completed its execution.

​

FIXED: The legacy query language error-throwing mechanism to accept invalid field names properly.

After upgrading to version 4.7 of the SDK, you may notice unexpected errors while using the legacy query builder language. These errors are likely due to invalid field names or path expressions previously undetected.

In addition, make sure to start unquoted_string with an underscore (), a letter, or an alphanumeric character. For example, the following equivalent regular expression: [A-Za-z][A-Za-z0-9_]*.

In the legacy query builder language, invalid field names and path expressions now throw explicit errors. Previously these invalid names did not throw and would cause undefined behavior. Note: This new error-throwing behavior applies only to the legacy query builder language; DQL is unaffected by this improvement.

​

REMOVED: The is_connected_to_big_peer field from the Small Peer Info document. (#11883)

​

CHANGED: The small peer info JSON metadata by increasing the depth limit to 64 levels. (#11891)

​

IMPROVED: Transaction contention time for the ATTACHMENT data type, reducing potential timeouts. (#11893)

Before this improvement, concurrent write transactions were long-lived, potentially causing deadlocks and request timeouts. Now you’ll experience more effective handling of multiple attachments being processed at the same time.

​

FIXED: The issue causing sync crashes due to missing update files. (#11952)

With the latest release, Ditto handles unexpected situations or errors in a way that allows it to recover and continue functioning without crashing or causing disruptions.

​

ADDED: Parameters for adjusting certain settings during runtime

This rollout includes new system parameters for the MeshChooser feature for configuring a limited set of variables, allowing you to adjust certain settings during runtime.

​

RESOLVED: The warning message when peers connect is no more. (#11977)

The warning message previously logged when peers connected within the mesh is no longer recorded or printed.

​

IMPROVED: Sync performance by sending the entire state of the data. (#12029)

Previously, when syncing in mixed mesh environments from Ditto SDK version 4.0.0 of the SDK to version 3.0.0, only partial updates were transmitted. With this release, all updates are transmitted, resulting in greater data consistency and completeness when syncing across different versions of the SDK.

​

FIXED: The issue causing the Small Peer Info collector to become stuck. (#12051)

Previously, the Small Peer Info collector would become stuck when attempting to upload logs in certain situations. Since resolving this issue, the Small Peer Info collector functions properly in those conditions.

​

FIXED: The issue causing fetch operations to appear unresponsive. (#9501, #12067)

With this fix, operations related to fetching attachments function as expected.

​

FIXED: The issue causing type check failures when setting duration parameters through ALTER SYSTEM SET. (#12070)

When using SystemParameter to configure timeouts before this fix, if using different types to encode the values representing the timeout, for instance, Milliseconds and Seconds, the values failed to downcast, an error log printed, and the new value ignored.

Now, when using mismatched wrapper types, Ditto performs a compatibility check, validates, and, if needed, converts them as appropriate.

​

Fixed (Linux): Similar to .NET, LAN communication (P2P) failed to advertise on newly added interfaces, such as enabling WiFi at runtime.

​

Added: Similar to .NET, setting the deviceName property of a Ditto instance after sync has started will now log a warning, indicating that changes to the value only take effect after restarting sync.

​

Fixed: The deviceName property on Ditto instances now correctly reflects the truncation of the value to 24 bytes when starting sync.

​

Changed: Documentation for DittoError has been updated to include a usage example.

​

Added: Implemented the connectionRequestHandler property in ditto.presence, enabling filtering of incoming connections from other peers.

​

Added: Introduced the peerMetadataJSONString property and the setPeerMetadataJSONString method to ditto.presence.

​

Added: Introduced the peerMetadata property and the setPeerMetadata() method to ditto.presence.

​

Added: Added properties peerMetadata and identityServiceMetadata to Peer.

​

Added: Implemented the peerKeyString property in Peer as a replacement for the now deprecated peerKey.

​

Deprecated: The peerKey property on Peer has been deprecated. Use the newly added peerKeyString property instead.

​

Fixed: Addressed an issue where DQL methods might include sensitive information in error logs. This fix removes the query string and query arguments from the context property of errors thrown by ditto.store.execute(), ditto.store.registerObserver(), and ditto.sync.registerSubscription() methods.