​

4.11.1 Kotlin & Java Specific Changes

Fixed: stopSync does not properly terminate WiFi Aware connections (#17235)

Added: system parameters to enable/disable WiFi Aware client/server (#17314)

Fixed: crash on startup on devices running Android 8.1 and below (#17460)

​

4.11.1 Common Changes

No Specific Changes

​

4.11 - DQL & Legacy Compatibility

With 4.11 we are excited to introduce legacy compatibility for DQL, which allows developers to use the new DQL APIs while maintaining compatibility with existing code and data. This capability, called STRICT MODE, is designed to ease the transition to the new APIs and ensure that existing applications continue to function as expected.

​

DQL Strict Mode

DQL Strict Mode provides control over how DQL statements are evaluated and how data types are inferred. When disabled, it offers more flexibility in working with documents without requiring explicit collection definitions.

With strict mode disabled (DQL_STRICT_MODE=false), Ditto is more flexible and will infer the CRDT type based on the document’s shape:

• Objects are treated as CRDT maps
• Scalars and arrays are treated as Registers
• Counters and attachments are inferred from operations
• Collection definitions are no longer required

Setting DQL Strict Mode

ALTER SYSTEM SET DQL_STRICT_MODE = false

​

DQL Differ

DQL 4.11 introduces a new Differ API for calculating diffs between query results. This API can be used with a store observer to understand how its results change over successive callbacks. The Differ API provides a way to track changes in the data and can be used to optimize data synchronization and conflict resolution.

Differ API documentation.

​

DQL Transactions

DQL 4.11 introduces support for transactions. Transactions can be used to group multiple DQL statements together, ensuring that either all of the statements are executed successfully or none of them are. This is particularly useful in scenarios where multiple updates or inserts need to be made to the database, and you want to ensure that either all of them are applied or none of them are.

Transactions API documentation.

​

Advanced DQL Features

DQL 4.11 introduces numerous advanced querying and manipulation features:

​

Tombstone Reaper

The tombstone reaper is now enabled by default for all SDKs. This feature helps clean up deleted document markers (tombstones) to improve database performance and reduce storage requirements.

The reaper:

• Automatically removes tombstones after a configurable time-to-live (TTL)
• Uses read transactions for efficiency
• Includes random scheduling variations to prevent performance problems
• Tracks the number of tombstones evicted for monitoring
• Remembers previous reap times across restarts

This improves overall system performance by reducing the long-term accumulation of tombstones when calling DELETE to remove documents.

​

4.11.0 Common Changelog

​

Added

• Introduced DQL_STRICT_MODE as a system parameter to opt-out of requiring collection definitions (#16573)
• Added transports_ble_adapter_mac system parameter to allow configuring bluetooth adapter MAC address (#16159)
• Small peers now log the result of OnlinePlayground authentication, including if the supplied shared token was incorrect (#16301)
• A cumulative running total of the number of tombstones ever evicted by the reaper to Small Peer Info (#16424)
• Added system parameters to fine-tune Ditto behavior when it encounters a locked persistence directory (#CORE-479)
• New DQL CAST function to explicitly cast between types (#QE-108)
• Conditional functions for unknowns (#QE-111)
• DQL DATE processing functions (#QE-112)
• New DQL clause USE IDS for direct document retrieval by id (#QE-114)
• Additional string functions (#QE-124)
• SIMILAR TO filter expression (#QE-125)
• PN_COUNTER support (APPLY clause in UPDATE statement) (#QE-127)
• UNSET clause to the UPDATE statement (#QE-128)
• MERGE as an alias for UPDATE in the ON ID CONFLICT clause of an INSERT statement (#QE-132)
• The ability to use runtime-evaluated expressions in object constructs in DQL statements (#QE-133)
• The ability to use runtime-evaluated expressions in array constructs in DQL statements (#QE-134)
• Static expressions in mutators (#QE-139)
• Additional object manipulation DQL scalar functions (#QE-159)
• EXPLAIN for query plan (#QE-105)
• More detailed messages about network problems are logged when authentication fails (#SEC-124)

​

Changed

• Use the modern rustls-platform-verifier TLS crate on all platforms rather than rustls-native-certs (#15956)
• The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer (#16237)
• The tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously (#16286)
• The tombstone reaper to be enabled by default for all SDKs (#16350)
• INSERT to now accept either DOCUMENTS or VALUES as the keyword introducing the list of values to insert (#QE-140)
• Date functions accept “local” as a timezone & use the executing peer’s local timezone (#QE-168)
• Unified dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)

​

Fixed

• Fixed issue where user_collection_sync_scopes could not be updated after initial set (#16299)
• Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)
• Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)

​

Removed

• Old multihop networking stack and system parameter network_use_nextgen_multihop_stack (#16370)
• TLS certificate compression, which led to incompatibilities (#SDKS-1046)

​

Performance

• The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database (#16338)
• Improved speed and efficiency of the tombstone reaping process (#16535)

​

4.11.0 Kotlin & Java Specific Changes

Added: DittoTransaction representing an active DQL transaction. (#15873)

Added: DittoTransactionInfo encapsulating information about a transaction. (#15873)

Added: Method transaction() on DittoStore allowing a DQL transaction to be performed. (#15873)

- DittoTransactionCompletionAction representing an action to be taken upon the completion of a transaction: commit or rollback. (#15873)

Added: An opt-in Android foreground service which allows Ditto to sync in the background. (#16097)

Added: The new DittoDocumentPath::number and DittoDocumentPath::numberValue accessors. (#16776)

Fixed: transport config changes that are made directly on the DittoTransportConfig object that a Ditto instance references are now applied when startSync is called. (#16835)

Fixed: WiFi Aware occasional failure to start advertising (#16844)

Fixed: Transaction hints in log messages from calls to upsert() included $name instead of the actual collection name. (#SDKS-1012)

Added: class DittoDiffer, which calculates diffs between query results. This can be used with a store observer to understand how its results change over successive callbacks. (SDKS-981)

Added: class DittoDiff to represent diffs produced by the DittoDiffer. (SDKS-981)

Changed: Unify dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)

​

4.10.5 Kotlin/Java Specific Changes

Fixed: transport config changes that are made directly on the DittoTransportConfig object that a Ditto instance references are now applied when startSync is called. (#16835)

Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)

Fixed: hard crash on Android 10 Bluetooth (#17073)

Fixed: stopSync` does not properly terminate WiFi Aware connections (#17235)

​

4.10.5 Common Changes

No Specific Changes

​

4.10.2 Common Changes

Removed: zlib certificate compression for small peer TLS 1.3 session negotiation, which caused incompatibilities (#1005)

​

4.10.1 Common Changes

Changed: The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer. (#16237)

Changed: the tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously. (#16286)

Fixed: Unable to update user_collection_sync_scopes after initial set bug (#16299)

Performance: The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database. (#16338)

Changed: The tombstone reaper to be enabled by default for all SDKs. (#16350)

​

4.10.0 New Capabilities

​

Small Peer User Collection Sync Scopes

User Collection Sync Scopes let you control how data from each user collection is shared with other connected Peers. This is useful when you want to fine-tune which collections do not sync, which collections sync with only Ditto Server and which collections sync only with connected Small Peer devices.

Sync scopes can be used to control how documents sync, reduce unnecessary data transfer, and improve performance.

​

Details

DQL Syntax for setting a Sync Scope

ALTER SYSTEM SET USER_COLLECTION_SYNC_SCOPES = {
    collection_1: "BigPeerOnly",
    collection_2: "SmallPeersOnly",
    collection_3: "LocalPeerOnly"
}

Example Setting A Sync Scope in the SDKs

In the example below we’ll set the collection local_mesh_orders to only sync with other small peers in the mesh and not with Ditto Serverusing the SmallPeersOnly sync scope.

var syncScopes = [
"local_mesh_orders": "SmallPeersOnly"
];
await ditto.store.execute(
    query: "ALTER SYSTEM SET USER_COLLECTION_SYNC_SCOPES = :syncScopes",
    arguments: ["syncScopes": syncScopes]);

​

Store Observers Run Multi-Threaded By Default

Ditto store observers (e.g ditto.store.registerObserver(...)) will now run multi-threaded on all platforms (except for in Web Browsers) by default. Prior to 4.10.0 all store observers would run on the same thread which could lead to performance issues for applications with multiple store observers. Users may see performance improvements on applications using more than one store observer, especially during high-volume usage.

Ditto guarantees order sequencing for a single observer. Because observers now run in parallel there is not a guarantee for order sequencing between observers. This behavior change should not have an impact on your application. If you have concerns or questions about this behavior change reach out to Ditto’s Customer Support.

To reset Ditto to only use a single thread for all Ditto observers use the following system parameter. For instruction on how to set a system parameter in your specific SDK see SDK>Customizing System Settings

ALTER SYSTEM SET STORE_OBSERVERS_NUM_THREADS = 1

​

DQL Support for LIKE operator

The LIKE operator can now be used for pattern matching of string values.

Example

SELECT * FROM my_collection
WHERE some_field LIKE 'abc%'

​

DQL New Objects operators

DQL now supports two new methods for querying over the keys and values in an object.

• object_keys(object): Returns an array with all the keys in the given object.
• object_values(object): Returns an array with all the values in object.

​

DQL LIMIT/ OFFSET / ORDER BY support for mutation operators

Users can now use LIMIT, OFFSET and ORDER BY in EVICT and UPDATE statements.

DQL Example

EVICT FROM users
WHERE status = 'inactive'
LIMIT 10;

​

4.10.0 Kotlin/Java Specific Changelog

• New triggerTestPanic method on DittoExperimental to intentionally trigger a panic in the Ditto Core for testing purposes. (#15408)
• ProGuard configuration file, ensuring isMinifyEnabled can be used with Ditto. (#15523)
• Repeated reads of a document within an ID-specific transaction scope of the query builder API now reflect changes made in the same transaction. (SDKS-245)
• (Experimental) Bus API correctly falls back to multihop connectivity if a direct connection is lost. (#15670)
• Clarified API docs for DittoStoreObserver and DittoSyncSubscription. (#SDKS-563)
• Deprecated DittoAddress.siteId. Use PeerKeyString instead. (#SDKS-589)
• Deprecated DittoBase.siteId. Use PeerKeyString instead. (#SDKS-589)
• Deprecated DittoIdentity.OfflinePlayground.siteId. Use PeerKeyString instead. (#SDKS-589)
• Deprecated DittoIdentity.SharedKey.siteId. Use PeerKeyString instead. (#SDKS-589)

​

4.10.0 Common Changelog

​

Added

• In DQL, an element of an array can be accessed by expression. (feat: Disallow double-quoted identifiers inside of new operators #12529)

• DQL field and element selection (DQL: Support field and element selection. #13920)

• Full support for DQL virtual collections, and the system:dual virtual collection (parser changes for virtual collections #14954)

• On Android, logs now indicate if a missing permission is not declared in an app’s AndroidManifest.xml. (Clarify Android missing permission messages based on manifest presence #15094)

• LIKE DQL operator. (#QE-104)

• Added object_keys and object_values scalar DQL functions. (#QE-73)

• A new system parameter network_use_nextgen_multihop_stack to switch between our existing and experimental next generation network stacks (#15362)

• Enabled zlib certificate compression (RFC8879) for small peer TLS 1.3 session negotiation ([security] enable TLS certificate compression #15801)

• A system parameter to selectively disable the BLE peripheral (server) transport. (New System Parameter: transports_ble_peripheral_is_enabled #15802)

• A STORE_OBSERVERS_NUM_THREADS system parameter to tweak the number of threads used by our store observer runtime to dispatch the sdk callback invocations. (Allow different live queries to be called in parallel #15814)

• The ability to configure a TTL for tombstones in the small peer store, which can be cleaned up via periodic sweeps from the new “reaper” module. (Finish porting the prototype tombstone reaping implementation into main #15886)

• Support for the new DQL DELETE statement on small peers. (Stabilize the DQL DELETE statement on SP #15899)

• A new Small Peer Info section named “reaper” and a field named “last_reap_time” that indicates the last time the tombstone reaper successfully completed its cleanup of expired tombstones. (Add small-peer-info field to track last completed reaper timestamp #16000)

• Add support for LIMIT, OFFSET and ORDER BY for mutation statements iin DQL (feat(query/dql): Support ORDER BY, LIMIT, OFFSET in mutation statements #16020)

• Added support for permanent system wide document deletes with the DELETE keyword in DQL for all Small Peer SDKs. This change requires a coordinated release with Ditto Server to ensure data is completely removed from the system. The Ditto Server change will be rolling out over the coming month and Ditto will be making a larger announcement upon release. For usage on deletes prior to the larger announcement see the following documents. Note that these documents are only accessible via their direct links and will not appear in the navigation bar just yet.

  • DQL DELETE Syntax
  • Removing Documents with Deletes

  For support on using deletes in your application reach out to Ditto Customer Support to get started.

​

Fixed

• A rare scenario where stopping sync with in-progress TCP connections could result in a crash. (#TRANS-303)
• If Ditto’s storage is cloned to another device (not recommended), clones which connect to each other will automatically create a new Peer Key and reauthenticate so that sync can proceed (Trigger reset peer identity when another peer clearly has the same Peer Key, add integration test #14653)
• On Android, missing system permissions will no longer crash Ditto. (Check for normal permissions in DittoSyncPermissions #15233)
• Fixed initialisation failing for the Linux Bluetooth transport due to invalid dbus messages (Linux BLE baseline bugfixes #15717)
• Fix OldQL and DQL comparison between large floats and integers returning an arbitrary result (Fix OldQL and DQL comparison between large floats and integers #15949)
• Reduced duplicate notifications for authentication state changes. (auth client: validity_update only on changes #15981)

​

Changed

• The link_enabled system parameter has been renamed to network_enable_multihop ([network] Add configurability for switching between legacy & next gen networking stacks #15362)
• Improve authentication flow over bad connections (Auth client/server improvements around challenge tokens #15430)
• Store observers are now invoked in parallel by default. (STORE_OBSERVERS_NUM_THREADS to default to 0 = num_cpus #16064)
• Synchronization protocol enhanced to reduce re-transmission of document data and metadata after eviction. (#DS-188)

​

4.9.4 Common Changes

Added: new system parameter which defines the number of threads used by the store observer runtime to dispatch the sdk callback invocations. For more information reach out to Ditto’s Support (#15814)

​

4.9.3 Common Changes

Changed: Added more actionable information to common connection handshake issues. (#15298)

Fixed: Fixed: Some Android devices could fail to export logs to the portal due to a file permissions error (#15582)

​

4.9.2 Kotlin Specific Changes

Fixed: Race condition causing NullPointerException trying to initialize Ditto.

​

4.9.1 Kotlin/Java Specific Changes

Fixed: Crash in Android WiFi Aware due to too many requests (#15288)

Fixed: avoid duplicated multicast connection on WiFi Aware (#15337)

Fixed: WiFi Aware does not restart after a stopSync/startSync cycle (#15349)

Fixed: Exceptions thrown from user-provided registerObserver lambdas are now logged as an error and re-thrown instead of triggering an unreadable rust panic. Use Thread.setDefaultUncaughtExceptionHandler to handle these exceptions.

​

4.9.1 Common Changes

Fixed: iOS 18 crashes related to Bluetooth LE are now resolved. (#15351)

Fixed: Disk usage stats are now reported in small peer info on android devices

​

4.9.0 Kotlin/Java Specific Changes

Added: Replacement suggestions to a few deprecated DittoError subclasses, making it easier to update error handling code. (#13162)

Added: parameters to control WiFi Aware staleness detection (#13438)

Added: INFO log for application background/foreground events (#13493)

Added: Error log & transport condition updates in case of insufficient BLE capabilities (#13793)

Added: transports_wifi_aware_background_mode system parameter to control WiFi Aware terminating on background behaviour (#13970)

Added: API docs for DittoSmallPeerInfo class. (#14202)

Added: New Ditto.updateTransportConfig() method which takes a lambda expression for modifying the active DittoTransportConfig. (#7377)

Fixed: properly handle all foreground events (#14118)

Fixed: Any exceptions thrown by a custom log callback will be logged as an error instead of crashing. (#14229)

Fixed: internal presence observing is now only active when at least one DittoPresenceObserver is active, improving performance. (#14256)

Deprecated: DittoBluetoothLeConfig.maxOutgoing. This value can now be configured using the mesh_chooser_max_active_ble_clients system parameter. See docs for details: Using Alter System. (#12320)

Deprecated: DittoLogger.ffiCustomLog(). Use DittoLog.i() and friends to write to the Ditto log. (#12617)

Removed (Experimental): Experimental Bus API pending a newer stable API. (#13777)

Removed: dittoHandle property, delete(), swigReleaseOwnership() and swigTakeOwnership() methods from Presence. These are internal memory management APIs that can cause memory leaks or crashes if used. (#14201)

​

4.9.0 Logging Clarity Improvements

This release comes with a number of logging clarity improvements. The goal of these changes are to make Ditto logs more consumable and actionable for end users. While there are wide ranging changes one of the principal goals of this work is to make our INFO level logs contain the right information to understand the state and health of the Ditto SDK.

• More information is included when physical connections start and end (such as the type of connection and the remote peer’s key)

• Devices that try to connect to themselves now say “device has connected to itself” instead of an error about the remote site ID being the same as the local peer

• Authentication client logs have been made more human readable. In particular:

• Recoverable errors are logged together with information about the recovery actions that are being undertaken

• Several abbreviations have been replaced with human-readable descriptions

• Errors are printed using their descriptions rather than by dumping debug representations

• Abbreviated jargon has been replaced with clearer wording in various log lines

• “phy ended” has become “physical connection ended”

• “error getting char” has become “required Ditto BLE characteristic not found”

• “error getting radio state” has become “error getting Windows Bluetooth radio state”

• Peer events that represent a transport starting or stopping are now logged with the message “transport started”/”transport ended” rather than as a raw peer_event=… log

• Reduced the noisiness of “starting” log lines around Ditto initialization time

• Noise from long peer keys in logs has been reduced in some places by replacing them with their truncated form

• Noise from duplicated peer ID/peer key information in replication logs has been reduced by consolidating the information

• “starting compression and export” is now followed by a “completed export” log at the same level

• Successful SQLite WAL recovery logs were further lowered from INFO to DEBUG

​

4.9.0 Common Changes

Added: regexp_like function in DQL. (#12129)

Added: peer_metadata and identity_service_metadata to the small peer info document (#12477)

Added: WebSocket connections now send X-DITTO-PEERPUBKEY and X-DITTO-ROUTING-HINT headers as part of the client connect handshake on all platforms except wasm32 (#13171)

Added: Small Peer Info now includes information about the current logger configuration (#14317)

Added: Permission rules can be written in DQL. (#14563)

Added: Ditto logs clearly indicate if a device has established a connection to itself. (#14634)

Added: the ability for DQL select statement to produce arbitrary projection lists. (#7979)

Added: local subscription info to small peer info (#9412)

Added: “device_disk_usage” information to small peer info (#9784)

Added: “store” information to small peer info (#9785)

Changed: Internal sync metadata format changed. Peers that downgrade to a previous SDK version after upgrading may require full resynchronization. (#14799)

Changed: Improved (de)serialization performance and reduced its contribution to the library size (#11545)

Changed: In updated replication protocol, verification of a received update file no longer requires both peers to use the same approach to subscription hashing. (#12672)

Changed: Experimental multihop connectivity is more resilient to mesh disruptions (#13648)

Changed: Synchronization protocol enhanced to reduce the amount of redundant data sent from small peers to big peers. (#13737)

Changed: Failed certificate verification due to a lagging device clock now results in a clearer error message (#14035)

Changed: Multicast peer-to-peer transport is no longer deprecated to improve support for environments where mDNS is unavailable (#14188)

Changed: Replication no longer redundantly re-transmits unchanged subscriptions after eviction. (#14290)

Changed: truncated peer keys are now logged in lieu of site IDs. (#14362)

Changed: improved logging in the authentication code to provide more useful error context. (#15038)

Changed: made the small peer “log request” feature much more responsive (#9777)

Fixed: Windows P2P LAN now can connect to IPv6 peers (#11468)

Fixed: In certain rare cases, resetting the same replication session multiple times could cause failure to converge. (#12616)

Fixed: Internal connection TLS caches are now bounded in size. This prevents a long living peer from holding onto more memory than required. (#13801)

Fixed: small peer info “metadata” is now persisted across app launches. (#13987)

Fixed: Bluetooth peers are handled better when disconnections occur, fixing a possible loop where a device might repeatedly try and fail to connect. (#14018)

Fixed: When replication sessions time out, write transactions on the document store may be blocked for a long period. (#14148)

Fixed: Multicast peer-to-peer transport (non-MDNS) is no longer influenced by TCP listen configuration in DittoTransportConfig (#14187)

Fixed: an issue preventing attachment transfers under high document sync load is now resolved. (#14189)

Fixed: A race condition that could cause attachments to unexpectedly crash under heavy load. (#14342)

Fixed: device name no longer incorrectly defaults to “ditto” in small peer info, when a named device first starts syncing (#14674)

Fixed: presence observers are now less-likely to fire when there is no observable change to presence data. (#14807)

Fixed: Logs report blocked_by=“no blocking transaction” when blocking transaction is known (#14966)

Fixed: In-band authentication data is now cleared on logout. (#15039)

Fixed: switching appIDs now invalidates cached authentication material correctly. (#15040)

Fixed: Transport conditions are now reported for a wider range of failures when starting Bluetooth and mDNS transports. (#8729)

Fixed: documentation for the stopSync() method to clarify that the Ditto store can be used locally after stopping sync. (#8849)

Fixed: Auth Client would hang when cached X.509 was expired. (#15110)

Deprecated:

Removed:

​

4.8.4 Common Release Notes

Fixed: Some Android devices could fail to export logs to the portal due to a file permissions error (#15582)

​

4.8.3 Common Release Notes

Fixed: iOS 18 crashes related to Bluetooth LE are now resolved. (#15351)

Changed: Added more actionable information to common connection handshake issues. (#15298)

​

4.8.2 Kotlin/Java Specific Changes

Fixed: Crash on WiFi Aware stalling and Ditto being re-created (#14423)

Fixed: Crash on Android after logout (#14562)

​

4.8.2 Common Changes

Fixed: Setting the minimum log level to WARN or ERROR will now correctly filter out very early logger initialization messages. (#14451)

Fixed: A bug that could prevent an attachment created with the same data on multiple peers from syncing. (#14481)

​

4.8.1 Kotlin/Java Specific Changes

Fixed: Crash in WiFi Aware when registering network publisher callback (#14013)

​

4.8.1 Common Changes

Added: The presence_use_multihop system parameter can be used to disable replicating presence information to other peers in the mesh, reducing network overhead. Use ONLY with guidance from the Ditto Customer Experience team. (#14128)

Changed: When failing to load document data sync (replication) metadata, Ditto now attempts to clear and regenerate the metadata. This is a behavior change where previously Ditto would fail to start. This serves to prevent problems after SDK version downgrades. (#13111)

Fixed: A rare issue where attachments may not be transferred between two peers.(#14158)

Fixed: On-disk logs are more robust in situations with unexpected Ditto shutdown. (#13723)

The version 4.8.0 SDK release update brings a substantial number of improvements and new features designed to enhance your development experience and provide even more powerful tools for building robust, real-time applications.

​

4.8.0 Kotlin/Java Specific Changes

Added: Method exportToFile() to DittoLogger, which exports collected logs to a compressed and JSON-encoded file on the local file system. Logs returned through this method are at DEBUG log level. This new API can be used to do forensic log gathering on a device. Logs are always collected and restricted to a fixed specific size limit to ensure they don’t grow endlessly. Once the size limit has reached new logs replace the oldest logs. (#11961)

Added: Synchronous method exportToFileBlocking() to DittoLogger for Java callers.

Added: A new getter and setter for Presence.connectionRequestHandler which take a DittoConnectionRequestHandlerCallback, making the API much easier to call from Java. (#12770)

Added: Properties peerKeyString1 and peerKeyString2 on DittoConnection replacing the deprecated properties peer1 and peer2 (#12799)

Added: Documentation for startSync() and disableSyncWithV3() to include a recommendation for performance improvement. (#13098)

Performance of initial sync when bootstrapping a new peer can be improved by calling disableSyncWithV3() before calling startSync().

Only do this when all peers in the mesh are known to be running Ditto v4 or higher.

Added: Documentation for property peerMetadata on DittoPeer to add information about how the property behaves over the lifecycle of the DittoPresenceGraph (#13479)

Added: optional hint parameter to upsert() on DittoCollection and DittoScopedWriteTransaction. By providing a hint you can better track the source of blocked write transactions (#7110)

Changed: Android Wifi Aware internal logic to avoid duplicate attempts to create the network and speed up time to successful connection (#12458)

Changed: Improve Android WifiAware stability by actively terminate when the application goes into the background and resume when the application comes back to the foreground (#12804)

Changed: Improve staleness detection in Android Wifi Aware leading to a more reliable connection. (#12866)

Changed: API docs no longer include unintentially exposed protected members. (#13200)

Changed: (Linux&Android) Increased mDNS multicast TTL from 1 to 255 to allow for cross-VLAN announcements (#13590)

Changed: Slow or blocked write transactions now log hints showing the waiting and blocking transactions (#7110)

Fixed: The SDK will no longer crash when it observes malformed mDNS traffic (#12396)

Fixed: The SDK will no longer crash when multiple threads call stopSync() (#11554)

Deprecated: Properties peer1 and peer2 on DittoConnection, please use peerKeyString1 and peerKeyString2 instead (#12799)

Deprecated: Property queryOverlapGroup on DittoPeer and DittoRemotePeerV2, always return 0 going forward. Query Overlap Groups was an experiemental feature that has been removed after it was determined this would not support production scenarios. (#13215)

Removed: delete(), swigReleaseOwnership() and swigTakeOwnership() methods from DittoLogger. These are internal memory management APIs that were unintentially exposed and can cause memory leaks or crashes if used (#12617)

Removed (Experimental): property queryOverlapGroup of DittoExperimental

Removed (Experimental): method setPriorityForQueryOverlapGroup() from DittoExperimental. (#13215)

Removed (Experimental): deprecated methods setQueryOverlapGroup(queryOverlapGroup:) and setPriority(_:forQueryOverlapGroup:ditto:) of DittoExperimental

​

4.8.0 Common Changes

Changed: The Logging output format for Ditto SDKs has been changed in 4.8.0 as part of our process to improve readablility and observability across the Ditto platform. Customers should validate the new data format in their existing data ingestion pipelines.

New Logging Schema:

YYYY-MM-DDTHH:MM:SS.ssssss LEVEL TARGET: MESSAGE {field1=value1, field2=value2, …}

• Timestamp (YYYY-MM-DDTHH:MM.ssssss):

• Format: ISO 8601 format (e.g., 2024-08-16T14:23:05.123456Z).

• Description: The exact time when the log entry was generated.

• Log Level (LEVEL):

• Format: One of the following levels— DEBUG, INFO, WARNING, ERROR.

• Description: Indicates the severity or importance of the log entry.

• Target (TARGET):

• Format: A string, representing the internal module/area

• Description: The source of the log entry, often corresponding to the internal module where the error was called. (e.g. ditto_init or peer_event_listener)

• Message (MESSAGE):

• Format: A human-readable string.

• Description: The main content of the log entry.

• Structured Fields ({field1=value1, field2=value2, …}):

• Format: A list of key-value pairs enclosed in curly braces.

• Description: Any additional structured data associated with the log entry. These fields can include custom data that you attach to the log entry using the tracing::info!(field1 = value1, “Message”) syntax.

4.7 and earlier Logging Format Example

[INFO] 2024-08-02T10:25:39.981Z: AuthClient: using cached SiteId; site_id = 16248337916605943930

4.8.0 and later Logging Format Example

2024-08-02T11:32:25.905562+01:00 INFO ditto_init:new_auth_client: ditto_auth: using cached SiteId site_id=16248337916605943930

Added: DQL support for string functions byte_length and char_length #12147

Added: DQL support for inline object and array literal creation #12312

Added: object_length and deserialize_json functions in DQL #12605

Added: DQL INSERT statements support inlining the construction of an object #12761

Added: Remote query capability allows Big Peer to execute read-only queries on Small Peers. Usagne currently internal to Ditto Customer Experience team members #12268

Added: Diagnostics are now provided when collecting Small Peer Info on-disk logs #12080

Added: output version, sdk and build information on startup for easier debugging #12845

Added: Introduced new internal capabilities ensure a large data payload (e.g. a large attachment file) cannot use all the bandwidth and prevent other data from being synced.

Added: mDNS transport can now be configured with a static port for the TCP server using the tcp_server_bind_mdns_server_port system parameter #13442

Changed: New logging format that includes detailed information #10794

Changed: The Ditto logger is now more verbose by default, with a minimum log level of INFO instead of WARNING #11141

Changed: small peer info collection is now enabled by default and its default sync scope is now BigPeerOnly #12709 (Device dashboard will populate information automatically)

Changed: (Experimental) Multihop connections are formed more efficiently, e.g., for Ditto Bus. This is an incompatible protocol change from previous versions #12108, #12623

Changed: Improve performance by avoiding deserializing the entire Document if only part of it is required #10887

Changed: Attachments now sync independently of documents, improving rainbow connection efficiency #11156

Changed: Updating several depdencies for security patching (Patch multiple old packages #12087)

Changed: How network OS capabilities are internally linked into the library to allow for more relaiable and reusable internal logic #12362

Changed: API documentation for execute method to include information about its interaction with sync subscriptions #12365

Changed: Default to creating backup Bluetooth connections when there is an existing WiFi connection (Default mesh_chooser_avoid_redundant_bluetooth to false #13398)

Changed: console logging will now output to STDERR instead of STDOUT ([logs] logging now outputs to STDERR instead of STDOUT #13477)

Fixed: Bytes and Arrays not being always comparable in the legacy query language #10887

Fixed: Legacy query language casting counters and maps with tombstones to the wrong boolean value #10887

Fixed: Lowered the log level of a few SQLite3 notices that were being unnecessarily logged as warnings #12032

Fixed: A crash that can occur if a DQL query contains a multi-byte character #12324

Fixed: An issue with the transports multiplexer that could lead to a hung process. This occured when dealing with large messages spanning multiple chunks (>64KB) #12428

Fixed: A rare edge case that could slow down data sync betten peers #12527

Fixed: Frequent subscription changes along with eviction calls could cause failure to converge on data leading to inconsistent data state #11431

Fixed: Memory leak in third party library used for peer-internal messaging #12895

Fixed: Transport inactive connection tracking no longer stores identifiers indefinitely. This could lead to a slow growth in memory consumption for long lived applications (~ 1MB/day on transports with high rate, unreliable discovery - such as BLE) #13147

Fixed: Unexpected failures to obtain a certificate for peer-to-peer sync are now logged at ERROR level instead of WARNING #13323

Removed: (Experimental) Ditto no longer supports the experimental Query Overlap Groups feature and support has been removed. This experimental feature has been removed after running a determining a lack of feasability in production scenarios. The team is iterating and plan on interoducing improved capabilitites. #13075

​

ADDED: Support for specifying the mDNS connection ports for Small Peer SDK instances

Allows you to specify a specifc port that can will be used for mDNS announcements and connections on a given device. This can be used for connect with peers across a VLAN while scanning for open ports.

Not setting the config will result in the current default behavior of a random port being assigned.

Setting the config

The config can be set using DQL ALTER SYSTEM and the tcp_server_bind_mdns_server_port config.

ALTER SYSTEM SET tcp_server_bind_mdns_server_port TO 1234

This command should be run using the ditto.store.execute(...) command within your application.

let result = await ditto.store.execute(query: "ALTER SYSTEM SET tcp_server_bind_mdns_server_port TO 1234")

​

FIXED: A finalizer issue that was causing crashes when closing sync subscriptions (#13849, #13732, #13738)

Resolves a race condition that could be hit when closing a sync subscription or stopping sync that caused Ditto to crash.

​

IMPROVED: (Linux/Android) Increased mDNS multicast TTL from 1 to 255 to allow for cross-VLAN announcements (#13590)

To support scenarios with cross-VLAN device discovery we increated the TTL to ensure message have enough time to reach their destination and get acknowledged.

​

Fixed: Intermittent crash when finalizing a legacy DittoSubscription that contained a sort() clause. (#13394)

Resolves and issue where using sort() in sync subsriptions with the legacy query builder would result in a crash.

​

Fixed: Included necessary symbols so that Ditto Core crashes can be properly symbolicated with crash reporters. (#13696)

Previously not all symbols were being included in the package bundle and this led to crashes not being able to be symbolicated and ultimately made readable.

Now the Kotlin bundle includes the correct packages

​

FIXED: An issue causing a race condition with a Live Query callback. (#13180)

This fix addresses the issue causing a Live Query callback to receive a delta update, known as a diff, before the initial document.

​

FIXED: A compatibility issue between documents created using different CRDT versions. (#13236)

Previously, there was an issue in which two documents on different internal versions could fail to integrate changes created using an older CRDT version.

​

IMPROVED: Logging for batch write transactions. (#13339, #13339)

Due to a race condition caused by the logging system mistakenly identifying the previous transaction as blocking the current one, the logger would inaccurately indicate the current transaction as being blocked by the previous one from an unknown originator.

​

FIXED: An issue in the Small Peer datastore causing slow performance or process termination. (#13395)

This issue was due to an aging dependency with a race condition, which caused thread tasks to get stuck and transactions to be blocked for extended periods.

​

IMPROVED: The Ditto SDKs compilation performance. (#13174)

We’ve optimized compilation by reducing the size of each binary slice of the Ditto SDKs by 19% and fixing incompatibility with PLCrashReporter.

​

FIXED: Kotlin and Java Android devices now appear as JVMBased in the Device Dashboard instead of Unknown. (#13251)

​

FIXED: An issue causing a crash with the message gatt must not be null. (#13411)

​

ADDED: mesh_chooser_avoid_redundant_bluetooth system parameter. (#12984)

This new system parameter allows greater control over the use of Bluetooth Low Energy (LE) connections in Ditto’s peer-to-peer mesh network. When set to false, Ditto creates Bluetooth LE connections alongside other peer-to-peer connections, serving as a fallback if the primary transport, such as Apple Wireless Direct Link (AWDL) or LAN, fails.

However, before you enable the mesh_chooser_avoid_redundant_bluetooth parameter, consider the following technical tradeoffs:

• Benefits:
  • Provides additional fallback to maintain connection stability when primary connections cease to function.
  • Reduces time to establish new connections since, by having multiple fallback options, Ditto can more quickly re-establish connections.
• Drawbacks:
  • Increases network traffic since using multiple transports results in additional network overhead
  • Using Bluetooth LE alongside other transports increases resource consumption. In the worst case, enabling this setting can result in slower sync performance and battery drain.

To enable the mesh_chooser_avoid_redundant_bluetooth parameter, use the following DQL query: ALTER SYSTEM SET mesh_chooser_avoid_redundant_bluetooth = false

​

CHANGED: The AuthClient by improving the log message for certificate issues returned from the Big Peer Authentication Service. (#13138)

​

FIXED: A performance-related issue affecting bulk eviction operations for documents with attachments. (#12331)

​

ADDED: Improved data compression in the handshake protocol when establishing connections to other peers. (#11264)

This allows peers with large permission sets to establish connections on low bandwidth transports like Bluetooth LE more quickly.

​

FIXED: The HTTP protocol by adding content-length: 0 header compliance in empty POST requests. (#12346)

Before, if you invoked an empty POST request, the cloud load balancer would reject your request with a 411 - Length Required error message.

Now you must explicitly state that the body of your POST request is empty by setting content-length: 0 in the header of your HTTP API call.

​

FIXED: Handling of malformed subscription queries in the legacy query builder API by adding validation logic early to ensure invalid queries are identified early and logged. (#12463)

There was an issue where invoking subscribe() on an invalid query caused sync to be silently disabled.

Invalid queries in the subscribe() method are now identified, disabled, and logged as errors.

​

FIXED: Subscription assumption logic so all acknowledgments are processed, regardless of subscription changes. (#12541)

In scenarios where subscriptions rapidly changed back and forth while documents were being concurrently altered, documents would diverge at merge.

We’ve removed subscription logic ignoring acknowledgements based on assumed subscription mismatch and introduced salting functionality to ensure each document is unique for each occurrence of a subscription. This helps differentiate between new and reverted subscription states.

​

FIXED: Sync inefficiency where changing a subscription resulted in unnecessary syncing of redundant update files. (#12967)

To eliminate redundant data from being synced unnecessarily, regardless of whether the remote peer previously acknowledged them, we’ve added validation checks ensuring that only unacknowledged data is synced.

This fix reduces sync update file sizes, leading to more performant session sync updates, particularly in apps consisting of large, stable subscriptions and infrequent document changes.

​

CHANGED: Reduced Bluetooth LE warn log noise. (#12674)

Changed: handle reverse DNS lookup queries in Android mdns to avoid mdns failing to connect in specific network configs

​

CHANGED: Android mdns to handle reverse DNS lookup queries. (#13000)

By adding reverse DNS lookup queries, Android mdns no longer fails to connect in specific network configurations.

​

FIXED: Wi-fi Aware so it is available on API level 32. (#12631)

​

FIXED: Missing version metadata. (#12843)

Now version metadata is correctly included in the string returned from Ditto.sdkVersion.

​

FIXED: Misleading errors related to attachment fetch operations. Now, associated error logs are more accurate and clear. (#12409)

​

FIXED: A specific issue where incorrect sync metadata rollback caused problems with data consistency at merge. (#12422)

​

FIXED: The documentation for the DittoStore.fetchAttachment API method. (#11783)

Before this patch release, the documentation incorrectly stated that the return value had to be kept in scope for the fetch to complete. As of this release, the documentation specifies that the return value does not need to be kept in scope.

​

ADDED: DittoExperimental.constructDittoFailable() API method. (#11801)

Previously, your app crashed if the persistence directory was already in use by another Ditto instance. Now, rather than crashing in such scenarios, Ditto throws a DittoError.LockedWorkingDirectoryError error.

​

ADDED: DittoExperimental.constructDittoFailable() API method.

Now, you can use this experimental alternative to the original Ditto constructor to prevent crashes when the persistence directory is still in use by another Ditto instance at the time of calling the constructor. Now, rather than crashing in such scenarios, Ditto throws a DittoError.LockedWorkingDirectoryError error. (#11801)

​

FIXED: Exceptions that occur while creating or fetching attachments now include more details about the underlying cause in their error message for easier debugging. (#12235)

​

Fixed: An issue that prevented some attachment fetcher events from being delivered to event handlers. (#12331)

​

FIXED: Closing an attachment fetcher that failed to start downloading no longer results in an exception. (#12490)

​

FIXED: Handling of document-paths containing non-alphanumeric characters. Previous versions may have removed these characters from document-paths. (#12556)

​

FIXED: The legacy query language error-throwing mechanism to accept invalid field names properly. (#11888)

After upgrading to version 4.7 of the SDK, you may notice unexpected errors while using the legacy query builder language. These errors are likely due to invalid field names or path expressions previously undetected.

In addition, make sure to start unquoted_string with an underscore (_), a letter, or an alphanumeric character. For example, the following equivalent regular expression: [A-Za-z_][A-Za-z0-9_]*.

​

REMOVED: The is_connected_to_big_peer field from the Small Peer Info document. (#11883)

​

CHANGED: The small peer info JSON metadata by increasing the depth limit to 64 levels. (#11891)

​

IMPROVED: Transaction contention time for the ATTACHMENT data type, reducing potential timeouts. (#11893)

Before this improvement, concurrent write transactions were long-lived, potentially causing deadlocks and request timeouts. Now you’ll experience more effective handling of multiple attachments being processed at the same time.

​

FIXED: The issue causing sync crashes due to missing update files. (#11952)

With the latest release, Ditto handles unexpected situations or errors in a way that allows it to recover and continue functioning without crashing or causing disruptions.

​

ADDED: Parameters for adjusting certain settings during runtime. (#11968)

This rollout includes new system parameters for the MeshChooser feature for configuring a limited set of variables, allowing you to adjust certain settings during runtime.

​

RESOLVED: The warning message when peers connect is no more. (#11977)

The warning message previously logged when peers connected within the mesh is no longer recorded or printed.

​

IMPROVED: Sync performance by sending the entire state of the data. (#12029)

Previously, when syncing in mixed mesh environments from Ditto SDK version 4.0.0 of the SDK to version 3.0.0, only partial updates were transmitted. With this release, all updates are transmitted, resulting in greater data consistency and completeness when syncing across different versions of the SDK.

​

FIXED: The issue causing the Small Peer Info collector to become stuck. (#12051)

Previously, the Small Peer Info collector would become stuck when attempting to upload logs in certain situations. Since resolving this issue, the Small Peer Info collector functions properly in those conditions.

​

FIXED: The issue causing fetch operations to appear unresponsive. (#9501, #12067)

With this fix, operations related to fetching attachments function as expected.

​

FIXED: The issue causing type check failures when setting duration parameters through ALTER SYSTEM SET. (#12070)

When using SystemParameter to configure timeouts before this fix, if using different types to encode the values representing the timeout, for instance, Milliseconds and Seconds, the values failed to downcast, an error log printed, and the new value ignored.

Now, when using mismatched wrapper types, Ditto performs a compatibility check, validates, and, if needed, converts them as appropriate.

​

FIXED: P2P LAN now advertises on newly added interfaces, such as enabling WiFi at runtime.

​

FIXED: The deviceName property on Ditto instances now reflects the truncation of the value to 24 bytes when starting sync.

​

ADDED: Property connectionRequestHandler to ditto.presence, including related types, allowing filtering of incoming connections from other peers.

​

Added: Properties peerMetadata and peerMetadataJsonString to ditto.presence.

​

Added: Property identityServiceMetadata to DittoPeer.

​

Added: Property peerKeyString to DittoPeer as a replacement for the now deprecated peerKey.

​

DEPRECATED: Property peerKey on DittoPeer, please use the newly added peerKeyString instead.

​

ADDED: Runtime checks and locks to guard against dangling raw Ditto pointer usage in extremely unlucky race condition scenarios.

​

FIXED: Possible crash while downloading an attachment due to improper decoding of the attachment token.