C# Release Notes
4.11 - DQL & Legacy Compatibility
With 4.11 we are excited to introduce legacy compatibility for DQL, which allows developers to use the new DQL APIs while maintaining compatibility with existing code and data. This capability, called STRICT MODE, is designed to ease the transition to the new APIs and ensure that existing applications continue to function as expected.
DQL Strict Mode
DQL Strict Mode provides control over how DQL statements are evaluated and how data types are inferred. When disabled, it offers more flexibility in working with documents without requiring explicit collection definitions.
DQL Strict Mode
With strict mode disabled (DQL_STRICT_MODE=false
), Ditto is more flexible and will infer the CRDT type based on the document’s shape:
- Objects are treated as CRDT maps
- Scalars and arrays are treated as Registers
- Counters and attachments are inferred from operations
- Collection definitions are no longer required
Setting DQL Strict Mode
DQL Differ
DQL 4.11 introduces a new Differ
API for calculating diffs between query results.
This API can be used with a store observer to understand how its results change
over successive callbacks. The Differ
API provides a way to track changes in
the data and can be used to optimize data synchronization and conflict resolution.
DQL Transactions
DQL 4.11 introduces support for transactions. Transactions can be used to group multiple DQL statements together, ensuring that either all of the statements are executed successfully or none of them are. This is particularly useful in scenarios where multiple updates or inserts need to be made to the database, and you want to ensure that either all of them are applied or none of them are.
Transactions API documentation.
Advanced DQL Features
DQL 4.11 introduces numerous advanced querying and manipulation features:
Feature | Description | Example |
---|---|---|
UNSET clause | Remove fields from documents | UPDATE orders UNSET items.abc WHERE _id = 'my-id' |
CAST function | Explicitly convert between data types | SELECT * FROM products WHERE CAST(price AS INT) > 10 |
USE IDS clause | Direct document retrieval by ID | SELECT * FROM products USE IDS ['123', '456'] |
SIMILAR TO | Pattern matching expressions | SELECT * FROM products WHERE name SIMILAR TO '%phone%' |
Runtime expressions | Use expressions in object/array construction | UPDATE products SET metadata = { 'updated_at': current_timestamp() } |
Counters | Support for legacy compatible counter type | UPDATE products APPLY in_stock PN_INCREMENT BY 1.0 |
In 4.11, functional operations are denoted with an arrow (->
) are deprecated.
Tombstone Reaper
The tombstone reaper is now enabled by default for all SDKs. This feature helps clean up deleted document markers (tombstones) to improve database performance and reduce storage requirements.
The reaper:
- Automatically removes tombstones after a configurable time-to-live (TTL)
- Uses read transactions for efficiency
- Includes random scheduling variations to prevent performance problems
- Tracks the number of tombstones evicted for monitoring
- Remembers previous reap times across restarts
This improves overall system performance by reducing the long-term accumulation of
tombstones when calling DELETE
to remove documents.
4.11.0 Common Changelog
Added
- Introduced
DQL_STRICT_MODE
as a system parameter to opt-out of requiring collection definitions (#16573) - Added transports_ble_adapter_mac system parameter to allow configuring bluetooth adapter MAC address (#16159)
- Small peers now log the result of OnlinePlayground authentication, including if the supplied shared token was incorrect (#16301)
- A cumulative running total of the number of tombstones ever evicted by the reaper to Small Peer Info (#16424)
- Added system parameters to fine-tune Ditto behavior when it encounters a locked persistence directory (#CORE-479)
- New DQL CAST function to explicitly cast between types (#QE-108)
- Conditional functions for unknowns (#QE-111)
- DQL DATE processing functions (#QE-112)
- New DQL clause USE IDS for direct document retrieval by id (#QE-114)
- Additional string functions (#QE-124)
- SIMILAR TO filter expression (#QE-125)
- PN_COUNTER support (APPLY clause in UPDATE statement) (#QE-127)
- UNSET clause to the UPDATE statement (#QE-128)
- MERGE as an alias for UPDATE in the ON ID CONFLICT clause of an INSERT statement (#QE-132)
- The ability to use runtime-evaluated expressions in object constructs in DQL statements (#QE-133)
- The ability to use runtime-evaluated expressions in array constructs in DQL statements (#QE-134)
- Static expressions in mutators (#QE-139)
- Additional object manipulation DQL scalar functions (#QE-159)
- EXPLAIN for query plan (#QE-105)
- More detailed messages about network problems are logged when authentication fails (#SEC-124)
Changed
- Use the modern rustls-platform-verifier TLS crate on all platforms rather than rustls-native-certs (#15956)
- The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer (#16237)
- The tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously (#16286)
- The tombstone reaper to be enabled by default for all SDKs (#16350)
- INSERT to now accept either DOCUMENTS or VALUES as the keyword introducing the list of values to insert (#QE-140)
- Date functions accept “local” as a timezone & use the executing peer’s local timezone (#QE-168)
- Unified dedicated TCP servers across different transports (Multicast, mDNS, and WiFi Aware) by utilizing a single static TCP server implementation (#TRANS-131)
Fixed
- Fixed issue where user_collection_sync_scopes could not be updated after initial set (#16299)
- Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)
- Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)
Removed
- Old multihop networking stack and system parameter
network_use_nextgen_multihop_stack
(#16370) - TLS certificate compression, which led to incompatibilities (#SDKS-1046)
Performance
- The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database (#16338)
- Improved speed and efficiency of the tombstone reaping process (#16535)
4.11.0 C# Specific Changes
Added: DittoTransaction
representing an active DQL transaction. (#15729)
Added: DittoTransactionCompletionAction
representing an action to be taken upon the completion of a transaction: commit or rollback. (#15729)
Added: DittoTransactionInfo
encapsulating information about a transaction. (#15729)
Added: Method TransactionAsync()
on DittoStore
allowing a DQL transaction to be performed. (#15729)
Added: An opt-in Android foreground service which allows Ditto to sync in the background. (#16097)
Added: Support for Mac Catalyst. (#16379)
Added: DittoSyncPermissions
helper for .NET MAUI to check and request platform runtime permissions required for sync features. (#SDKS-1018)
Added: New Ditto.UpdateTransportConfig()
method which takes a lambda expression for modifying the active DittoTransportConfig
. (#SDKS-691)
Added: class DittoDiffer
, which calculates diffs between query results. This can be used with a store observer to understand how its results change over successive callbacks. (SDKS-979)
Added: class DittoDiff
to represent diffs produced by the DittoDiffer
. (SDKS-979)
4.10.5 C++ Specific Changes
Fixed: Windows devices failed to connect to Android on LAN (#17199)
Fixed: Bluetooth connectivity on Apple devices works when background Bluetooth permissions are missing. (#17065)
Fixed: A rare crash on Android devices when shutting down Bluetooth. (#17069)
Fixed: hard crash on Android 10 Bluetooth (#17073)
Fixed: stopSync
does not properly terminate WiFi Aware connections (#17235)
4.10.5 Common Changes
No Specific Changes
4.10.2 Common Changes
Removed: zlib certificate compression for small peer TLS 1.3 session negotiation, which caused incompatibilities (#1005)
4.10.1 Common Changes
Changed: The tombstone reaper scheduling to remember the previous reap times across shutdown and restart of the small peer. (#16237)
Changed: the tombstone reaper scheduling code to add random variations, which helps prevent performance problems if/when all peers run the reaper simultaneously. (#16286)
Fixed: Unable to update user_collection_sync_scopes after initial set bug (#16299)
Performance: The tombstone reaper uses read transactions for some of its work instead of holding open a long-running write transaction, thus reducing contention on the small peer database. (#16338)
Changed: The tombstone reaper to be enabled by default for all SDKs. (#16350)
4.10.0 New Capabilities
Small Peer User Collection Sync Scopes
User Collection Sync Scopes let you control how data from each user collection is shared with other connected Peers. This is useful when you want to fine-tune which collections do not sync, which collections sync with only Ditto Server and which collections sync only with connected Small Peer devices.
Sync scopes can be used to control how documents sync, reduce unnecessary data transfer, and improve performance.
SDK>Configuring Collection Sync
startSync()
is called USER_COLLECTION_SYNC_SCOPES
cannot be updated without
closing and reopening a new Ditto instance. This will be resolved in the next release. For more information
see Updating Sync Scopes
Details
Sync Scope | Purpose |
---|---|
”AllPeers” (default) | Sync with both Ditto Server and other Small Peers. This is the default behavior for all collections. |
”BigPeerOnly” | Sync only with Ditto Server (Big Peer). Do no sync with other Small Peers. |
”SmallPeersOnly” | Sync only with other Small Peers. Do not sync with Ditto Server. |
”LocalPeerOnly” | Do not sync this collection to Ditto Server or other Small Peers. |
DQL Syntax for setting a Sync Scope
Example Setting A Sync Scope in the SDKs
In the example below we’ll set the collection local_mesh_orders
to only sync with other small peers in the mesh and not with Ditto Serverusing the SmallPeersOnly
sync scope.
Store Observers Run Multi-Threaded By Default
Ditto store observers (e.g ditto.store.registerObserver(...)
) will now run multi-threaded on all platforms (except for in Web Browsers)
by default. Prior to 4.10.0
all store observers would run on the same thread which could lead to performance issues for applications
with multiple store observers. Users may see performance improvements on applications using more than one store observer, especially during
high-volume usage.
Ditto guarantees order sequencing for a single observer. Because observers now run in parallel there is not a guarantee for order sequencing between observers. This behavior change should not have an impact on your application. If you have concerns or questions about this behavior change reach out to Ditto’s Customer Support.
To reset Ditto to only use a single thread for all Ditto observers use the following system parameter. For instruction on how to set
a system parameter in your specific SDK see SDK>Customizing System Settings
DQL Support for LIKE
operator
The LIKE
operator can now be used for pattern matching of string values.
DQL>Operator Expressions
Pattern | Meaning |
---|---|
% | Matches zero or more characters (like .* in regex) |
_ | Matches exactly one character (like . in regex) |
Example
DQL New Objects operators
DQL now supports two new methods for querying over the keys and values in an object.
object_keys(object)
: Returns anarray
with all the keys in the givenobject
.object_values(object)
: Returns anarray
with all the values inobject
.
DQL>Operator Expressions
DQL LIMIT/ OFFSET / ORDER BY support for mutation operators
Users can now use LIMIT
, OFFSET
and ORDER BY
in EVICT
and UPDATE
statements.
DQL Example
4.10.0 C# Specific Changelog
- Added:
customAuthUrl
parameter to DittoOnlinePlayground identity. (#15745) - Fixed: DittoLogger API docs and ensured it is included in the API Reference page. (#15796)
- Fixed: Repeated reads of a document within an ID-specific transaction scope of the query builder API now reflect changes made in the same transaction. (SDKS-245)
- (Experimental) Bus API correctly falls back to multihop connectivity if a direct connection is lost. (#15670)
4.10.0 Common Changelog
Added
-
In DQL, an element of an array can be accessed by expression. (feat: Disallow double-quoted identifiers inside of new operators #12529)
-
DQL field and element selection (DQL: Support field and element selection. #13920)
-
Full support for DQL virtual collections, and the system:dual virtual collection (parser changes for virtual collections #14954)
-
On Android, logs now indicate if a missing permission is not declared in an app’s AndroidManifest.xml. (Clarify Android missing permission messages based on manifest presence #15094)
-
LIKE DQL operator. (#QE-104)
-
Added
object_keys
andobject_values
scalar DQL functions. (#QE-73) -
A new system parameter network_use_nextgen_multihop_stack to switch between our existing and experimental next generation network stacks (#15362)
-
Enabled zlib certificate compression (RFC8879) for small peer TLS 1.3 session negotiation ([security] enable TLS certificate compression #15801)
-
A system parameter to selectively disable the BLE peripheral (server) transport. (New System Parameter: transports_ble_peripheral_is_enabled #15802)
-
A STORE_OBSERVERS_NUM_THREADS system parameter to tweak the number of threads used by our store observer runtime to dispatch the sdk callback invocations. (Allow different live queries to be called in parallel #15814)
-
The ability to configure a TTL for tombstones in the small peer store, which can be cleaned up via periodic sweeps from the new “reaper” module. (Finish porting the prototype tombstone reaping implementation into main #15886)
-
Support for the new DQL DELETE statement on small peers. (Stabilize the DQL DELETE statement on SP #15899)
-
A new Small Peer Info section named “reaper” and a field named “last_reap_time” that indicates the last time the tombstone reaper successfully completed its cleanup of expired tombstones. (Add small-peer-info field to track last completed reaper timestamp #16000)
-
Add support for LIMIT, OFFSET and ORDER BY for mutation statements iin DQL (feat(query/dql): Support ORDER BY, LIMIT, OFFSET in mutation statements #16020)
-
Added support for permanent system wide document deletes with the DELETE keyword in DQL for all Small Peer SDKs. This change requires a coordinated release with Ditto Server to ensure data is completely removed from the system. The Ditto Server change will be rolling out over the coming month and Ditto will be making a larger announcement upon release. For usage on deletes prior to the larger announcement see the following documents. Note that these documents are only accessible via their direct links and will not appear in the navigation bar just yet.
For support on using deletes in your application reach out to Ditto Customer Support to get started.
Fixed
- A rare scenario where stopping sync with in-progress TCP connections could result in a crash. (#TRANS-303)
- If Ditto’s storage is cloned to another device (not recommended), clones which connect to each other will automatically create a new Peer Key and reauthenticate so that sync can proceed (Trigger reset peer identity when another peer clearly has the same Peer Key, add integration test #14653)
- On Android, missing system permissions will no longer crash Ditto. (Check for normal permissions in DittoSyncPermissions #15233)
- Fixed initialisation failing for the Linux Bluetooth transport due to invalid dbus messages (Linux BLE baseline bugfixes #15717)
- Fix OldQL and DQL comparison between large floats and integers returning an arbitrary result (Fix OldQL and DQL comparison between large floats and integers #15949)
- Reduced duplicate notifications for authentication state changes. (auth client: validity_update only on changes #15981)
Changed
- The link_enabled system parameter has been renamed to network_enable_multihop ([network] Add configurability for switching between legacy & next gen networking stacks #15362)
- Improve authentication flow over bad connections (Auth client/server improvements around challenge tokens #15430)
- Store observers are now invoked in parallel by default. (STORE_OBSERVERS_NUM_THREADS to default to 0 = num_cpus #16064)
- Synchronization protocol enhanced to reduce re-transmission of document data and metadata after eviction. (#DS-188)
4.9.4 C# Specific Changes
Added: customAuthUrl
parameter to DittoOnlinePlayground identity.
This extends support new cloud URL(#15745)
4.9.4 Common Changes
Added: new system parameter which defines the number of threads used by the store observer runtime to dispatch the sdk callback invocations. For more information reach out to Ditto’s Support (#15814)
4.9.3 Common Changes
Changed: Added more actionable information to common connection handshake issues. (#15298)
Fixed: Fixed: Some Android devices could fail to export logs to the portal due to a file permissions error (#15582)
No C# Specific Changes
4.9.1 Common Changes
Fixed: iOS 18 crashes related to Bluetooth LE are now resolved. (#15351)
Fixed: Disk usage stats are now reported in small peer info on android devices
4.9.0 C# Specific Changes
Changed: Upgraded from .NET 6 to .NET 8. (#13288)
Fixed: DittoAttachmentFetcher.Dispose() would not cancel the onFetchEvent callbacks. (#13859)
Removed: experimental DittoBus APIs. (#13741)
4.9.0 Logging Clarity Improvements
This release comes with a number of logging clarity improvements. The goal of these changes are to make Ditto logs more consumable and actionable for end users. While there are wide ranging changes one of the principal goals of this work is to make our INFO level logs contain the right information to understand the state and health of the Ditto SDK.
-
More information is included when physical connections start and end (such as the type of connection and the remote peer’s key)
-
Devices that try to connect to themselves now say “device has connected to itself” instead of an error about the remote site ID being the same as the local peer
-
Authentication client logs have been made more human readable. In particular:
-
Recoverable errors are logged together with information about the recovery actions that are being undertaken
-
Several abbreviations have been replaced with human-readable descriptions
-
Errors are printed using their descriptions rather than by dumping debug representations
-
Abbreviated jargon has been replaced with clearer wording in various log lines
-
“phy ended” has become “physical connection ended”
-
“error getting char” has become “required Ditto BLE characteristic not found”
-
“error getting radio state” has become “error getting Windows Bluetooth radio state”
-
Peer events that represent a transport starting or stopping are now logged with the message “transport started”/“transport ended” rather than as a raw peer_event=… log
-
Reduced the noisiness of “starting” log lines around Ditto initialization time
-
Noise from long peer keys in logs has been reduced in some places by replacing them with their truncated form
-
Noise from duplicated peer ID/peer key information in replication logs has been reduced by consolidating the information
-
“starting compression and export” is now followed by a “completed export” log at the same level
-
Successful SQLite WAL recovery logs were further lowered from INFO to DEBUG
4.9.0 Common Changes
Added: regexp_like function in DQL. (#12129)
Added: peer_metadata and identity_service_metadata to the small peer info document (#12477)
Added: WebSocket connections now send X-DITTO-PEERPUBKEY and X-DITTO-ROUTING-HINT headers as part of the client connect handshake on all platforms except wasm32 (#13171)
Added: Small Peer Info now includes information about the current logger configuration (#14317)
Added: Permission rules can be written in DQL. (#14563)
Added: Ditto logs clearly indicate if a device has established a connection to itself. (#14634)
Added: the ability for DQL select statement to produce arbitrary projection lists. (#7979)
Added: local subscription info to small peer info (#9412)
Added: “device_disk_usage” information to small peer info (#9784)
Added: “store” information to small peer info (#9785)
Changed: Internal sync metadata format changed. Peers that downgrade to a previous SDK version after upgrading may require full resynchronization. (#14799)
Changed: Improved (de)serialization performance and reduced its contribution to the library size (#11545)
Changed: In updated replication protocol, verification of a received update file no longer requires both peers to use the same approach to subscription hashing. (#12672)
Changed: Experimental multihop connectivity is more resilient to mesh disruptions (#13648)
Changed: Synchronization protocol enhanced to reduce the amount of redundant data sent from small peers to big peers. (#13737)
Changed: Failed certificate verification due to a lagging device clock now results in a clearer error message (#14035)
Changed: Multicast peer-to-peer transport is no longer deprecated to improve support for environments where mDNS is unavailable (#14188)
Changed: Replication no longer redundantly re-transmits unchanged subscriptions after eviction. (#14290)
Changed: truncated peer keys are now logged in lieu of site IDs. (#14362)
Changed: improved logging in the authentication code to provide more useful error context. (#15038)
Changed: made the small peer “log request” feature much more responsive (#9777)
Fixed: Windows P2P LAN now can connect to IPv6 peers (#11468)
Fixed: In certain rare cases, resetting the same replication session multiple times could cause failure to converge. (#12616)
Fixed: Internal connection TLS caches are now bounded in size. This prevents a long living peer from holding onto more memory than required. (#13801)
Fixed: small peer info “metadata” is now persisted across app launches. (#13987)
Fixed: Bluetooth peers are handled better when disconnections occur, fixing a possible loop where a device might repeatedly try and fail to connect. (#14018)
Fixed: When replication sessions time out, write transactions on the document store may be blocked for a long period. (#14148)
Fixed: Multicast peer-to-peer transport (non-MDNS) is no longer influenced by TCP listen configuration in DittoTransportConfig (#14187)
Fixed: an issue preventing attachment transfers under high document sync load is now resolved. (#14189)
Fixed: A race condition that could cause attachments to unexpectedly crash under heavy load. (#14342)
Fixed: device name no longer incorrectly defaults to “ditto” in small peer info, when a named device first starts syncing (#14674)
Fixed: presence observers are now less-likely to fire when there is no observable change to presence data. (#14807)
Fixed: Logs report blocked_by=“no blocking transaction” when blocking transaction is known (#14966)
Fixed: In-band authentication data is now cleared on logout. (#15039)
Fixed: switching appIDs now invalidates cached authentication material correctly. (#15040)
Fixed: Transport conditions are now reported for a wider range of failures when starting Bluetooth and mDNS transports. (#8729)
Fixed: documentation for the stopSync() method to clarify that the Ditto store can be used locally after stopping sync. (#8849)
Fixed: Auth Client would hang when cached X.509 was expired. (#15110)
Deprecated:
Removed:
4.8.4 Common Release Notes
Fixed: Some Android devices could fail to export logs to the portal due to a file permissions error (#15582)
4.8.3 Common Release Notes
Fixed: iOS 18 crashes related to Bluetooth LE are now resolved. (#15351)
Changed: Added more actionable information to common connection handshake issues. (#15298)
4.8.2 C# Specific Changes
No C# Specific Changes
4.8.2 Common Changes
Fixed: Setting the minimum log level to WARN or ERROR will now correctly filter out very early logger initialization messages. (#14451)
Fixed: A bug that could prevent an attachment created with the same data on multiple peers from syncing. (#14481)
4.8.1 C# Specific Changes
Fixed: iOS devices may fail to sync on LAN when the app is backgrounded then foregrounded. (#14150)
4.8.1 Common Changes
Added: The presence_use_multihop system parameter can be used to disable replicating presence information to other peers in the mesh, reducing network overhead. Use ONLY with guidance from the Ditto Customer Experience team. (#14128)
Changed: When failing to load document data sync (replication) metadata, Ditto now attempts to clear and regenerate the metadata. This is a behavior change where previously Ditto would fail to start. This serves to prevent problems after SDK version downgrades. (#13111)
Fixed: A rare issue where attachments may not be transferred between two peers.(#14158)
Fixed: On-disk logs are more robust in situations with unexpected Ditto shutdown. (#13723)
4.8.0 C# Specific Changes
Added: BLE and Wi-Fi Aware transports on Android. #11288
Added: method ExportToFileAsync() to DittoLogger, which exports collected logs to a compressed and JSON-encoded file on the local file system. Logs returned through this method are at DEBUG log level. This new API can be used to do forensic log gathering on a device. Logs are always collected and restricted to a fixed specific size limit to ensure they don’t grow endlessly. Once the size limit has reached new logs replace the oldest logs. #11961
Added: Properties PeerKeyString1 and PeerKeyString2 on DittoConnection replacing the deprecated properties Peer1 and Peer2. This is to align with our usage of the PeerKey string identifer across the system in place of the Peer object. #12799
Added: Method GetAllConnectionsByID on DittoPresenceGraph providing easy access to all connections in the presence graph, grouped by their IDs. #12799
Added: Documentation guidance for StartSync() and DisableSyncWithV3() to include a recommendation for performance improvement. #13098
Performance of initial sync when bootstrapping a new peer can be improved by calling DisableSyncWithV3() before calling StartSync(). Only do this when all peers in the mesh are known to be running Ditto v4 or higher.
Added: Documentation for property PeerMetadata on DittoPeer to add information about how the property behaves over the lifecycle of the DittoPresenceGraph. #13479
Added: Login method to the DittoAuthenticator that provides access to authentication feedback on the returned clientInfoJson string. #4804
Changed: (Linux) Increased mDNS multicast TTL from 1 to 255 to allow for cross-VLAN announcements #13590
Fixed: No longer losing precision while working with larger floating-point numbers. #13229
Deprecated: Properties Peer1 and Peer2 on DittoConnection, please use PeerKeyString1 and PeerKeyString2 instead #12799
Deprecated: Method LoginWithToken from DittoAuthenticator. Use the newly added Login method instead. #4804
4.8.0 Common Changes
Changed: The Logging output format for Ditto SDKs has been changed in 4.8.0 as part of our process to improve readablility and observability across the Ditto platform. Customers should validate the new data format in their existing data ingestion pipelines.
New Logging Schema:
YYYY-MM-DDTHH:MM:SS.ssssss LEVEL TARGET: MESSAGE {field1=value1, field2=value2, …}
-
Timestamp (YYYY-MM-DDTHH:MM.ssssss):
-
Format: ISO 8601 format (e.g., 2024-08-16T14:23:05.123456Z).
-
Description: The exact time when the log entry was generated.
-
Log Level (LEVEL):
-
Format: One of the following levels— DEBUG, INFO, WARNING, ERROR.
-
Description: Indicates the severity or importance of the log entry.
-
Target (TARGET):
-
Format: A string, representing the internal module/area
-
Description: The source of the log entry, often corresponding to the internal module where the error was called. (e.g. ditto_init or peer_event_listener)
-
Message (MESSAGE):
-
Format: A human-readable string.
-
Description: The main content of the log entry.
-
Structured Fields ({field1=value1, field2=value2, …}):
-
Format: A list of key-value pairs enclosed in curly braces.
-
Description: Any additional structured data associated with the log entry. These fields can include custom data that you attach to the log entry using the tracing::info!(field1 = value1, “Message”) syntax.
4.7 and earlier Logging Format Example
[INFO] 2024-08-02T10:25:39.981Z: AuthClient: using cached SiteId; site_id = 16248337916605943930
4.8.0 and later Logging Format Example
2024-08-02T11:32:25.905562+01:00 INFO ditto_init:new_auth_client: ditto_auth: using cached SiteId site_id=16248337916605943930
Added: DQL support for string functions byte_length and char_length #12147
Added: DQL support for inline object and array literal creation #12312
Added: object_length and deserialize_json functions in DQL #12605
Added: DQL INSERT statements support inlining the construction of an object #12761
Added: Remote query capability allows Big Peer to execute read-only queries on Small Peers. Usagne currently internal to Ditto Customer Experience team members #12268
Added: Diagnostics are now provided when collecting Small Peer Info on-disk logs #12080
Added: output version, sdk and build information on startup for easier debugging #12845
Added: Introduced new internal capabilities ensure a large data payload (e.g. a large attachment file) cannot use all the bandwidth and prevent other data from being synced.
Added: mDNS transport can now be configured with a static port for the TCP server using the tcp_server_bind_mdns_server_port system parameter #13442
Changed: New logging format that includes detailed information #10794
Changed: The Ditto logger is now more verbose by default, with a minimum log level of INFO instead of WARNING #11141
Changed: small peer info collection is now enabled by default and its default sync scope is now BigPeerOnly #12709 (Device dashboard will populate information automatically)
Changed: (Experimental) Multihop connections are formed more efficiently, e.g., for Ditto Bus. This is an incompatible protocol change from previous versions #12108, #12623
Changed: Improve performance by avoiding deserializing the entire Document if only part of it is required #10887
Changed: Attachments now sync independently of documents, improving rainbow connection efficiency #11156
Changed: Updating several depdencies for security patching (Patch multiple old packages #12087)
Changed: How network OS capabilities are internally linked into the library to allow for more relaiable and reusable internal logic #12362
Changed: API documentation for execute method to include information about its interaction with sync subscriptions #12365
Changed: Default to creating backup Bluetooth connections when there is an existing WiFi connection (Default mesh_chooser_avoid_redundant_bluetooth to false #13398)
Changed: console logging will now output to STDERR instead of STDOUT ([logs] logging now outputs to STDERR instead of STDOUT #13477)
Fixed: Bytes and Arrays not being always comparable in the legacy query language #10887
Fixed: Legacy query language casting counters and maps with tombstones to the wrong boolean value #10887
Fixed: Lowered the log level of a few SQLite3 notices that were being unnecessarily logged as warnings #12032
Fixed: A crash that can occur if a DQL query contains a multi-byte character #12324
Fixed: An issue with the transports multiplexer that could lead to a hung process. This occured when dealing with large messages spanning multiple chunks (>64KB) #12428
Fixed: A rare edge case that could slow down data sync betten peers #12527
Fixed: Frequent subscription changes along with eviction calls could cause failure to converge on data leading to inconsistent data state #11431
Fixed: Memory leak in third party library used for peer-internal messaging #12895
Fixed: Transport inactive connection tracking no longer stores identifiers indefinitely. This could lead to a slow growth in memory consumption for long lived applications (~ 1MB/day on transports with high rate, unreliable discovery - such as BLE) #13147
Fixed: Unexpected failures to obtain a certificate for peer-to-peer sync are now logged at ERROR level instead of WARNING #13323
Removed: (Experimental) Ditto no longer supports the experimental Query Overlap Groups feature and support has been removed. This experimental feature has been removed after running a determining a lack of feasability in production scenarios. The team is iterating and plan on interoducing improved capabilitites. #13075
ADDED: Support for specifying the mDNS connection ports for Small Peer SDK instances
Allows you to specify a specifc port that can will be used for mDNS announcements and connections on a given device. This can be used for connect with peers across a VLAN while scanning for open ports.
Not setting the config will result in the current default behavior of a random port being assigned.
If running multiple application on the same device, for example a debug and/or test flight application along side a production application, each application needs to have a unique port.
Failure to do this will result in only one application being able to establish connections at a given time.
Setting the config
The config can be set using DQL ALTER SYSTEM
and the tcp_server_bind_mdns_server_port
config.
This command should be run using the ditto.store.execute(...)
command within your application.
tcp_server_bind_mdns_server_port
needs to be set before startSync()
is called to be applied.
FIXED: A finalizer issue that was causing crashes when closing sync subscriptions (#13849, #13732)
Resolves a race condition that could be hit when closing a sync subscription or stopping sync that caused Ditto to crash.
IMPROVED: (Linux) Increased mDNS multicast TTL from 1 to 255 to allow for cross-VLAN announcements (#13590)
To support scenarios with cross-VLAN device discovery we increated the TTL to ensure message have enough time to reach their destination and get acknowledged.
FIXED: An issue causing a race condition with a Live Query callback. (#13180)
This fix addresses the issue causing a Live Query callback to receive a delta update, known as a diff, before the initial document.
FIXED: A compatibility issue between documents created using different CRDT versions. (#13236)
Previously, there was an issue in which two documents on different internal versions could fail to integrate changes created using an older CRDT version.
IMPROVED: Logging for batch write transactions. (#13339, #13339)
Due to a race condition caused by the logging system mistakenly identifying the previous transaction as blocking the current one, the logger would inaccurately indicate the current transaction as being blocked by the previous one from an unknown
originator.
FIXED: An issue in the Small Peer datastore causing slow performance or process termination. (#13395)
This issue was due to an aging dependency with a race condition, which caused thread tasks to get stuck and transactions to be blocked for extended periods.
IMPROVED: The Ditto SDKs compilation performance. (#13174)
We’ve optimized compilation by reducing the size of each binary slice of the Ditto SDKs by 19% and fixing incompatibility with PLCrashReporter
.
ADDED: mesh_chooser_avoid_redundant_bluetooth
system parameter. (#12984)
This new system parameter allows greater control over the use of Bluetooth Low Energy (LE) connections in Ditto’s peer-to-peer mesh network. When set to false
, Ditto creates Bluetooth LE connections alongside other peer-to-peer connections, serving as a fallback if the primary transport, such as Apple Wireless Direct Link (AWDL) or LAN, fails.
However, before you enable the mesh_chooser_avoid_redundant_bluetooth
parameter, consider the following technical tradeoffs:
- Benefits:
- Provides additional fallback to maintain connection stability when primary connections cease to function.
- Reduces time to establish new connections since, by having multiple fallback options, Ditto can more quickly re-establish connections.
- Drawbacks:
- Increases network traffic since using multiple transports results in additional network overhead
- Using Bluetooth LE alongside other transports increases resource consumption. In the worst case, enabling this setting can result in slower sync performance and battery drain.
To enable the mesh_chooser_avoid_redundant_bluetooth
parameter, use the following DQL query:
ALTER SYSTEM SET mesh_chooser_avoid_redundant_bluetooth = false
CHANGED: The AuthClient
by improving the log message for certificate issues returned from the Big Peer Authentication Service. (#13138)
FIXED: A performance-related issue affecting bulk eviction operations for documents with attachments. (#12331)
ADDED: Improved data compression in the handshake protocol when establishing connections to other peers. (#11264)
This allows peers with large permission sets to establish connections on low bandwidth transports like Bluetooth LE more quickly.
FIXED: The HTTP protocol by adding content-length: 0 header
compliance in empty POST
requests. (#12346)
Before, if you invoked an empty POST
request, the cloud load balancer would reject your request with a 411 - Length Required
error message.
Now you must explicitly state that the body of your POST
request is empty by setting content-length: 0
in the header of your HTTP API call.
FIXED: Handling of malformed subscription queries in the legacy query builder API by adding validation logic early to ensure invalid queries are identified early and logged. (#12463)
There was an issue where invoking subscribe()
on an invalid query caused sync to be silently disabled.
Invalid queries in the subscribe()
method are now identified, disabled, and logged as errors.
FIXED: Subscription assumption logic so all acknowledgments are processed, regardless of subscription changes. (#12541)
In scenarios where subscriptions rapidly changed back and forth while documents were being concurrently altered, documents would diverge at merge.
We’ve removed subscription logic ignoring acknowledgements based on assumed subscription mismatch and introduced salting functionality to ensure each document is unique for each occurrence of a subscription. This helps differentiate between new and reverted subscription states.
FIXED: Sync inefficiency where changing a subscription resulted in unnecessary syncing of redundant update files. (#12967)
To eliminate redundant data from being synced unnecessarily, regardless of whether the remote peer previously acknowledged them, we’ve added validation checks ensuring that only unacknowledged data is synced.
This fix reduces sync update file sizes, leading to more performant session sync updates, particularly in apps consisting of large, stable subscriptions and infrequent document changes.
FIXED: An issue causing DittoValidationException
to attempt to fetch an attachment from a dictionary representation. (#12668)
FIXED: The DittoPeer.IsDittoCloudConnected
property to correctly reflect the actual connection status. (#13062)
Fixed: Misleading errors related to attachment fetch operations. Now, associated error logs are more accurate and clear. (#12409)
Fixed: A specific issue where incorrect sync metadata rollback caused problems with data consistency at merge. (#12422)
Fixed: Handling of document paths so non-alphanumeric and non-underscore characters function without causing errors. (#12556)
Previous versions may have removed these characters from document paths.
FIXED: The legacy query language error-throwing mechanism to accept invalid field names properly. (#11888)
After upgrading to version 4.7 of the SDK, you may notice unexpected errors while using the legacy query builder language. These errors are likely due to invalid field names or path expressions previously undetected.
In addition, make sure to start unquoted_string
with an underscore (_
), a letter, or an alphanumeric character. For example, the following equivalent regular expression: [A-Za-z_][A-Za-z0-9_]*
.
In the legacy query builder language, invalid field names and path expressions now throw explicit errors. Previously these invalid names did not throw and would cause undefined behavior.
Note: This new error-throwing behavior applies only to the legacy query builder language; DQL is unaffected by this improvement.
REMOVED: The is_connected_to_big_peer
field from the Small Peer Info document. (#11883)
CHANGED: The small peer info JSON metadata by increasing the depth limit to 64 levels. (#11891)
IMPROVED: Transaction contention time for the ATTACHMENT
data type, reducing potential timeouts. (#11893)
Before this improvement, concurrent write transactions were long-lived, potentially causing deadlocks and request timeouts. Now you’ll experience more effective handling of multiple attachments being processed at the same time.
FIXED: The issue causing sync crashes due to missing update files. (#11952)
With the latest release, Ditto handles unexpected situations or errors in a way that allows it to recover and continue functioning without crashing or causing disruptions.
ADDED: Parameters for adjusting certain settings during runtime
This rollout includes new system parameters for the MeshChooser feature for configuring a limited set of variables, allowing you to adjust certain settings during runtime.
RESOLVED: The warning message when peers connect is no more. (#11977)
The warning message previously logged when peers connected within the mesh is no longer recorded or printed.
IMPROVED: Sync performance by sending the entire state of the data. (#12029)
Previously, when syncing in mixed mesh environments from Ditto SDK version 4.0.0 of the SDK to version 3.0.0, only partial updates were transmitted. With this release, all updates are transmitted, resulting in greater data consistency and completeness when syncing across different versions of the SDK.
FIXED: The issue causing the Small Peer Info collector to become stuck. (#12051)
Previously, the Small Peer Info collector would become stuck when attempting to upload logs in certain situations. Since resolving this issue, the Small Peer Info collector functions properly in those conditions.
FIXED: The issue causing fetch operations to appear unresponsive. (#9501, #12067)
With this fix, operations related to fetching attachments function as expected.
FIXED: The issue causing type check failures when setting duration parameters through ALTER SYSTEM SET. (#12070)
When using SystemParameter
to configure timeouts before this fix, if using different types to encode the values representing the timeout, for instance, Milliseconds
and Seconds
, the values failed to downcast, an error log printed, and the new value ignored.
Now, when using mismatched wrapper types, Ditto performs a compatibility check, validates, and, if needed, converts them as appropriate.
FIXED: Peer-to-peer LAN now advertises on newly added interfaces, such as enabling Wi-Fi at runtime. (#10265)
This addresses an issue where mDNS on Android was slow to recover when transitioning Wi-Fi from disabled to enabled.
ADDED: A warning notifying that the DeviceName
changes after sync start. (#11160)
We’ve added a warning message that logs when the DeviceName
property is set after the sync process has started.
Previously, setting DeviceName
of a Ditto instance after starting sync did not trigger a warning.
This new message warns you that the new value you’ve set only takes effect after you’ve restarted the sync process in your app.
FIXED: The DeviceName
property on Ditto instances now reflects the truncation of the value to 24 bytes when starting sync. (#11611)
ADDED: Properties PeerKeyString
property to DittoPeer
as a replacement for the now deprecated PeerKey
. (#11611)
ADDED: Added the PeerMetadataJsonString
property and the SetPeerMetadataJsonString
method to Ditto.Presence
. (#11611)
ADDED: Introduced the PeerMetadata
property and the SetPeerMetadata
method to Ditto.Presence
. (#11611)
ADDED: Added properties PeerMetadata
and IdentityServiceMetadata
to DittoPeer
. (#11611)
ADDED: The ConnectionRequestHandler
property in Ditto.Presence
, allowing filtering of incoming connections from other peers.
DEPRECATED: The DittoPresence.Exec()
method has been deprecated in favor of using the newly introduced Graph
property on Ditto.Presence
. (#11611)
DEPRECATED: The PeerKey
property on DittoPeer
. (#11611)
From now on, use the newly added PeerKeyString
property instead.