Release Notes

Fixed:

• Fixed query failure caused by use of id as a field name in DQL queries

Big Peer 1.47.0 enhances system reliability with critical fixes for CDC session persistence and connection stability, while introducing powerful new DQL capabilities including virtual collections, scalar functions, and dynamic field references.

Added:

• DQL projection, distinct, and aggregate features are now out of preview and ready for production use
• DQL SYSTEM:COLLECTIONS and SYSTEM:ALL_COLLECTIONS virtual collections for listing accessible collections
• DQL scalar functions for duration calculations and time-based operations
• Support for dynamic field references in DQL statements, enabling more flexible queries
• Virtual collections for monitoring active and historical DQL engine requests
• Support for unary plus operator in DQL expressions
• Wildcard expansion now supports top-level fields with left-to-right, last-wins projection aliasing
• The USE IDS clause now accepts expressions that can be statically evaluated

Fixed:

• CDC sessions are now properly persisted across immediate restarts, preventing session loss
• UPDATE statements can now modify multiple subfields within a single operation (e.g., SET a.b = 1, a.c = 2)
• CDC can now automatically split large individual change payloads to ensure reliable data transmission, even with large documents
• Resolved a TLS public key race condition that could cause connection failures
• Syntax error identification in UPDATE statements now provides clearer guidance

Ditto Server 1.45.2 expands query flexibility with configurable DQL_STRICT_MODE, delivers powerful new DQL features such as DISTINCT selection and advanced scalar functions, and improves performance and reliability across replication, storage and query execution.

Added:

• DQL_STRICT_MODE can now be toggled directly from the DQL editor
• DQL DISTINCT selection support for deduplicating query results
• Additional DQL scalar functions for advanced object manipulation, and support for passing an ARRAY of values to the IN operator.
• Document IDs generated via the /store/write HTTP API now use the more compact ObjectID format (previously UUIDv4).
• DELETE statements and automatic tombstone eviction are now generally available on Big Peer
• DQL date functions accept the special timezone value "local", evaluating in the executing peer’s local timezone
• Connected SDK clients automatically migrate to CRDT v6 on first sync with a Big Peer
• Added the v5/store/execute HTTP API endpoint, which operates with DQL_STRICT_MODE=false semantics

Deprecated:

• Legacy Small Peers (pre v2 and below) can no longer authenticate to Online With Authentication or Online Playground modes
• Collection aliasing in INSERT statements is no longer supported

Fixed:

• DQL_STRICT_MODE=false is now the default for CDC pipelines, Big‑Peer‑to‑Big‑Peer subscriptions and Data Bridge filters, reducing the need for manual configuration.
• Resolved an issue where queries protected by WITH PERMISSION clauses could short‑circuit and omit valid results
• Corrected index‑planning for conditional queries using functions such as IS NULL or IS NOT MISSING, preventing documents from being skipped
• SDK clients now migrate more reliably to the latest CRDT version during initial synchronisation

Ditto Server 1.42.3 introduces several new features, including CDC Data Bridges, automatic tombstone eviction and numerous DQL capabilities.

Added:

• CDC now supports Data Bridges, allowing the creation of multiple individually filtered Kafka topics or webhook integrations
• Ditto Server is now able to automatically evict tombstones produced from document DELETEs. This can be enabled on request by contacting Ditto Support
• DQL EXPLAIN to show query plans
• DQL CAST function to explicitly cast between types
• DQL clause USE IDS for direct document retrieval by ID
• DQL SIMILAR TO filter expression
• New DQL DATE processing functions
• New DQL string functions
• DQL PN_COUNTER support (APPLY clause in UPDATE statement)
• DQL support for UNSET clause in UPDATE statements
• DQL MERGE now acts as an alias for UPDATE in the ON ID CONFLICT clause of an INSERT statement
• DQL support to use runtime-evaluated expressions in object constructs in DQL statements
• DQL support for static expressions in mutators
• DQL INSERT now accepts either DOCUMENTS or VALUES as the keyword introducing the list of values to insert
• DQL now has the ability to use runtime-evaluated expressions in array constructs in DQL statements

Fixed:

• Improved Ditto Server query timeout behaviour
• Various CDC reliability improvements
• Improvements to Ditto Server memory management and resilience
• Improved sync performance in memory constrained scenarios

Ditto Server 1.41.0 delivers major performance enhancements for heavy workloads, introduces powerful new DQL features, and includes numerous optimizations and fixes.

Added:

• Major performance improvements and resource usage reductions when under load
• Support for LIMIT, OFFSET and ORDER BY for mutating DQL statements
• Support for the LIKE DQL operator
• Support for object_keys and object_values scalar DQL functions
• Improved performance for DQL queries with IS NOT MISSING predicates on MAP fields
• Improved concurrency on Ditto Server’s subscription server
• Improved performance around document eviction

Fixed:

• Addresses an issue where the legacy query language sometimes failed to return appropriate results using the @ Operator
• Addresses an consistency issue when performing DQL comparison between large floats and integers
• Optimizations to authentication and device user management

Ditto Server 1.40 introduces an overhaul of the underlying data structures of Ditto Server’s store, resulting in substantial improvement to performance and reliability, while unlocking new capabilities for upcoming features.

In addition, 1.40 contains a number new improvements and fixes.

Added:

• A new data architecture for Ditto Server, improving reliability, performance and unlocking future capabilities
• Ditto Server evictions are now available as a preview feature. Contact us to enable this feature for your instances
• Synchronization protocol enhanced to reduce the amount of redundant data sent from small peers to Ditto Servers
• Bump the minimum supported CRDT version for Ditto Server apps to V6
• Several improvements to Ditto Server reliability and deployment efficacy

Fixed:

• Addresses a rare issue decompressing attachments during replication
• Performance optimisations made to document deduplication on the subserver
• Device name no longer incorrectly defaults to “ditto” in device table, when a named device first starts syncing
• Fixed an issue where HTTP API calls might incorrectly fail when passing a transaction ID as part of the call
• Fixes a potential deadlock condition in Ditto Server
• Align Kafka producer and consumer message sizes to prevent potential backpressure scenarios
• Network policy improvements to enable new feature development

Added:

• Networking policy enhancements to support architectural improvements

Fixed:

• Addresses a rare issue decompressing attachments during replication

Added:

• Improved the API Key validation of the HTTP API to now automatically retry a number of times in case of transient issues, before throwing an error to the caller

Fixed:

• An issue where HTTP API calls might incorrectly fail when passing a transaction ID as part of the call
• A failure when executing remote queries that contained non-ASCII characters in the SYNC CONTEXT clause
• An issue that prevented attachment replication when a large number of documents were also being simultaneously replicated
• A number of issues related to data convergence during replication in rare scenarios

Added:

• Improved the API Key validation of the HTTP API to now automatically retry a number of times in case of transient issues, before throwing an error to the caller

Added:

• DQL INSERT statements run via the HTTP API now support inline object literals.
• The /api/v4/attachments/ HTTP endpoint now supports HEAD requests to check for attachment’s existence.
• Improved the performance of subscriptions with limits.
• Improved the performance of replication during garbage collection.

Fixed:

• Fixed a failure in replication in cases where there are frequent subscription changes and on-device evictions.
• Fixed an issue which could cause a crash during document merges when using older versions of the SDK.

Added:

• Improved query performance for DQL queries that access attachments, through the HTTP data API.
• Improved handling of slow WebHook targets by enforcing a 30-second timeout to outgoing WebHook calls, this prevents lockups and other unexpected behavior when using WebHook connections.
• Improved replication to avoid resending metadata, resulting in a reduction in overall replication traffic and processing when sync subscriptions change.

Added:

• Added support for inline object and array definition to DQL through the HTTP API.
• Added object_length() and deserialize_json() functions to DQL through the HTTP API.
• Improved the error message for cases where the query result is too large.
• Removed the //distinct_values HTTP endpoint.
• Removed support for X-DITTO-CLIENT-ID, X-DITTO-ACTOR-ID, X-HYDRA-CLIENT-ID from the HTTP API.

Fixed:

• Fixed a hang when replicating documents or attachments larger than 64KB.
• Fixed the default HTTP API response format to correctly be application/json if not specified in the request.
• Fixed a rare edge case that could slow down sync when using very small documents.

• Synchronized version bump release – No changes.

Fixed:

• Potential crash as a result of a DQL query containing a multi-byte character. (#12324)
• A hang in the transports multiplexer when handling large messages spanning multiple chunks larger than 64KB. (#12428)

Added:

• Attachments now sync independently of documents, improving rainbow connection efficiency (#11156)

Fixed:

• Sync crashes due to missing update files and other unexpected conditions are now handled recoverably (#11952)

Added:

• Improve performance by avoiding deserializing the entire Document if only part of it is required. (#10887)
• Removed is_connected_to_big_peer field from the Small Peer Info document. (#11883)
• Increased the limit on the allowable depth of the Small Peer Info JSON metadata. (#11891)
• Attachments are more performant when working with many attachments concurrently
• Spread work more evenly across Ditto Server store writer threads. (#11939)
• MeshChooser now supports a limited set of runtime configurable values via the mesh_chooser_* system parameters. (#11968)
• String functions byte_length and char_length. (#12147)

Fixed:

• Resolved legacy query language casting counters and maps with tombstones to the wrong boolean value (#10887)
• Resolved Bytes and Arrays not being always comparable in the legacy query language. (#10887)
• Resolved UST initial polling unbounded error retry. (#11841)
• Resolve legacy query language accepting invalid field names. (#11888)
• Resolved HTTP API OpenAI spec that will cause duplicated Authentication header in certain generated clients. (#11936)
• A warning message that appears when peers connect is no longer logged (No handle found to notify the shutdown; multiple message handling loops??) (#11977)
• When replicating in mixed meshes from V4 to V3 full state will be sent. (#12029)
• Lowered the log level of some SQLite3 notices that were being unnecessarily logged as warnings. (#12032)
• The Small Peer Info collector no longer gets stuck waiting to upload logs under certain conditions. (#12070)
• Duration parameters can now be set via DQL, whereas before they would fail to type check. (#12070)
• Ensure document cache GC always progresses regardless of configured time slice and load. (#12091)

Fixed:

• Revert “Fix DocumentCache cache wrong result for Sort + Limit queries. (#9435)
• Query plan for queries using the contains function with an empty array. (#12010)

Fixed:

• Stream-splitter consumer migration applying old offset. (#11985)
• Query plan for queries using the contains function with an empty array. (#12010)

Added:

• Update stream-splitter to respect Ditto login retry interval and attempts. (#11785)

Fixed:

• Stream-splitter checks for kafka.enabled before spawning user-consumable streams. (#11788)
• Tweak hydra-cdc-heartbeat to avoid stuck consumer. (#11815)
• Application crashes while preparing to syncing data to remote peer. (#11843)
• Fix Subserver caching wrong result for queries with sort and limit. (#9435)

Added:

• Improve UST polling resilience in HTTP API and subserver (#11701)

Added:

• DQL queries can now query for the id, len and metadata fields of attachments (#11481)

Fixed:

• Do not error in DQL inserts with the DO NOTHING conflict policy (#11730)

Added:

• Synchronization protocol enhanced to prevent accumulation of unsubscribed documents after subscription change, and limits redundant data exchanged when re-synchronizing. (#10440)
• Reduce metadata lock contention while applying transactions (Avoid metadata read lock in apply_txns_internal #11654)

Fixed:

• Fix race between applying of transactions and reconfiguration of replicas (#11361)

Fixed:

• Legacy query language casting counters and maps with tombstones to the wrong boolean value
• Legacy query language not allowing comparison against attachment objects.
• Bug where certificate request would not check identity data hash with JWT’s
• Bug where the certificate request would fail with big permissions

Added:

• Improve read query performance of legacy contains([..], _id) syntax (#10987)
• Parser error messages emphasize the location of the error and de-emphasize a potential fix. (#11040)
• Reduce HTTP API latency with improved lock map version tracking (#11278)
• Improved read query performance for DQL queries using array_contains (#11338)

Fixed:

• An issue where an internal data decoding function could panic on certain inputs (#10376)
• Return 500 instead of 401 upon internal error while validating API key. (#11178)
• A bug where Link could run into an infinite loop. (#11188)

Added:

• Make actix-web keep alive timeout configurable for Ditto Server (#9578)

Added:

• Preliminary support for runtime configuration using DQL ALTER SYSTEM and SHOW statements (Hook up DQL entrypoint to parameter store #10401)

Added:

• Peers use latest supported protocol version, even if the application is using backwards-compatible CRDT formats. (Disentangle protocol version from protocol flags #10875)

Fixed:

• Fix deadlock when replicating attachments to Ditto Server (Fix AttachmentsPeer deadlock when setting up outbound backoff #11036)

Added:

• Improve read query performance on queries with multiple ranges (#10605)

Fixed:

• Fix incorrect DQL literal comparison behavior in Ditto Server planning. This fixes a correctness issue where Ditto Server could return no results when it should return all documents in a collection. This also makes Ditto Server avoid a full scan in certain cases where it should return no results. (#10940)

Added:

• SDK version/language/platform and device name are now included in the small peer info document. (#10515)

Added:

• Improved read query performance by pre-filtering non-matching document ids (#10620)
• Improve memory usage and performance of Documents (#10736)
• Performance by implementing bipartite distance-aware mesh topology (#10632)

Fixed:

• Mesh would choose random connections (#10711)
• Fix NOT on non-Boolean panics on Ditto Server (#10762)
• Fixed a bug with Ditto’s internal crash handler (#10022)
• Initial logging events now use the same format as all other logs (#10611)

Fixed:

• Fix hashing bug in Ditto Server store node controller

Fixed:

• Fix a bug preventing updates to clusters from rolling out properly.

Fixed:

• Fix bug related to Kafka transaction chunking

Fixed:

• Improved run-time validation of corrupted attachments being inserted into documents (#10552)

• Synchronized version bump release – No changes.

Deprecated:

• Deprecated experimental time-series HTTP APIs

Fixed:

• Ditto Server queries matching on nested objects or arrays return the expected documents (#10382)
• Addressed a warning from SQLite that sometimes occurs on startup about a schema change (#10501)

Fixed:

• Fixed HTTP API output format for Attachments when using SerializedAs::LatestValues

Added:

• Remove doc_ids from requery_required message in CDC
• Reduced metadata accumulation after eviction by clearing remote summaries

Added:

• Add attachments support for HTTP API RPC v4

Added:

• Optimized Ditto Server internal data serialization performance.
• Optimized the updates of internal system collections when doing writes with the HTTP RPC API.

Fixed:

• Fixed a bug around where queries could return an older version of documents around a server reboot.

Added:

• Add RPC V4 HTTP API.
• Update our OpenAPI specification to include the new count request.

Added:

• Made the Ditto Server more resilient to Kafka broker disconnects
• Improve our HTTP API implementation to reduce Timeout related errors

Fixed:

• Fix bug in Ditto Server device certificate CA rotation process
• Improved performance when multiple peers with distinct permissions work with the same cloud endpoint

Fixed:

• Various fixes and improvements for better reliability, security, and performance

Fixed:

• Minor bug fixes and performance improvements.

Added:

• Changed: new serialization format improves efficiency of replication, particularly in the Ditto Server

Fixed:

• Fix detection of broken stream when there are multiple apps in the transaction log

Added:

• Initial release of the lazy deserialization optimization for the Ditto Server.
• Tighten network policies for increased security and stability

Fixed:

• Fix bug where Ditto Server would sometimes panic if an SDK peer managed to create multiple Ditto instances with the same persistence directory. We are seeing this error more than we originally hoped. This change makes any occurrence non-fatal until we fix the root cause. There will be another followup change that will re-enable alerting for this event.

Fixed:

• Fix bug where HTTP API upserts would not always be reflected in subsequent find requests.

Added:

• Properly consider limit+offset+order_by for active queries

Fixed:

• Minor bug fixes and performance improvements

Added:

• Supports Ditto SDK through version 2.0.x.

Fixed:

• Minor bug fixes and performance improvements

Fixed:

• Minor bug fixes and performance improvements.

Added:

• Add collection and document id to RequeryRequired requests as part of the live query WebHook and CDC.

Fixed:

• Minor bug fixes and performance improvements

Fixed:

• Minor bug fixes and performance improvements

Fixed:

• Minor bug fixes and performance improvements

Fixed:

• Minor bug fixes and performance improvements

Added:

• Supports Ditto SDK through version 1.1.6
• Parity between document and timeseries HTTP APIs
• HTTP API calls that were GET with a body now also accept POST. They will accept both methods and will behave the same no matter which method is used.

Added:

• Supports Ditto SDK through version 1.1.5, including improvements to replication efficiency.

Fixed:

• Adds HTTP API RPC Remove and Update commands (remove + update). With remove, users may provide a query and delete all matching records. The update command mimics Ditto SDK update behavior and applies a series of updates to all matching documents.
• The old header names are now deprecated (All HTTP API Headers used by Ditto will be prefixed with X-DITTO-…).