Documentation Index
Fetch the complete documentation index at: https://docs.ditto.live/llms.txt
Use this file to discover all available pages before exploring further.
5.0 - Built for Speed and Developer Experience
Ditto 5.0 brings significant performance improvements and a developer experience redesigned for usability. Your applications run faster with optimized queries and sync, while modern APIs and simplified patterns eliminate complexity at every turn.Faster local queries
Automatic optimizations make your apps more responsive with zero code changes
No more schemas
Start building immediately without upfront type definitions
Simpler initialization
Clear, modern patterns replace confusing legacy APIs
One query language
DQL handles all data operations consistently across platforms
Built-in observability
Query system metrics and configuration directly with DQL
25% smaller footprint
Faster builds, lower memory usage, cleaner codebase
Plus much more
Advanced query features, networking improvements, and enhanced reliability
Our Most Rigorously Tested Release
Ditto 5.0 has undergone extensive validation in our internal mesh lab, ensuring reliability across real-world scenarios before reaching production.Testing scope:- Multi-device mesh scenarios - Validated with up to 40 concurrent devices across iOS, Android, and budget hardware
- Cross-platform compatibility - Tested heterogeneous meshes mixing iOS, Android, and version combinations
- Extended reliability testing - 24-hour continuous operation tests validating memory stability and connection resilience
- Real-world datasets - From small 1MB datasets to large 50MB+ product catalogs
- Network conditions - LAN, BLE, WiFi Aware, AWDL, and mixed transport scenarios
- Lifecycle edge cases - Background/foreground transitions, network partitions, and recovery scenarios
In This Release
This is a major version with breaking changes, but maintains backwards compatibility with v4 deployments (4.11+). Migrate at your own pace, and upgrade to v4.14 first for the smoothest transition.DQL for All Data Operations:Core Capabilities:Performance & Platform:- DQL Query Performance Improvements
- Data Sync Performance Improvements
- Local System Observability
- Additional Improvements
DQL for All Data Operations
DQL for All Data Operations
Ditto 5.0 completes the transition to DQL (Ditto Query Language) as the single API for all data operations. The legacy query builder has been removed.What Changed
- Legacy query builder removed: All
store.collection()methods and fluent query APIs are no longer available - Full feature parity: Every legacy operation has a DQL equivalent
- Single query language: DQL handles reads, writes, subscriptions, and observers
- Works everywhere: Same syntax across mobile, server, and web
- SQL-familiar: Standard SQL patterns for immediate productivity
Why One Query Language
A unified query language means one implementation to maintain, faster feature delivery, and consistent behavior across all platforms. New capabilities and optimizations benefit every SDK simultaneously.Migration Path
All legacy query operations have direct DQL equivalents:Update Operations:Complete migration examples for all legacy query patterns are available in the Legacy to DQL Migration Guide.
Core Improvements
Simplified Initialization & Configuration
Ditto 5.0 introduces a completely redesigned initialization flow built around the newDittoConfig pattern. This replaces the previous Identity-based approach with a clearer, more predictable setup that aligns with modern best practices.What’s New
The new configuration system provides:- Unified configuration object: All initialization parameters are set through
DittoConfigfactory methods - Fallible initialization: Explicit error handling during setup catches configuration issues early
- Asynchronous patterns: Native async/await support where appropriate for each platform
- Simplified authentication: Clearer authorization client that’s easier to understand and implement
What This Solves
The previous initialization flow required multiple unintuitive settings due to legacy compatibility concerns. For example, developers had to setdisableCloudSync = true to connect to a Big Peer, which was confusing and non-obvious.The new pattern consolidates all configuration into a single, coherent flow that makes the relationship between settings explicit and easier to reason about.Accessing Configuration at Runtime
After initializing Ditto, you can access the configuration object to retrieve settings like the database ID. This replaces methods likegetAppID() from v4:Availability
TheDittoConfig pattern was introduced as an option in v4.12 and becomes the only initialization method in v5.0.Migration guides for each SDK are available in the v5 documentation. If you’re currently on v4.11 or later, the migration path is straightforward.
Migration Example
The initialization flow has changed fromIdentity-based to DittoConfig-based patterns. Here’s how to migrate:Accessing Configuration at Runtime
After initialization, you can access your Ditto configuration to retrieve settings like the database ID:Schema-Free Data Modeling
Ditto 5.0 transforms the developer experience by making DQL strict mode disabled by default. This eliminates the need to define collection schemas or CRDT types upfront, allowing you to insert nested objects and data structures without pre-defining types.What’s New
With strict mode disabled by default, Ditto changes how objects are stored and synchronized:Objects default to MAP type instead of REGISTER typeThis fundamental change means:- Field-level sync: Ditto syncs individual field changes instead of replacing entire objects
- Automatic type inference: No need to pre-define collection schemas or CRDT types
- Nested structures: Insert complex JSON-like documents without type definitions
- Concurrent updates merge: When peers update different fields simultaneously, both changes are preserved
What This Solves
This change provides a more simplified data management structure with two key benefits:- Add-wins behavior on objects: When peers create or modify objects, additions are preserved rather than overwritten
- Field-level delta sync on all nodes: Every field in your document syncs independently, reducing bandwidth and enabling fine-grained conflict resolution throughout the entire document structure
New Customers
Schema-free data modeling is enabled by default in Ditto 5.0. No action is needed—simply start building with automatic type inference and field-level sync.Migrating Customers
Migration Considerations
Strict mode is a local configuration setting on each device that controls how DQL interprets and writes data structures.How it works across peers:- Data syncs successfully between peers regardless of different strict mode settings
- Each peer interprets data based on its own strict mode setting when reading/writing
- With
DQL_STRICT_MODE=false: Objects are inferred as MAPs (field-level merging) - With
DQL_STRICT_MODE=true: Objects without explicit type definitions are treated as REGISTERs (whole object replacement)
- Collections/documents created, modified, or read while strict mode is disabled will use inferred types (objects → MAPs by default)
- Collections/documents created, modified, or read while strict mode is enabled use default REGISTER type and require explicit definitions for other types
- If mixing settings, explicitly define MAP types in collection definitions on peers with strict mode enabled
Learn more about strict mode, cross-peer synchronization, and troubleshooting in the DQL Strict Mode documentation.
How It Works
The key difference is whether you need to explicitly define MAP types for objects:New DQL Query Features
Ditto 5.0 introduces several new DQL syntax features that expand query capabilities and make complex queries easier to write.CASE Statements
Add conditional logic to queries with CASE expressions:BETWEEN Expressions
TheBETWEEN operator provides a shorthand for defining an inclusive numeric range for an expression result.Syntax:(a >= 1 AND a <= 10).Array & Object Search Syntax
Test elements within arrays or objects usingANY, EVERY, or ANY AND EVERY operators:Syntax:- IN - Searches the array/object directly
- WITHIN - Searches recursively through nested structures
- ANY - Returns true if at least one element matches
- EVERY - Returns true if all elements match (empty arrays/objects pass)
- ANY AND EVERY - Like EVERY, but empty arrays/objects fail
Array & Object Transformation Syntax
Create new arrays and objects by transforming existing data structures:Array Transformation Syntax:- If
sourceevaluates toMISSING, the result isMISSING - For arrays: if
sourceis not an array/object, the result isNULL - For objects: duplicate field names will overwrite previous values
- Elements/fields where
valueExprevaluates toMISSINGare excluded from the result
Extended String Literals
Support for escape sequences in strings:Hexadecimal Numeric Constants
Additional numeric literal formats:Performance & Platform
DQL Query Performance Improvements
Your applications will feel noticeably faster and more responsive. Data queries complete faster, delivering a snappier user experience whether users are searching, filtering, or loading content. These performance gains happen automatically—no code changes required.Query Planner Enhancements
The DQL query planner has been enhanced to recognize more optimization opportunities:- Automatic ID scan conversion: Equality filters on
_idfields automatically convert to ID scans, bypassing index lookups when exact document IDs are known - Deferred document fetching: Query planner can defer fetching full documents until after sorting and applying offset/limit when index access supports it
- Improved covering index support: More scenarios where the query planner can satisfy queries entirely from index data without retrieving documents
- Index-only queries: Additional cases where queries can be answered using only index scans
Streaming Query Execution
The query engine now uses streaming interfaces internally:- Reduced memory overhead: Results are streamed rather than fully materialized where possible
- DISTINCT operator streaming: DISTINCT queries now stream results, reducing memory usage for large result sets
- Improved operator inlining: Query operators can be inlined into producers for better performance
Shared Statement Cache
Ditto 5.0 introduces a shared statement cache that stores and reuses compiled query plans:- Plan reuse: Compiled query plans are cached and reused for identical or similar statements, eliminating redundant parsing and planning overhead
- Automatic validation: Cached plans are automatically verified and invalidated when collection schemas or system directives change
- Dynamic sizing: The cache automatically resizes based on workload patterns
- Improved large statement performance: Particularly benefits complex queries and larger statements by avoiding expensive re-compilation
These improvements are automatic - no code changes required. Your existing DQL queries will benefit from the enhanced query planner.
Data Sync Performance Improvements
Building on the performance improvements delivered in v4.13 & v4.14, Ditto 5.0 further optimizes data synchronization through tiered blob storage and protocol enhancements, making sync operations faster and more efficient.What’s New
Version 5.0 introduces:- Tiered blob storage: Smaller sync updates avoid unnecessary disk I/O by using optimized storage tiers
- Improved session handling: Better avoidance of session resets on reconnection when in-flight updates were lost
- Optimized fsync policy: Document sync avoids forcing files to disk by default, decreasing I/O and improving latency
Impact
Applications upgrading to v5.0 will experience:- Faster sync operations: Reduced disk I/O overhead leads to quicker data synchronization
- Lower latency: Optimized file handling decreases sync latency across the board
- Better reconnection handling: Fewer redundant updates after temporary disconnections
- Improved efficiency: Reduced disk operations lower memory and CPU pressure during sync
Local System Observability
Gain unprecedented visibility into how Ditto operates on your devices. Query real-time metrics, inspect runtime configuration, and monitor system health directly using DQL—giving you deeper insights than ever before to debug issues, optimize performance, and understand your application’s behavior.New Virtual Collections
system:metrics- Query performance metrics and diagnostics in real-time
- Access counters, timers, and other operational metrics via DQL
- Monitor system health and performance without external tools
system:system_info- Query
peer_key,database_id, and configuration settings - Inspect runtime configuration and system parameters
- Useful for debugging and operational awareness
system:shared_statements- Inspect the query plan cache
- View cached statements and their execution plans
- Supports DELETE operations to clear specific cached statements
- Helps optimize query performance and troubleshoot query planning
Local-Only Collections: System collections are local to each peer and are not replicated across the mesh. To query these collections on remote peers, use Remote Query from the Ditto Portal.
Usage Example
Additional Improvements
Reliability & Error Handling
- Enhanced diagnostics: Improved logging when peers receive data that cannot be deserialized
- Recovery mechanisms: Additional recovery paths for document deserialization errors
- Smart log levels: Connection failures start at warning level, escalate to error only after repeated failures
- Panic messages: Filtered to remove internal Rust machinery frames for improved readability
Networking Improvements
- Graceful shutdown: Network connections close cleanly when Ditto is stopped
- Faster disconnection detection: When a peer crashes, Ditto stops attempting to connect within 15 seconds (previously up to 75 minutes)
- mDNS improvements: More reliable mDNS discovery, configurable service names, better address filtering
- BLE improvements: Fixed connection issues on Android 9 and earlier devices
- WebSocket BYOD support: Bring Your Own Discovery now supports WebSocket connections
- Connection cleanup: Fixed deadlock where devices could fail to establish new P2P connections until restarted
DQL Engine Improvements
Beyond the query performance improvements detailed above, v5 includes:- Better error messages: Improved parser error messages for invalid DQL syntax
- Transaction safety: Fixed deadlock scenarios in concurrent transactions
- Index correctness: Fixed issues where index scans could yield incorrect results on document deletion
Logging & Diagnostics
- Better disk utilization: On-disk logs resume writing to incomplete files, making better use of available space
- Compressed size limits: Log file limits now apply to compressed size, significantly increasing retention
- Explicit flushing: Logs explicitly flushed before aborting due to panic
- Virtual collections: New
system:metricsandsystem:system_infocollections for DQL access to metrics and system information
Platform Support
- Linux aarch64: Kotlin SDK now supports ARM64 Linux (Raspberry Pi, AWS Graviton, etc.)
- Swift 6: Full Swift 6 support with Sendable conformance
- 16KB alignment: React Native Android meets Google Play’s November 2025 requirement
Upgrading to v5
Node.js Migration Guide
Upgrading to Ditto 5.0 requires updating your initialization code and migrating from legacy query APIs to DQL. The migration process involves:- Updating from
DittoIdentitytoDittoConfig-based initialization - Replacing legacy query builder operations with DQL statements
- Migrating collection observers to DQL observers
- Updating authentication patterns
Terminology Updates
Ditto 5.0 updates terminology across the platform to align with industry standards and reduce confusion.Database ID (formerly App ID)
appID→databaseIDin all configuration methodsgetAppId()→getConfig().databaseIdin SDK APIs- Portal and documentation updated to use “Database ID” terminology
Ditto Server (formerly Ditto Cloud)
isConnectedToDittoCloud→isConnectedToDittoServerin presence APIs- Documentation updated to use “Ditto Server” terminology
Migration
Update your code to use the new terminology:Breaking Changes
Ditto 5.0 is a major version release that removes deprecated APIs and legacy features. For migration guidance, see Migration Guidance.Removed APIs
Legacy Query Builder (All SDKs)
All legacy query builder APIs have been removed:store.collection()→ Use DQLINSERT,UPDATE,EVICTstatementscollection.find()→ Use DQLSELECTqueriescollection.findById()→ Use DQL with_idfilter- Live queries → Use DQL observers with
store.registerObserver() - Write transactions → Use DQL
store.write()with transactions
Legacy Initialization (All SDKs)
Identityclasses and all subclasses removedDitto(identity:, persistenceDirectory:)constructors removed- Use
DittoConfigfactory methods andDitto.open()instead
Sync Methods Moved
ditto.startSync()→ditto.sync.start()ditto.stopSync()→ditto.sync.stop()ditto.isSyncActive→ditto.sync.isActive
Other Removals
disableSyncWithV3()- no longer needed, v3 sync removed entirelyAttachmentToken- use dictionary variant- Transport diagnostics APIs - obsolete, removed
- Various deprecated presence properties (
queryOverlapGroup,meshRole, etc.) - Emoji log level headings - setting had no effect, removed
Behavioral Changes
Several default behaviors have changed in v5:- DQL strict mode: Now defaults to
false- no schema definitions required, automatic CRDT type inference - String literals in DQL: Double quotes now delimit strings (not identifiers) for JSON compatibility
- Subscription queries: Reject
LIMITandORDER BYunlessDQL_RESTRICT_SUBSCRIPTION=false - Observer ordering: Observers require explicit
ORDER BYclause for stable ordering - WebSocket sync: Disabled by default in new
TransportConfiginstances - must explicitly enable - Document IDs:
nullis no longer allowed as a document ID
These behavioral changes may affect existing code. Review your DQL queries and subscription logic when migrating to v5.
SDK Size Reduction
The removal of legacy APIs has reduced SDK footprint by approximately 25%, resulting in:- Smaller application binary sizes
- Reduced memory usage
- Faster SDK initialization
- Simpler maintenance and debugging
JavaScript Specific Changes
JavaScript-Specific Changes
The JavaScript SDK has additional platform-specific changes in v5.0 beyond the common breaking changes.Platform Support
macOS:- Removed support for Intel Macs (x86_64 architecture)
- Use Apple Silicon Macs with ARM64 architecture instead
- Library is now 16KB aligned to meet Google Play’s November 2025 requirement
Dependencies
Zero External Dependencies:- Removed all external
package.jsondependencies (reduced from 3 to 0) cbor-reduxis now vendored internally
API Changes
Query Arguments:SyncSubscription.queryArgumentsno longer guaranteed to be strictly equal to original arguments due to serialization roundtripStoreObserver.queryArgumentsno longer guaranteed to be strictly equal to original arguments due to serialization roundtrip- Added
queryArgumentsCBORDataandqueryArgumentsJSONStringproperties for custom decoding SyncSubscriptionclass no longer has generic type for query argumentsStoreObserverclass no longer has generic type for query arguments- Added
queryArgumentsCBORDataproperty toStoreObserver - Added
queryArgumentsJSONStringproperty toStoreObserver
- Added
ConnectionRequest.peerKeyproperty (replaces removedpeerKeyString) - Changed
Connection.peer1andConnection.peer2types fromUint8Arraytostring(no longer deprecated) - Changed
Peer.peerKeytype fromUint8Arraytostring(no longer deprecated) - Removed
ConnectionRequest.peerKeyString- UseConnectionRequest.peerKeyinstead - Removed
Connection.peerKeyString1andConnection.peerKeyString2- UseConnection.peer1andConnection.peer2instead - Removed
Peer.peerKeyString- UsePeer.peerKeyinstead - Renamed
isConnectedToDittoCloudtoisConnectedToDittoServer
- Added
isCompatibleproperty toPeertype to indicate compatibility with local peer
- Added runtime parameter validation for authentication methods for better error messages
- New
TransportConfiginstances now have HTTP listener websocket sync disabled by default
setOfflineOnlyLicenseToken()now throwsTypeErrorwhen passed non-string values likeundefined
Type Improvements
TypeScript Definitions:- Fixed type definitions to use
bigintinstead ofBigInt
Error Handling
Error Codes:- Changed
internal/unknown-errortounknownfor alignment with other SDKs - Changed
sdk/unsupportedtounsupportedfor alignment with other SDKs
Ditto.observeTransportConditions()with non-function parameter now throwsTypeErrorDitto.observePeers()andPresence.observe()with non-function parameters now throwTypeError- Errors thrown from transport conditions observer callbacks no longer cause other registered observers to fail
Bug Fixes
Android BLE:- Android BLE now gracefully handles
DeadSystemRuntimeExceptionandDeadObjectExceptionwhen Bluetooth system service crashes
Full Changelog
JavaScript Specific Changes
JavaScript Specific Changes
JavaScript & React Native Specific Changes
Fixed:setOfflineOnlyLicenseToken()now throws TypeError when passed non-string values like undefined (#17791)- Android BLE now gracefully handles
DeadSystemRuntimeExceptionandDeadObjectExceptionwhen Bluetooth system service crashes (#NETW-1010) - Type definitions now use
bigintinstead ofBigInt(#SDKS-1622) - New
TransportConfiginstances now have HTTP listener websocket sync disabled by default (#SDKS-1711) - Calling
Ditto.observeTransportConditions()with a non-function parameter now throws aTypeError(#SDKS-2044) - Errors thrown from transport conditions observer callbacks, registered through
Ditto.observeTransportConditions(), no longer cause other registered transport conditions observers to fail being called for changed transport conditions (#SDKS-2045) - Calling
Ditto.observePeers()andPresence.observe()with non-function parameters now throws aTypeError(#SDKS-2050) - React Native iOS and macOS builds on Xcode 26.4 (#SDKS-3242)
- The
queryArgumentsproperty onSyncSubscriptions is no longer guaranteed to be strictly equal to the original arguments passed when registering the sync subscription. This is due to a serialization roundtrip, which may affect equality checks, particularly for non-primitive values. If you want to decode query arguments into a specific type, then use thequeryArgumentsCBORDataorqueryArgumentsJSONStringproperties now available onSyncSubscriptioninstances and decode things as required (#17003) - The
SyncSubscriptionclass does not have a generic type representing the type of its query arguments anymore (#17003) - The
queryArgumentsproperty onStoreObservers is no longer guaranteed to be strictly equal to the original arguments passed when registering the store observer. This is due to a serialization roundtrip, which may affect equality checks, particularly for non-primitive values. If you want to decode query arguments into a specific type, then use thequeryArgumentsCBORDataorqueryArgumentsJSONStringproperties now available onStoreObserverinstances and decode things as required (#CORE-303) - The
StoreObserverclass does not have a generic type representing the type of its query arguments anymore (#CORE-303) Connection.peer1andConnection.peer2types fromUint8Arraytostring. These properties are no longer deprecated (#SDKS-1183)Peer.peerKeytype fromUint8Arraytostring. This property is no longer deprecated (#SDKS-1183)- Renamed
isConnectedToDittoCloudtoisConnectedToDittoServeron thePeertype to better reflect that it indicates connection to any Ditto server, not just the cloud (#SDKS-2187) - The React Native Android library is now 16KB aligned to meet Google Play’s November 2025 requirement (#SDKS-2483)
- Error code
internal/unknown-errorhas been replaced withunknownto align with other SDKs. ADittoErrorhas acodeproperty with this value when an unexpected internal error is encountered by Ditto (#SDKS-286) - Error code
sdk/unsupportedhas been replaced withunsupportedto align with other SDKs. ADittoErrorhas acodeproperty with this value when accessing an SDK feature that is not supported on the current platform, e.g. file operations in a browser environment (#SDKS-286)
queryArgumentsCBORDataandqueryArgumentsJSONStringproperties toSyncSubscriptioninstances. If you want to decode query arguments into a specific type, then use these and decode things as required (#17003)queryArgumentsCBORDataandqueryArgumentsJSONStringproperties toStoreObserverinstances. If you want to decode query arguments into a specific type, then use these and decode things as required (#CORE-303)- Property
queryArgumentsCBORDatato classStoreObserver(#CORE-303) - Property
queryArgumentsJSONStringto classStoreObserver(#CORE-303) ConnectionRequest.peerKeyproperty. This replaces the removedpeerKeyStringproperty (#SDKS-1183)- Runtime parameter validation for authentication methods to improve readability of error messages (#SDKS-1494)
isCompatibleproperty toPeertype to indicate whether a peer is compatible with the local peer, matching other SDK implementations (#SDKS-1909)
- All external
package.jsondependencies (reduced from 3 to 0).cbor-reduxis now vendored (#17698) - Support for Intel Macs with CPU architecture x86_64. Use Apple Silicon Macs with ARM64 architecture instead (#DEVX-491)
ConnectionRequest.peerKeyString. UseConnectionRequest.peerKeyinstead (#SDKS-1183)Connection.peerKeyString1andConnection.peerKeyString2. UseConnection.peer1andConnection.peer2instead (#SDKS-1183)Peer.peerKeyString. UsePeer.peerKeyinstead (#SDKS-1183)- Deprecated APIs (#SDKS-1628)
- Method
Ditto.disableSyncWithV3(), which is no longer needed (#SDKS-1628) - Class
AttachmentToken(#SDKS-1628)
Common Changes
Common Changes
5.0.0 Common Changelog
Performance:- Improved the underlying representation of Ditto documents for better performance (#17509)
- Document synchronization between peers now batches updates more efficiently, reducing processing time for large document sets (#19273)
- Document sync avoids sending large, redundant updates after reconnecting a long-dormant session between two peers that are otherwise well-synced with the mesh (#DS-433)
- Improved avoidance of large redundant doc sync updates after session reconnect, based on the number of diffs sent in the initial post-reconnect update (#DS-475)
- Faster initial sync when processing rkyv-encoded document diffs (#DS-773)
- Faster eviction of indexed documents when using rkyv (#DS-774)
- Improved performance of initial index generation when using rkyv as a document format (#DS-774)
- Reduced redundant replication GC work during rapid eviction bursts by debouncing and coalescing compatible GC tasks (#CORE-1466)
- Synchronization protocol enhanced to better avoid resetting sessions on reconnect if in-flight updates were lost (#DS-820)
- Document Sync now uses tiered blob store for update file storage by default, allowing smaller updates to avoid unnecessary disk I/O (#DS-836)
- Document sync implementation avoids forcing outbound update files to disk by default, decreasing disk I/O and improving sync latency (#DS-921)
- Observers to avoid sort by id on non order by query / add limit to sort operator (#QE-261)
- Improved out-the-box performance for larger statements (#QE-377 & QE-378)
- Improved the performance of IN-list evaluation for large lists of static values (#QE-386)
- An issue with BLE on some Android 9 and earlier devices that prevented connection establishment (#17760)
- Multiple concurrent DQL transactions can no longer possibly lead to a deadlock (#17816)
- A bug where x509 refresh could speed up uncontrollably (#17947)
- A bug where a peer could get stuck with incorrect connection information (#17967)
- Network connections close gracefully when Ditto is stopped (#18053)
- The
system:data_sync_infocollection may briefly report sync immediately after connecting (#18137) - An issue where peers could fail to connect other local peers via mDNS on macOS (#18488)
- A bug that meant that document id indexes were not created (#18512)
- A bug where forced TCP connections would be retried more frequently than expected (#19727)
- A bug where mDNS registration would fail with
NamingConflict(#20411) - mDNS discovery should support TCP and UDP independently (#20490)
- Connection manager now correctly cleans up orphaned connecting state when tasks are cancelled during shutdown (e.g., during auth refresh), preventing “Already connecting” errors on reconnection (#20377)
- A deadlock could occur where a device would fail to establish new P2P connections until restarted (#20810)
- A race condition where
fetchAttachmentcould permanently fail to find locally-created attachments due to stale in-memory cache entries (#21146) - A bug that caused small peers to re-upload remotely requested files (e.g. logs) on every startup (#CORE-810)
- High latency when write transactions trigger evictions (#CORE-1453)
- Live queries being unable to use indices (#DS-447)
- A rare scenario where an attachment fetch could be delayed by up to 60 seconds (#DS-461)
- Deadlock in sync session metadata cleanup caused cascading lockups for doc sync and/or local doc store access (#DS-485)
- Post eviction cleanup of disconnected document sync sessions now retains metadata for non-evicted documents (#DS-487)
- Document sync session metadata became unlinked after session reset (#DS-509)
- DQL SELECT queries on indexed collections could deadlock if executed in an explicit transaction (#DS-648)
- DQL queries using indexes could yield wrong results upon document deletion (#DS-851)
- Inconsistent internal hashing of Document IDs could cause failure to sync documents (#DS-944)
- Premature connection cleanup causing duplicate connections (#NETW-1021)
- When another peer crashes, Ditto will stop attempting to connect to it in under 15 seconds. Previously, connection attempts would occur for up to 75 minutes at 5 second intervals (#NETW-856)
- Changed live queries to use the new streaming interface for query execution (#QE-261)
- Use of DISTINCT with ORDER BY & OFFSET/LIMIT (#QE-281)
- In BP profile directive does not produce path in collection scan operator profile (#QE-294)
- Handling of references to group keys that include array element selection (#QE-306)
- Use deterministic summaries for documents created as part of observer evaluation (#QE-310)
- Removed fabricated descriptor information from collection scans in profile/explain information (#QE-315)
- Performance of the DQL parser for large statements (#QE-321)
- DQL duration function floating-point rounding errors (#QE-402)
- Incorrect association of terms following an
IN exprclause (#QE-418) - Executing index scans in observers preserves document ids to provide consistent ordering (#QE-429)
- The query planner misses using a Filter operator if collection scan filter pushdown is enabled (#QE-437)
- Small peer collection scan handles offset and limit incorrectly (#QE-438)
- Changed DQL planner index selection process to correctly handle predicate paths that haven’t been formalised avoiding incorrect index selection (#QE-456)
- The DQL query planner will no longer generate index scans on fields which have been specified in a COLLECTIONS clause as the variant specified may differ from the indexed variant. This prevents incorrect query results arising from the mismatch (#QE-475)
- The small peer DQL query planner to not produce index scans when DQL_STRICT_MODE is set to true. This avoids incorrect results from filters on fields where the latest variant is not REGISTER (#QE-478)
- The store SQLITE logging callback has been changed to not report schema change errors avoiding flooding the logs with spurious warnings from SQLITE internal operations (#QE-479)
- A panic in the DQL parser when an invalid JSON directive is used prior to the PROFILE keyword (#QE-490)
- -9223372036854775808 is now parsed and handled as an integer value by DQL (#QE-499)
- Queries in the legacy language might fail to surface documents with settable counters (#QE-512)
- Documentation for on-disk logger incorrectly stated logs are retained for 3 days; actual retention is 15 days (#SDKS-1726)
- FFI resource cleanup now safely handles null context pointers in release callbacks, preventing potential null pointer dereferences during Ditto shutdown (#SDKS-2744)
- Attachment callback dispatch now properly handles thread pool dispatch failures by retaining context to prevent use-after-free errors when the callback pool is unavailable (#SDKS-2744)
- Transport watchdog no longer logs spurious “Address already in use” errors when restarting TCP/HTTP servers. The watchdog now waits for previous server tasks to fully exit and release their sockets before attempting to rebind ports, preventing EADDRINUSE races when servers are invalidated during e.g. app backgrounding (#SPO-127)
- Attachment garbage collection now removes empty shard directories, preventing inode leaks (#SPO-158)
- Bug which meant a Ditto starting log failed to be logged (#SPO-626)
- Android devices that fail to acquire an IP address will now continue to sync over LAN (#TRAN-725)
- A long-running on connecting callback no longer causes
ReceiveTimeoutfailures in connectivity (#TRAN-729)
network_enable_multihopsystem parameter renamed tonetwork_enable_ngn, gating NGN features which include multihop (#17882)- Enabled opentelemetry tracing in core (#18478)
- Connection failures now use smart log levels, starting at warning and escalating to error only after repeated failures (#18779)
- mDNS now only advertises connectable addresses based on TCP server binding address. TCP Client now attempts to connect to all addresses in parallel (#18930)
- The default value of
ENABLE_ATTACHMENT_PERMISSION_CHECKSstore configuration parameter tofalse(#18931) - mDNS service name is now configurable via the
transports_mdns_service_nameSystemParameter (default:_http-alt), ensuring both TCP and UDP transports use the same service name (#20289) - Panic messages now filter out internal Rust machinery frames (backtrace capture, panic handling, runtime startup) for improved readability, with an environment variable
DITTO_PANIC_WITH_FULL_STACK_TRACE=1available to show complete stack traces when needed (#20401) - On-disk logs now make better use of available disk space by resuming writing to incomplete files (#CORE-561)
- On-disk log file limits now apply to the compressed size, significantly increasing log retention (#CORE-729)
- Write transaction diagnostic logs now include the blocking transaction’s current operation, elapsed time, and queue depth (#CORE-1449)
- More information is logged at debug level around peer GC and eviction progress including what remote peers are being deleted (#CORE-1455)
- Significantly lowered the maximum values of some system parameters governing the on-disk rotating file logger’s behavior (#CORE-848)
- More information is logged when a Ditto peer receives data that it cannot deserialize due to a hash mismatch (#CORE-897)
- We now make an additional attempt to recover from some kinds of document deserialization errors, which may reduce errors or crashes due to deserialization (#CORE-916)
- Document sync protocol now supports rkyv-encoded diffs, for more efficient initial synchronization (#DS-531)
- TCP server is no longer auto-enabled when TCP is disabled in config, NGN and UDP is enabled with LAN discovery (#NETW-1039)
- Mutators are preserved for Big Peer V4 (#QE-126)
- Disallow null as an id (#QE-202)
- DQL_STRICT_MODE default to false (#QE-267)
- Planning for statements using DISTINCT projections, in some circumstances (#QE-282)
- system:vitals outputs timers as sub-document (#QE-312)
- DQL queries run via observers now require that the user provides stable ordering themselves via a suitable ORDER BY clause (#QE-427)
- Double quotes delimit strings, not identifiers (JSON compatibility) (#QE-44)
- DQL query planner now recognises additional index access plans that eliminate the need for document retrieval (#QE-449)
- Added the key sorting direction to the
system:indexesvirtual collection output (#QE-467) - The DQL planner to automatically convert equality filters on the
_idfield to ID scans, improving performance by skipping index use when exact document IDs are known up front (#QE-469) - The DQL query planner can now create a query plan that defers fetching full documents until after sorting and application of offset and limit, if the index access portion of the plan can support it. This means performance improvements for queries with affected plans (#QE-473)
- Revised the Query Engine DISTINCT operator to stream results reducing memory overhead (#QE-474)
- The DQL planner to consider additional cases where an index may cover a query leading to improved performance in those scenarios (#QE-491)
- The DQL Query Engine planner can now produce specialised higher performance plans for Ditto Server
COUNT(*)queries that include simple filters (#QE-510) - The Query Engine DQL planner can now generate index-access plans against Ditto Server improving performance queries with filters that can be applied when scanning an index and where combining multiple index scans is beneficial (#QE-511)
- The Query Engine intersect scan operator can now stop before all inputs have completed, once it has been established that no further complete intersections can be produced, improving the performance where the number of values produced by each input differs greatly (#QE-530)
- Added specialisation of simple queries to take advantage of small peer indexing (#QE-266)
- Added support for microseconds to duration scalar functions (#QE-396)
- Removed the Parseable query trait to minimise the impact of the query parser maintenance on the monorepo (#QE-421)
- System collection
system:ditto_metricsrenamed tosystem:metricsfor consistency with naming conventions. Existing queries usingsystem:ditto_metricswill need to be updated to usesystem:metrics(#SDKS-2653) - Ditto shutdown logging promoted to info level (#SPO-626)
ENABLE_ATTACHMENT_PERMISSION_CHECKSALTER SYSTEM parameter to be set tofalseto avoid certain rare cases of attachment fetcher hanging (#18016)- Bring Your Own Discovery now supports WebSocket connections (#18965)
system:metricsvirtual collection for DQL access to metrics (#MESHCON-53)- Introduced new entries to the
system:system_infocollection forpeer_keyanddatabase_id(#19435) DATA_SYNC_ENABLEDsystem parameter which, while set tofalse, halts the data sync machinery, but without loss of network connectivity (allowing for operations such as remote query) (#19643)DITTO_USE_TIERED_BLOB_STORE_DOC_SYNCsystem parameter env var to improve the performance of doc sync in heavy mesh scenarios, at the expense of sync chatter overhead upon restart (#19847)- The new
transport_tcp_connect_timeoutsystem parameter which shortcuts long platform-based TCP connection timeouts (#20782) - Explicit log flushing before aborting due to panic (#CORE-723)
system:system_infovirtual collection for DQL access to system info (#CORE-751)- small_peer_info subscription queries now include arguments (#DS-1027)
- system:system_info subscription queries now include arguments; query location moved from key to value.query (#DS-1027)
- Improved logging for document deserialization errors (#DS-568)
- System parameters that can be used to configure sqlite pragmas (#DS-682)
- System parameter
doc_sync_outbound_update_fsync_policyto govern the use of fsyncs when creating doc sync update files (#DS-817) - Channel open retries and watchdog to tear down VirtualConnections that never open a channel, preventing resource exhaustion from stalled peers (#NETW-1098)
- A system parameter
transports_dns_sd_backendfor choosing the mDNS backend implementation (#NETW-940) - Ability to specify documents to insert in arrays in DQL (#QE-118)
- USE IDS LIST syntax (#QE-119)
- Support for settable counters to DQL (#QE-130, QE-393, QE-394, QE-395)
- Specialisation of COUNT(*) DQL queries on the big peer (#QE-138)
- Subscription queries reject LIMIT and ORDER by unless DQL_RESTRICT_SUBSCRIPTION is set to false (#QE-218)
- INTERSECT and UNION (index) scans (#QE-230_and_240)
- Covering index scans (#QE-239)
- Support for the BETWEEN DQL expression (#QE-26)
- Streaming interfaces for query execution (#QE-262)
- Array & object search syntax to DQL (#QE-265)
- Ability to inline consumers into producer operators (#QE-272)
- Consolidated results across subservers for queries against ACTIVE_REQUESTS, REQUEST_HISTORY and VITALS (#QE-273)
- Array transformation DQL syntax (#QE-274)
- Object transformation syntax to DQL (#QE-276)
- Promethus metrics to the DQL Query engine (#QE-296)
- Automatic generation of USE IDS clause from _id equality predicates when possible (#QE-313)
- A shared statement cache and virtual collection (#QE-316)
- Configurable concurrent request limit to the Query engine (#QE-317)
- Periodic dumping of request history cache entries to the log (SP only) (#QE-322)
- The ability for the shared statement to store, verify, amend and use existing plans for qualifying statements (#QE-324)
- PROFILE keyword as the equivalent to the #profile directive (#QE-336)
- CASE statement (#QE-41)
- Support for settable counters in INSERT DQL statements (#QE-406)
- The statement caches invalidates existing statements if the default directives change (#QE-407)
- Support for extended string literals that can contain escape sequences in DQL (#QE-410)
- Support for hexadecimal numeric constants in DQL statements (#QE-413)
- Statement cache resizing (#QE-414)
- The ability for Remote Query to handle DELETE, EVICT, TOMBSTONE DQL statement (#QE-441)
- Enable sending all statements via the SYNC CONTEXT statement (#QE-447)
- Support for DQL DELETE against the
system:shared_statementsvirtual collection (#QE-450) - Publish sync scopes in small peer info document (#SPO-276)
- The static path option has been removed from TransportConfig. Please use attachments to serve content instead (#TRAN-256)
- Deprecated fields in presence have been removed across all SDKs (
queryOverlapGroup,meshRole,approximateDistanceInMeters,rssi) (#TRAN-680)