Release Notes
Big Peer 1.40 introduces an overhaul of the underlying data structures of Big Peer’s store, resulting in substantial improvement to performance and reliability, while unlocking new capabilities for upcoming features.
In addition, 1.40 contains a number new improvements and fixes.
Added:
- A new data architecture for Big Peer, improving reliability, performance and unlocking future capabilities
- Big Peer evictions are now available as a preview feature. Contact us to enable this feature for your instances
- Synchronization protocol enhanced to reduce the amount of redundant data sent from small peers to Big Peers
- Bump the minimum supported CRDT version for Big Peer apps to V6
- Several improvements to Big Peer reliability and deployment efficacy
Fixed:
- Addresses a rare issue decompressing attachments during replication
- Performance optimisations made to document deduplication on the subserver
- Device name no longer incorrectly defaults to “ditto” in device table, when a named device first starts syncing
- Fixed an issue where HTTP API calls might incorrectly fail when passing a transaction ID as part of the call
- Fixes a potential deadlock condition in Big Peer
- Align Kafka producer and consumer message sizes to prevent potential backpressure scenarios
- Network policy improvements to enable new feature development
Added:
- Networking policy enhancements to support architectural improvements
Fixed:
- Addresses a rare issue decompressing attachments during replication
Added:
- Improved the API Key validation of the HTTP API to now automatically retry a number of times in case of transient issues, before throwing an error to the caller
Fixed:
- An issue where HTTP API calls might incorrectly fail when passing a transaction ID as part of the call
- A failure when executing remote queries that contained non-ASCII characters in the SYNC CONTEXT clause
- An issue that prevented attachment replication when a large number of documents were also being simultaneously replicated
- A number of issues related to data convergence during replication in rare scenarios
Added:
- Improved the API Key validation of the HTTP API to now automatically retry a number of times in case of transient issues, before throwing an error to the caller
Added:
- DQL INSERT statements run via the HTTP API now support inline object literals.
- The /api/v4/attachments/ HTTP endpoint now supports HEAD requests to check for attachment’s existence.
- Improved the performance of subscriptions with limits.
- Improved the performance of replication during garbage collection.
Fixed:
- Fixed a failure in replication in cases where there are frequent subscription changes and on-device evictions.
- Fixed an issue which could cause a crash during document merges when using older versions of the SDK.
Added:
- Improved query performance for DQL queries that access attachments, through the HTTP data API.
- Improved handling of slow WebHook targets by enforcing a 30-second timeout to outgoing WebHook calls, this prevents lockups and other unexpected behavior when using WebHook connections.
- Improved replication to avoid resending metadata, resulting in a reduction in overall replication traffic and processing when sync subscriptions change.
Added:
- Added support for inline object and array definition to DQL through the HTTP API.
- Added object_length() and deserialize_json() functions to DQL through the HTTP API.
- Improved the error message for cases where the query result is too large.
- Removed the //distinct_values HTTP endpoint.
- Removed support for X-DITTO-CLIENT-ID, X-DITTO-ACTOR-ID, X-HYDRA-CLIENT-ID from the HTTP API.
Fixed:
- Fixed a hang when replicating documents or attachments larger than 64KB.
- Fixed the default HTTP API response format to correctly be application/json if not specified in the request.
- Fixed a rare edge case that could slow down sync when using very small documents.
- Synchronized version bump release – No changes.
Fixed:
- Potential crash as a result of a DQL query containing a multi-byte character. (#12324)
- A hang in the transports multiplexer when handling large messages spanning multiple chunks larger than 64KB. (#12428)
Added:
- Attachments now sync independently of documents, improving rainbow connection efficiency (#11156)
Fixed:
- Sync crashes due to missing update files and other unexpected conditions are now handled recoverably (#11952)
Added:
- Improve performance by avoiding deserializing the entire Document if only part of it is required. (#10887)
- Removed is_connected_to_big_peer field from the Small Peer Info document. (#11883)
- Increased the limit on the allowable depth of the Small Peer Info JSON metadata. (#11891)
- Attachments are more performant when working with many attachments concurrently
- Spread work more evenly across Big Peer store writer threads. (#11939)
- MeshChooser now supports a limited set of runtime configurable values via the mesh_chooser_* system parameters. (#11968)
- String functions byte_length and char_length. (#12147)
Fixed:
- Resolved legacy query language casting counters and maps with tombstones to the wrong boolean value (#10887)
- Resolved Bytes and Arrays not being always comparable in the legacy query language. (#10887)
- Resolved UST initial polling unbounded error retry. (#11841)
- Resolve legacy query language accepting invalid field names. (#11888)
- Resolved HTTP API OpenAI spec that will cause duplicated Authentication header in certain generated clients. (#11936)
- A warning message that appears when peers connect is no longer logged (No handle found to notify the shutdown; multiple message handling loops??) (#11977)
- When replicating in mixed meshes from V4 to V3 full state will be sent. (#12029)
- Lowered the log level of some SQLite3 notices that were being unnecessarily logged as warnings. (#12032)
- The Small Peer Info collector no longer gets stuck waiting to upload logs under certain conditions. (#12070)
- Duration parameters can now be set via DQL, whereas before they would fail to type check. (#12070)
- Ensure document cache GC always progresses regardless of configured time slice and load. (#12091)
Fixed:
- Revert “Fix DocumentCache cache wrong result for Sort + Limit queries. (#9435)
- Query plan for queries using the contains function with an empty array. (#12010)
Fixed:
- Stream-splitter consumer migration applying old offset. (#11985)
- Query plan for queries using the contains function with an empty array. (#12010)
Added:
- Update stream-splitter to respect Ditto login retry interval and attempts. (#11785)
Fixed:
- Stream-splitter checks for kafka.enabled before spawning user-consumable streams. (#11788)
- Tweak hydra-cdc-heartbeat to avoid stuck consumer. (#11815)
- Application crashes while preparing to syncing data to remote peer. (#11843)
- Fix Subserver caching wrong result for queries with sort and limit. (#9435)
Added:
- Improve UST polling resilience in HTTP API and subserver (#11701)
Added:
- DQL queries can now query for the id, len and metadata fields of attachments (#11481)
Fixed:
- Do not error in DQL inserts with the DO NOTHING conflict policy (#11730)
Added:
- Synchronization protocol enhanced to prevent accumulation of unsubscribed documents after subscription change, and limits redundant data exchanged when re-synchronizing. (#10440)
- Reduce metadata lock contention while applying transactions (Avoid metadata read lock in apply_txns_internal #11654)
Fixed:
- Fix race between applying of transactions and reconfiguration of replicas (#11361)
Fixed:
- Legacy query language casting counters and maps with tombstones to the wrong boolean value
- Legacy query language not allowing comparison against attachment objects.
- Bug where certificate request would not check identity data hash with JWT’s
- Bug where the certificate request would fail with big permissions
Added:
- Improve read query performance of legacy contains([..], _id) syntax (#10987)
- Parser error messages emphasize the location of the error and de-emphasize a potential fix. (#11040)
- Reduce HTTP API latency with improved lock map version tracking (#11278)
- Improved read query performance for DQL queries using array_contains (#11338)
Fixed:
- An issue where an internal data decoding function could panic on certain inputs (#10376)
- Return 500 instead of 401 upon internal error while validating API key. (#11178)
- A bug where Link could run into an infinite loop. (#11188)
Added:
- Make actix-web keep alive timeout configurable for Big Peer (#9578)
Added:
- Preliminary support for runtime configuration using DQL ALTER SYSTEM and SHOW statements (Hook up DQL entrypoint to parameter store #10401)
Added:
- Peers use latest supported protocol version, even if the application is using backwards-compatible CRDT formats. (Disentangle protocol version from protocol flags #10875)
Fixed:
- Fix deadlock when replicating attachments to Big Peer (Fix AttachmentsPeer deadlock when setting up outbound backoff #11036)
Added:
- Improve read query performance on queries with multiple ranges (#10605)
Fixed:
- Fix incorrect DQL literal comparison behavior in big peer planning. This fixes a correctness issue where big peer could return no results when it should return all documents in a collection. This also makes big peer avoid a full scan in certain cases where it should return no results. (#10940)
Added:
- SDK version/language/platform and device name are now included in the small peer info document. (#10515)
Added:
- Improved read query performance by pre-filtering non-matching document ids (#10620)
- Improve memory usage and performance of Documents (#10736)
- Performance by implementing bipartite distance-aware mesh topology (#10632)
Fixed:
- Mesh would choose random connections (#10711)
- Fix
NOT
on non-Boolean panics on big peer (#10762) - Fixed a bug with Ditto’s internal crash handler (#10022)
- Initial logging events now use the same format as all other logs (#10611)
Fixed:
- Fix hashing bug in Big Peer store node controller
Fixed:
- Fix a bug preventing updates to clusters from rolling out properly.
Fixed:
- Fix bug related to Kafka transaction chunking
Fixed:
- Improved run-time validation of corrupted attachments being inserted into documents (#10552)
- Synchronized version bump release – No changes.
Deprecated:
- Deprecated experimental time-series HTTP APIs
Fixed:
- Big Peer queries matching on nested objects or arrays return the expected documents (#10382)
- Addressed a warning from SQLite that sometimes occurs on startup about a schema change (#10501)
Fixed:
- Fixed HTTP API output format for Attachments when using SerializedAs::LatestValues
Added:
- Remove doc_ids from requery_required message in CDC
- Reduced metadata accumulation after eviction by clearing remote summaries
Added:
- Add attachments support for HTTP API RPC v4
Added:
- Optimized Big Peer internal data serialization performance.
- Optimized the updates of internal system collections when doing writes with the HTTP RPC API.
Fixed:
- Fixed a bug around where queries could return an older version of documents around a server reboot.
Added:
- Add RPC V4 HTTP API.
- Update our OpenAPI specification to include the new count request.
Added:
- Made the Big Peer more resilient to Kafka broker disconnects
- Improve our HTTP API implementation to reduce Timeout related errors
Fixed:
- Fix bug in Big Peer device certificate CA rotation process
- Improved performance when multiple peers with distinct permissions work with the same cloud endpoint
Fixed:
- Various fixes and improvements for better reliability, security, and performance
Fixed:
- Minor bug fixes and performance improvements.
Added:
- Changed: new serialization format improves efficiency of replication, particularly in the Big Peer
Fixed:
- Fix detection of broken stream when there are multiple apps in the transaction log
Added:
- Initial release of the lazy deserialization optimization for the Big Peer.
- Tighten network policies for increased security and stability
Fixed:
- Fix bug where Big Peer would sometimes panic if an SDK peer managed to create multiple Ditto instances with the same persistence directory. We are seeing this error more than we originally hoped. This change makes any occurrence non-fatal until we fix the root cause. There will be another followup change that will re-enable alerting for this event.
Fixed:
- Fix bug where HTTP API upserts would not always be reflected in subsequent find requests.
Added:
- Properly consider limit+offset+order_by for active queries
Fixed:
- Minor bug fixes and performance improvements
Added:
- Supports Ditto SDK through version 2.0.x.
Fixed:
- Minor bug fixes and performance improvements
Fixed:
- Minor bug fixes and performance improvements.
Added:
- Add collection and document id to RequeryRequired requests as part of the live query WebHook and CDC.
Fixed:
- Minor bug fixes and performance improvements
Fixed:
- Minor bug fixes and performance improvements
Fixed:
- Minor bug fixes and performance improvements
Fixed:
- Minor bug fixes and performance improvements
Added:
- Supports Ditto SDK through version 1.1.6
- Parity between document and timeseries HTTP APIs
- HTTP API calls that were GET with a body now also accept POST. They will accept both methods and will behave the same no matter which method is used.
Added:
- Supports Ditto SDK through version 1.1.5, including improvements to replication efficiency.
Fixed:
- Adds HTTP API RPC Remove and Update commands (remove + update). With remove, users may provide a query and delete all matching records. The update command mimics Ditto SDK update behavior and applies a series of updates to all matching documents.
- The old header names are now deprecated (All HTTP API Headers used by Ditto will be prefixed with X-DITTO-…).
Was this page helpful?