Big Peer 1.65.1 is a patch release that adds optional sharding of CDC output across multiple Kafka producers to raise produce throughput, and fixes a webhook dispatcher failure that could stop webhooks being delivered after a Kafka reconnect. Added: Fixed:
- CDC can shard its output across multiple idempotent Kafka producers, controlled by the new
PRODUCER_SHARDSsetting, raising produce throughput to downstream consumers under sustained write load. The default of1leaves behavior unchanged
This does not change CDC’s ordering contract. Changes for an individual document are still delivered sequentially, and global ordering across documents is not guaranteed either way - see Event Ordering. Contact Ditto before raising
PRODUCER_SHARDS above 1, as it has a deployment prerequisite.- The webhook dispatcher no longer aborts when its Kafka consumer reconnects, so a broker restart or partition rebalance no longer leaves it restarting repeatedly instead of delivering webhooks
Big Peer 1.65.0 is a minor release with a broad set of DQL enhancements. It adds new scalar functions, request profiling, and execution-time limits. It also brings significant memory reductions for query results and store observers, plus a range of reliability and safety fixes across sync, DQL, and the HTTP API. Added: Changed:
- A new
EXECUTE FUNCTIONDQL statement for running DQL functions with side effects in a controlled environment - Additional DQL scalar functions, including common trigonometric functions and further array-processing functions
- An optional DQL parameter that limits a request’s execution time, stopping requests that exceed the limit and completing them with an error
- Automatic diagnostic logging for long-running DQL requests, controlled by the new
DQL_SLOW_REQUEST_WARN_SECONDSsystem parameter - Two new
request_historyvirtual collection qualifiers: one that captures requests carrying profiling information (thePROFILEkeyword or#profiledirective) when set toTRUE, and one that captures requests byrequestType - A new
transports_websocket_watchdog_interval_secssystem parameter for tuning the WebSocket client watchdog interval - A guard that halts a node rather than applying a change produced by a newer, unsupported release, preventing silent, permanent divergence if a node is rolled back to an older release after the rest of the cluster has upgraded
- A new metric that measures Ditto auth time separately from customer webhook time
This guard does not change behavior on its own. It establishes 1.65.0 as a minimum baseline: future releases may require every node in a cluster to be running 1.65.0 or higher before upgrading, so that this protection is already in place.
- Oversized documents written through the HTTP API now return
413with a message explaining the document is too large (v4store/writeupsert and v5store/execute); the v4 update result reports them in a newdocumentTooLargecount, instead of a misleading 500/503 or a silent success - Query plan specialisation for
COUNT(*)queries now applies when there are top-level fieldIS NULLpredicates - Structured peer ID fields now render resolvable peer-key prefixes instead of legacy suffixes; field names are unchanged
- DQL queries with long
AND/ORoperator chains no longer cause a Subscription Server panic from exceeding parser recursion limits - Invalid TLS identity and certificate configuration now returns actionable errors instead of being silently ignored or panicking
- The DQL
object_contentscalar function now correctly processes the"nesting"configuration value"only"when passed as part of an object - The DQL
substrnegative index calculation and thelpadandrpadscalar functions now handle multi-byte characters correctly
- DQL query results are held in a compact form and expanded only when read, reducing the memory retained by query results and store observers, especially for large result sets
- DQL store observers no longer keep observed documents fully decoded between updates, and observers whose query cannot reuse the previous result incrementally (for example projections or index-backed lookups) no longer retain a copy of the previous result set between updates
- Reading a document — including the documents a store observer delivers on each update — no longer pins its decoded form in memory, reducing the memory retained by long-lived observers
Big Peer 1.64.4 is a small patch release that adds an opt-in transaction-log apply pipeline for higher sustained apply throughput. Performance:
- An opt-in transaction-log apply pipeline that overlaps polling and deserialization with apply, and batches remote multi-write dispatch, raising sustained apply throughput. Enabled with the
TXLOG_PIPELINE_ENABLEDsetting; when it is unset the store runs the previous behavior
Big Peer 1.64.3 is a small patch release that optimises CDC performance. Performance:
- CDC optimised to improve throughput to downstream consumers under sustained write load
Big Peer 1.64.2 is a small patch release that bounds how far Change Data Capture is allowed to fall behind, protecting store storage growth when a CDC consumer lags. Added:
- A bound on how far behind a Change Data Capture consumer may fall, configured with the new
MAX_SESSION_VERSION_LAGsetting. A consumer that lags without a bound holds back store cleanup and lets storage grow; with a bound in place the store stops holding data back past the configured lag. Leaving it unset keeps the previous unbounded behavior
Big Peer 1.64.1 delivers better resource management for CDC, improved performance and reliability around evictions, and a range of reliability improvements across sync, the MongoDB Connector, and the query engine. Added:
- Better resource management for CDC under heavy workloads
- Improved performance and reliability under eviction-heavy workloads
- Kafka consumer stall detection that automatically reconnects when no messages are received, preventing silent hangs during broker failures
- A new
transport_websocket_connect_timeoutsystem parameter for tuning WebSocket connect timeouts - More predictable cleanup of disconnected document sync sessions, reducing memory usage on long-running deployments
- The HTTP API
/store/writeendpoint now returns503 Service Unavailablewhen its first command fails to commit due to an internal error, instead of a200with aninternalErrorcount. Failures of later commands in a batch are still reported in the response body so successful results are preserved
- Improved CDC durability across restarts so transactions are reliably persisted when upquery results are still in flight at the time of a crash
- Improved hydra-store recovery so Big Peer nodes whose data directory has been wiped now rebuild automatically by replaying the Kafka log or backfilling from peers, instead of stalling in recovery
- Improved reliability of long-lived Document Sync sessions by addressing an internal subscription bookkeeping issue that could cause them to be disabled due to spurious capacity errors
- Improved shutdown behavior of the MongoDB Connector when MongoDB is unreachable, so the pod now exits cleanly and is restarted by Kubernetes
- Raised the MongoDB Connector default rate limit for initial sync and resync from 500 to 3000 docs/sec