Tracing Document Propagation

Propagation tracing is a beta feature in SDK 5.2. Its configuration and trace details may change.

Use propagation tracing to see which peers handled a document change and how long it took to reach them. A trace follows one specific change. Each peer records its participation in the propagation as a timed span. The peer that made the change starts the root span, and receiving peers add spans as the change moves through Ditto’s mesh. An OpenTelemetry backend correlates the spans into a single trace. Ditto provides two ways to start propagation traces:

Automatic collection tracing requests traces for eligible changes in selected collections. Configure the collection list in the Ditto instance configuration or set it with the ALTER SYSTEM DQL statement. You can also change it for a particular peer at runtime through Remote Query.
Programmatic tracing uses the system_trace_sync_propagation DQL function inside a write transaction. It lets the application choose document IDs to trace or limit the number of traces requested in that transaction.

Both methods are beta. The DQL function is also a provisional DQL API and requires preview mode.

Requirements

To observe propagation between peers, use at least two peers that sync the change. Use Ditto SDK 5.2 or later on every peer along the path that you want represented in the trace. Older peers can still receive and forward document changes, but the trace cannot continue through them.
An OpenTelemetry Protocol (OTLP)-compatible collector and trace backend.
The OTLP endpoint configured on each peer whose spans you want to export.

Use OTLP to collect spans as a correlated trace. Some propagation lifecycle events also appear in peer logs, which can provide additional diagnostic information. Logs are separate for each peer and do not include the complete span timeline and relationships available through OTLP.

Configure OpenTelemetry export

Propagation tracing uses Ditto’s OpenTelemetry exporter. If the peer is already configured to export Ditto spans through OpenTelemetry, you may skip this section. Propagation spans use the same OTLP configuration. Otherwise, set the following environment variables in the peer’s process environment before Ditto initializes its logging system. Set them before the first Ditto SDK or logger call. How you configure the process environment depends on the platform. These are not Ditto system parameters and cannot be changed with ALTER SYSTEM.

Environment variable	Example value	Description
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://otel-collector.example.com:4317`	The OTLP collector endpoint. Replace the example host and port with an endpoint reachable from each peer that exports spans. If unset, OTLP export is disabled.
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	The protocol, either `grpc` or `http/protobuf`. If unset, Ditto infers it from the endpoint port. If it cannot infer the protocol, Ditto defaults to gRPC.
`OTEL_EXPORTER_SAMPLING_RATIO`	`1`	Sampling ratio from `0` to `1`. The default is `1`; keep it at `1` while evaluating so sampling does not hide traces.

For a local collector example, see Try Jaeger locally.

Automatic collection tracing

Automatic tracing is disabled by default. It requests traces for eligible changes made by local, app-initiated write transactions in the selected collections. The app does not need to call a tracing function for each transaction. The peer’s rate and retention limits still apply; see Set trace retention and rate limits. Use ALTER SYSTEM to set propagation_trace_auto_trace_collections to the list of collection names to trace automatically:

DQL

ALTER SYSTEM SET propagation_trace_auto_trace_collections = ['tasks']

To enable tracing on a deployed peer without changing the app, send this statement through Remote Query. Remote Query persists the parameter override on native peers, so it remains set after a restart. Remote Query must be enabled and allow ALTER SYSTEM statements. The peer also needs its OTLP exporter configured to send spans to your collector. If the app sets the parameter locally with ALTER SYSTEM, the setting applies only to the current Ditto instance. It must be applied again after each initialization. To set it at startup, configure propagation_trace_auto_trace_collections in the peer’s Ditto instance configuration. This may require changing the app or its deployment configuration. The change applies to application-initiated write transactions that start after it is set. Transactions already in progress retain their existing configuration. The collection list controls which changes start traces on that peer. A receiving peer continues a trace it receives even if the collection is not in its own list. Add a collection to each peer where you also want changes made there to start traces. Configure OTLP export on every peer whose spans you want to collect.

Programmatic tracing in a transaction

Call system_trace_sync_propagation from a DQL write transaction when application code needs to choose which changes to trace. This function requests traces for matching changes made later in that transaction. You can select document IDs and cap the number of requests. The peer’s rate and retention limits still apply.

This is a provisional Small Peer DQL API. Its name, arguments, and behavior may change before general availability. The function requires dql_enable_preview_mode, which is off by default. This parameter enables preview DQL features generally; it is not specific to propagation tracing. Enable it on each peer whose app calls the function. A peer that only receives and forwards a trace does not need preview mode.

Enable preview DQL mode on each peer before its app calls the function:

DQL

ALTER SYSTEM SET dql_enable_preview_mode = true

When the app sets this parameter locally with ALTER SYSTEM, the setting applies only to the current Ditto instance. Apply it after each initialization and before the transaction that calls the function. You can also set it through Remote Query or in the Ditto instance configuration. See Customizing System Settings for configuration details. Create a write transaction through the SDK transaction API. Execute the function before the document changes you want to trace. Send the function and each document change as separate DQL statements against the same transaction, then commit it. The example requests a trace for task-1 and inserts that document later in the transaction:

DQL

EXECUTE FUNCTION system_trace_sync_propagation(
  "tasks",
  { "ids": ["task-1"], "max_traces": 1 }
)

DQL

INSERT INTO tasks DOCUMENTS ({ "_id": "task-1", "status": "in progress" })

See Transactions for details on creating, using, and committing a transaction in your SDK. The function returns NULL when the request succeeds.

Function arguments

The first argument names the collection. The optional second argument is an object that narrows which changes in that collection are eligible:

Option	Type	Behavior when present
`ids`	Array of document IDs	Only changes to the listed documents are eligible.
`max_traces`	Nonnegative integer	Limits how many matching changes this request can trace in the transaction.

If you omit an option, the request has no constraint for it. An empty ids array or max_traces value of 0 makes the request a no-op. Peer-wide limits still apply. Unknown option fields, invalid collection names or document IDs, and invalid option types return errors.

Transaction behavior

The function only applies to changes made later in the same transaction. Changes made before the call are not traced.
A successful commit makes the requested traces available for propagation. A rollback discards them.
If you execute the function outside an explicit transaction, Ditto runs it in an implicit write transaction. The call can succeed, but starts no trace by itself because no document change follows in that transaction.
The function requires a write transaction; it cannot run in a read-only transaction.
Multiple calls add separate requests and limits. A document gets at most one trace in a transaction, even if it matches multiple requests.
The peer’s trace rate and retention limits apply to these requests. See Set trace retention and rate limits.

Set trace retention and rate limits

Without limits, propagation tracing could consume significant memory on peers and increase network and backend load. These peer-level parameters bound those costs for both automatic and programmatic trace requests. You can change them at runtime with the ALTER SYSTEM statement.

Rate limiting

Parameter: propagation_trace_max_established_per_minute Default value: 12 Description: Sets the maximum steady-state rate at which the peer starts new traces. When the limit is reached, eligible changes are not traced until capacity becomes available. Requests are not queued; available capacity applies to later trace requests. For example, if an app requests traces for 120 eligible changes per minute and the peer uses the default limit of 12, the peer will generally start traces for about 10% of those changes over time. Each peer imposes this limit on the new traces it starts. It is not a shared budget. As the number of peers in the mesh increases, so can the overall rate of traces started by the mesh.

Local TTL

Parameter: propagation_trace_ttl_secs Default value: 60 Description: Sets the maximum time the peer retains its local knowledge of a trace after first observing the change. Because a peer can’t predict when other peers will reconnect to it, it can never know whether it has retransmitted a given change for the last time or not. To be absolutely certain it always propagates traces appropriately, it would theoretically have to retain knowledge of each trace forever. This system parameter gives the peer a time-based escape hatch that allows it to assume a trace is no longer relevant after it has held it for a certain period of time. For traces received from another peer, this TTL is relative to the moment that the local peer became aware of the trace, not from the moment the trace started. This avoids clock skew issues that could otherwise cause the trace to be prematurely terminated. A longer TTL reduces the likelihood that a trace will stop before reaching a late-joining peer or crossing a slow connection. It also keeps local spans open longer, delaying their visibility in the OpenTelemetry backend.

Maximum retained traces

Parameter: propagation_trace_max_count Default value: 100 Description: Sets the maximum number of active propagation traces that the peer retains at once. If the peer reaches this limit, it may expire its oldest trace records before their TTL. The peer logs a trace lifecycle event when this happens. The main purpose of this limit is to control memory use. If a peer frequently reaches this limit, and traces are incomplete as a result, you can:

Increase propagation_trace_max_count to retain more concurrent traces locally.
Decrease propagation_trace_max_established_per_minute on all the peers in the mesh to limit the total number of concurrent traces that need to be handled at a time.

Read and interpret a trace

In your OpenTelemetry backend, filter for service ditto and span name doc_trace. Backend interfaces may use different labels for these filters. Each trace follows one document change. The root span captures the activity on the peer where the change was made. Receiving peers add spans, and parent-child relationships show the route the change took through the mesh. The span attributes identify the peers and document involved:

local.peer_id identifies the peer represented by the span.
received_from.remote.peer_id identifies the peer that supplied the change.
collection and document_id identify the document. Within that document, dot identifies the specific change being traced.

The trace contains sync metadata, not document fields or content. If a peer receives the same change from multiple peers, the trace may show the redundant deliveries as links between spans instead of additional parent-child steps. To estimate how long a change took to reach another peer, compare the root span’s start time with the start time of that peer’s span. To estimate the time between two hops, compare a receiving span’s start time with its parent’s. These estimates use timestamps from different device clocks, so the clocks need to be reasonably synchronized. The trace TTL must also be long enough to cover the propagation delay you want to observe.

Limitations

Trace timing and visibility

No single peer manages the lifetime of an OpenTelemetry trace. Each peer keeps a local record containing its span and the context needed to continue the trace. propagation_trace_ttl_secs limits how long the peer keeps that record after first observing the change. When the record expires, the peer closes its span and exports it if OTLP is configured. Other peers may still have their own records and open spans. Span duration therefore reflects how long that peer retained the trace record, not how long it took the change to reach or pass through the peer. Use span start times to estimate propagation delay. TTL expiry is not a scheduled export deadline. A peer closes its span when trace maintenance notices that its local record has expired. Maintenance runs during Small Peer write transactions and when Ditto checks traces while preparing outgoing sync updates. When multihop presence is enabled (the default), Small Peers also write presence updates when connectivity changes and send keepalives every 30 seconds. These writes run trace maintenance too. In an active mesh, sync, store, and presence activity tend to close local spans near their TTL. An idle or offline peer, or a peer with multihop presence disabled, may keep an expired record longer. The span may then appear in the backend later than its TTL.

Trace coverage

Complete propagation tracing is only possible along paths where every peer can continue the trace and export its spans. When a peer sends a traced change to a peer that cannot continue tracing, it logs that the trace was not sent. The document change may still continue through the mesh, but the trace does not cross that boundary. Peers that cannot continue propagation traces include:

Small Peers running Ditto SDK versions earlier than 5.2.
Ditto Server.

Other conditions can also interrupt a trace:

A peer may restart after receiving a traced change but before forwarding it. The restart discards its local trace record.
A peer may wait longer than the configured TTL between receiving a traced change and relaying it. For example, if it is disconnected long enough for its local trace record to expire, the trace will not accompany the change when it later relays it.
Ditto’s optional reliable UDP multicast data-sync transport is a private beta and does not currently propagate trace context.

WASM-based SDK 5.2 peers do not include OpenTelemetry integration. They can log lifecycle events and forward trace context, but they cannot export their own spans. Other peers can continue the trace across them, so the resulting parent-child span relationships skip the WASM peers. Propagation tracing shows the route and timing, but does not explain why a peer takes a long time to relay a change. Use the spans to locate a slow hop. Then check that peer’s network, sync, DQL, and application telemetry for the cause.

Troubleshooting

Symptom	Check
No spans appear	Confirm that `OTEL_EXPORTER_OTLP_ENDPOINT` is set before Ditto initializes, that each exporting peer can reach the endpoint, and that `OTEL_EXPORTER_SAMPLING_RATIO` is set high enough to include the trace.
Automatic tracing does not start	Confirm that the collection is in `propagation_trace_auto_trace_collections`, the setting was applied before the write transaction began, and the peer has not reached its trace admission rate limit.
A trace stops at a peer	Check the SDK version and sync path. Peers using an SDK earlier than 5.2 and Ditto Server do not emit or forward propagation spans.
A trace appears later than expected	Check whether the peer has performed trace maintenance since the TTL elapsed. A later write, outgoing sync check, or (if enabled) multihop presence update may trigger maintenance and export. Idle or offline peers may retain expired records longer. See Trace timing and visibility.
The programmatic trace request fails	Confirm that `dql_enable_preview_mode` is enabled and that the function runs in an active write transaction before the target changes.

Appendix: Try Jaeger locally

Jaeger accepts OTLP and provides a web interface for viewing traces. To run it locally with Docker:

docker run --rm -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

If the Ditto process can reach Jaeger at localhost, set OTEL_EXPORTER_OTLP_ENDPOINT to http://localhost:4317 for gRPC or http://localhost:4318 for http/protobuf. If Ditto runs in a container or on another device, use an endpoint reachable from that process. Open http://localhost:16686 on the machine running Jaeger to view traces. For production, use your organization’s collector and backend configuration.

Overview & Setup

SDKs

Accessing Data

Syncing Data

Authentication and Authorization

Managing the Mesh

Deploying your App

Tracing Document Propagation

Requirements

Configure OpenTelemetry export

Automatic collection tracing

Programmatic tracing in a transaction

Function arguments

Transaction behavior

Set trace retention and rate limits

Rate limiting

Local TTL

Maximum retained traces

Read and interpret a trace

Limitations

Trace timing and visibility

Trace coverage

Troubleshooting

Appendix: Try Jaeger locally

​Requirements

​Configure OpenTelemetry export

​Automatic collection tracing

​Programmatic tracing in a transaction

​Function arguments

​Transaction behavior

​Set trace retention and rate limits

​Rate limiting

​Local TTL

​Maximum retained traces

​Read and interpret a trace

​Limitations

​Trace timing and visibility

​Trace coverage

​Troubleshooting

​Appendix: Try Jaeger locally

Requirements

Configure OpenTelemetry export

Automatic collection tracing

Programmatic tracing in a transaction

Function arguments

Transaction behavior

Set trace retention and rate limits

Rate limiting

Local TTL

Maximum retained traces

Read and interpret a trace

Limitations

Trace timing and visibility

Trace coverage

Troubleshooting

Appendix: Try Jaeger locally