Skip to main content
A virtual collection is a view of some data presented as a DQL accessible collection.
The database_id field in system:active_requests, system:request_history, and system:shared_statements holds your database ID. SDK 5.0 called it app_id; SDK 5.1 and later use database_id.
A number of virtual collections are available in the system namespace to monitor the DQL environment, peer information, and provide statement diagnostics. They exist solely for local inspection and debugging, providing runtime visibility into the current peer’s state, active connections, and query execution details. These collections are local-only and are not synced to other peers or Ditto Server.

system:active_requests

ACTIVES is an alias for ACTIVE_REQUESTS.
Uses timer driven observer refreshing. See: timer driven observers.
Reports information on currently executing requests. The information is dynamic and will change as request processing takes place.
The output includes dynamic timing information and the query execution plan which can be useful when diagnosing issues. Some points to note about the content:
  1. When executing in an environment with multiple subscription servers, only data from the servicing server is reported.
  2. It is expected to see any query retrieving data from active_requests in the output in “executing” state.
  3. Operators will record information whilst processing so it is normal to see changes in the content for active_requests.
  4. Zero values (times, counts) are omitted from the output.

system:all_collections

Reports all collections in all namespaces.

system:collections

Reports all collections in the default namespace.

system:data_sync_info

Reports statistics related to data sync operations.

system:indexes

Reports the user defined indexes (Ditto SDK only).
Format Change in SDK 5.0+: The index information now includes the direction field for each index key, indicating whether the index is sorted in ascending ("asc") or descending ("desc") order. Earlier versions did not include this field.

system:metrics

Available in SDK version 5.0+. Uses timer driven observer refreshing. See: timer driven observers.
Reports Ditto’s internal metrics (counters, gauges and histograms) as queryable documents. You can read a device’s current metrics directly with DQL without setting up a monitoring pipeline or dashboard. To use this collection, set the metrics_exporter_virtual_collection_enabled system parameter to true at startup. Changing it at runtime has no effect. If it isn’t set, the collection returns one document with a status of "disabled".
Each document represents a single metric instance:
  1. key is the metric name.
  2. labels holds the labels for this metric, such as remote for a connection’s address.
  3. level, unit and description describe the metric.
  4. Counters and gauges report three values:
    • current is the current value.
    • updates is the number of updates recorded.
    • delta is the change since the previous query.
  5. Histograms report count, sample_rate, sum and abs_max. They also report one field for each quantile set in metrics_exporter_virtual_collection_histogram_quantiles. By default these fields are min, p50, p90, p95, p99 and max.
  6. dcount, dsum, dp50 and other fields that start with d are histogram delta fields. A delta field covers only the values recorded since your previous query, and each histogram field has a delta version.
The metrics_exporter_virtual_collection_level system parameter controls which metrics are included. The default is "info". Set it to "debug" or "trace" to include more detailed metrics.

system:network_quic

Available in SDK version 5.1+
Reports the peer’s authenticated QUIC connections, with one document per connection. Each document shows the remote peer, the local peer’s role and the connection’s settings.
Some points to note about the content:
  1. This collection doesn’t include runtime data such as round-trip time, discovered MTU or byte counts. To get these values, query system:metrics for keys under the ditto.network.quic prefix. Each value’s remote label holds the remote address.
  2. layer is the network layer, based on the local address prefix. It is l2, l3 or unknown.
  3. side shows whether the local peer is the client or the server for the connection.
  4. The collection is empty unless next-generation networking (NGN) is enabled.
  5. Documents have no _id field.

system:network_routes

Available in SDK version 5.1+
Reports the IP router’s route table, with one document per route. The output is similar to the output of the Linux ip -6 route command.
Some points to note about the content:
  1. active is true for installed routes and false for inactive (backup) routes.
  2. held_down marks a route that the routing protocol has withdrawn but that stays installed until its hold-down timer ends. A held-down route still reports active as true.
  3. The collection is empty unless next-generation networking (NGN) is enabled.
  4. Documents have no _id field.

system:peers

Available in SDK version 5.1+
Reports the peers that the presence system can see, with one document per peer. The results include the local peer.
Some points to note about the content:
  1. peerKey is the peer’s public key.
  2. peerMetadata holds metadata that your app attaches to this peer’s presence.
  3. identityServiceMetadata holds metadata from the identity service.

system:sync_degradations

Available in SDK version 5.2+
Reports documents whose sync is degraded, with one result per affected document. Sync is degraded when an incoming update can’t be applied to a document. For example, the update would make the document larger than the hard size limit.
Some points to note about the content:
  1. _id is a hash of the collection name and document ID.
  2. senderPeerKey is the key of the peer that sent the update.
  3. reason is currently always documentTooBig.
  4. Ditto Server doesn’t support this collection. Queries on Ditto Server return an error.

system:transport_connections

Available in SDK version 5.1+
Reports the state of the mesh’s transport connections, with one document per connection. The results include connection candidates.
Some points to note about the content:
  1. state is one of discovered, connecting, connected or disconnecting.
  2. side shows whether the local peer is the client or the server for the connection.
  3. type is force for connections your app configures, such as WebSocket URLs, and Ditto always tries them.
  4. type is candidate for peers that Ditto discovers, and Ditto connects to them only when the mesh needs them.
  5. remote_peer is only present once the remote peer’s key is known.
  6. Ditto Server doesn’t support this collection. Queries on Ditto Server return an empty result.

system:shared_statements

Available in SDK version 5.0+
Uses timer driven observer refreshing. See: timer driven observers.
Reports information about statements in the shared statement cache. The shared statement cache stores prepared query plans to avoid re-planning identical queries, improving query performance.
This virtual collection provides visibility into which queries are being cached and can help diagnose query planning behavior. Related Configuration:
  • Use the #reprepare directive to bypass the cache for specific queries
  • Set dql_default_directives with #reprepare to control caching behavior globally
See Directives - Default Directives for more information on controlling statement cache behavior.

system:support_bundles

Available in SDK version 5.2+. Not available in the JavaScript Web SDK.
Lists the diagnostic support bundles retained on the device (Ditto SDK only). Rows appear when a bundle is generated with EXECUTE FUNCTION system_request_support_bundle() (see Capturing Support Bundles).
  • _id.pk: the peer key of the device that generated the bundle; _id.created_at: the capture timestamp (RFC 3339).
  • reason: USER_INITIATED for on-demand requests, CRASH for crash-triggered captures.
  • file: an attachment token for the bundle tar; pass it to the attachment-fetch API to retrieve the bytes. file.metadata carries any caller-supplied metadata from the request.
Related Configuration:
  • support_bundle_request_rate, support_bundle_request_timeout, and support_bundle_retention_limit system parameters control admission, capture time, and on-device retention (ALTER SYSTEM)

system:dual

Uses timer driven observer refreshing. See: timer driven observers. This collection’s content is static so an observer of it is not recommended.
Contains a single document with a single field. It can be used to execute and test DQL statements.

system:request_history

Records information about requests that have completed. This is the same basic information as reported by active_requests. Information is only recorded when some aspect of the request execution meets the user-configurable qualifiers. (See configuration.)
This is an in-memory cache and is not persisted across processes.
When creating an observer that selects from system:request_history care should be taken that the observer statement itself is not captured by the qualifiers else an infinite loop may be established.For example, setting the threshold qualifier to 0 means the observer statement is guaranteed to be entered into the history cache triggering another invocation of the observer statement.In this situation it is important to note too that observer statement filtering (e.g. WHERE requestType != "observer") will only stop the callback from firing and not the infinite execution loop.
The output includes complete timing information and the query execution plan, along with the qualifier that resulted in the request information being captured, all of which can be useful when diagnosing issues. request_history has a configurable maximum number of documents it will keep with oldest being dropped when attempting to add beyond the limit. (See configuration.) Some points to note about the content:
  1. When executing in an environment with multiple subscription servers, only data from the servicing server is reported.
  2. There should not be any requests in request_history in “executing” state
    1. The state should reflect the final state of the request: successful completion or failure.
  3. Zero values (times, counts) are omitted from the output.
The cache content may be (selectively) cleared with a DELETE statement. See also: Diagnosing problems

SYSTEM:SYSTEM_INFO

Reports a system information dump’s contents.

SYSTEM:TRANSPORTS_INFO

Reports available transports information.

SYSTEM:VITALS

Uses timer driven observer refreshing. See: timer driven observers.
Reports overview statistics about the query engine. It is typically the starting point for general query engine diagnostics. When executing in an environment with multiple subscription servers, only data from the servicing server is reported.
Some points to note about the content:
  1. The histograms decay over a 15 minute interval.
  2. “other” is all statements not covered by the individual metrics, e.g. ALTER SYSTEM
  3. All timings are in nanoseconds

Configuration

Where appropriate, the virtual collections content is determined by configuration settings allowing a user some control over what information is available.

Request History

The system:request_history collection’s size is determined by the DQL_REQUEST_HISTORY_SIZE system parameter, defaulting to 4096, indicating the number of entries to retain. The larger the size and the larger the statements captured, the greater the memory use.

Qualifiers

Which statements qualify for inclusion in the system:request_history collection is determined by the DQL_REQUEST_HISTORY_QUALIFIERS system parameter which can be set to an object listing all desired qualifiers. Qualifiers not included in the object are removed (or reset). The following qualifiers can be set:
  1. threshold - the total execution time in milliseconds over which qualifies the request for inclusion. This is set by default to 1000 (1 second) and is always active. It can be set to 0 to include all statements.
  2. failed - if a statement’s execution fails. true by default.
  3. system_delete - any delete statement run against a system namespace collection. true by default.
  4. mutation - if a statement is a mutation of any sort. Not active (false) by default.

Slow Request Warnings

Ditto can periodically log a warning to the Ditto log for any DQL request that is still running after a configurable threshold, containing the same information reported by system:active_requests. This is particularly useful on Small Peers, where querying system:active_requests live isn’t always practical, and helps distinguish a “hanging” request from one that’s simply processing a large or slow statement. This is controlled by the DQL_SLOW_REQUEST_WARN_SECONDS system parameter, defaulting to 60 seconds. The warning repeats at this interval for as long as the request keeps running.
Set to 0 to disable slow-request warnings. For more information, see ALTER SYSTEM.

Request Timeout

The DQL_REQUEST_TIMEOUT_SECONDS system parameter limits how long, in seconds, a DQL request is permitted to run before it is cancelled, defaulting to 0 (no limit).
Cancellation is signalled asynchronously to the execution pipeline rather than interrupting the currently executing operator, so it does not guarantee that execution stops immediately once the limit is reached. For more information, see ALTER SYSTEM.

Diagnosing problems

Profile analysis

The profile (#stats elements added to the plan) information permits assessment of where in the execution the time was spent and is therefore useful in directing next steps. request_history entries show how many results the statement and each of its operators processed. This may give insight into what may need to be changed in a statement. Analysis of the profile from multiple runs of the same statement may highlight things like selectivity-skew in filter values.
Any definition of “high” times should be with respect to the volume of data processed.
Some possible indications from profile analysis include:
  1. Stalled execution: If a request appears in the active_requests output but doesn’t appear to be making any progress it likely indicates an issue with the earliest (apart from sequence) operator in the plan.
    1. Commonly indicated by unchanging document counts anywhere in the profile.
  2. Excessive processing: An operator with a high document count potentially indicates this.
    1. Possibly a filter isn’t as selective as thought, etc.
    2. It may also show where “dam” operators are requiring all prior plan processing to complete before the remainder of the plan can run.
      1. e.g. a GROUP BY needs all input records before it can begin emitting the groups.
  3. Data access issues: Underlying collection access issues may be indicated by high scan operator receive times.
    1. High scan or fetch times are potentially indicative of collection access issues.
    2. High index scan receive times are potentially indicative of index access issues.
  4. Selectivity issues: Indications of index selectivity as compared to the entire predicate’s selectivity may be highlighted with a large number of documents being excluded by the filter operator.
    1. The difference in document in & out counts for the filter operator indicate the filtering taking place.
      1. A large amount of filtering may not indicate a problem if there are filters that cannot be applied by index scans.
      2. A large amount of filtering may indicate the query would benefit from different or additional indexes.
  5. Memory requirements: Document counts for operators that must process the entire prior pipeline - such as sorting and grouping - may give an indication of which statements will have larger memory requirements.
Further:
  1. Bottlenecks downstream in a plan may be indicated by a high operator send time.
    1. This may be observed for any operator and attention should be focused on later operators.
  2. Bottlenecks upstream in a plan may be indicated by a high operator recv time.
    1. This may be observed for any operator and attention should be focused on earlier operators.
  3. Possible problems in an operator may be highlighted by a high exec time.
    1. This must be considered in conjunction with the number of documents being processed.
    2. Operator issues should be reported via support channels.

SYSTEM:ACTIVE_REQUESTS & SYSTEM:REQUEST_HISTORY

When initially developing DQL statements the explain statement is used to determine the execution plan a query will use. Once deployed it may not always be possible to revert to examination of such a plan or there might be concern that the plan has materially changed. The system:active_requests and system:request_history collections contain1 instances of the actual execution plan used by a request, annotated with timing and document count information. These can be used to discover where in the execution a statement is taking time and can highlight things like unexpectedly low selectivity of filters or failure to find a suitable index.
Sometimes it may be possible to execute a statement directly when diagnosing issues. The PROFILE keyword appends the request_history entry (without having to pass qualification) to the statement’s results. This allows dynamic access to all the same information as is included in request_history without having to capture a live instance of its execution.
Producer operators are executed in parallel with their consumers and may show a much higher "documentsOut" count than the consumer’s "documentsIn". For completed requests, this is an indication that the consumer has stopped processing early and has not processed all of the documents sent by the producer. This might be perfectly normal, for instance when a LIMIT operator has hit the document count and has therefore stopped processing, or it may be that a downstream operator has encountered an error and has stopped the statement execution as a result.

1. Currently only available for select statements.

Timer driven observers

Store observers selecting from virtual collections with timer driven observer triggering are run in response to the timer trigger and not on collection content changes. The default timer interval is 500 milliseconds and can be changed with the live_query_system_collection_refresh_interval system parameter. The timer trigger only causes the observer DQL statement to be re-run. As with all observers, if the result set hasn’t changed since the previous run the observer callback is not invoked. Virtual collections drive observers via the timer mechanism when driving based on changes is not feasible, for example system:actives gains an entry when the observer query itself runs, so driving on changes would result in an infinite loop.