The
database_id field in system:active_requests, system:request_history, and system:shared_statements holds your database ID. SDK 5.0 called it app_id; SDK 5.1 and later use database_id.system:active_requests
ACTIVES is an alias for ACTIVE_REQUESTS.Uses timer driven observer refreshing. See: timer driven observers.
- When executing in an environment with multiple subscription servers, only data from the servicing server is reported.
- It is expected to see any query retrieving data from
active_requestsin the output in “executing” state. - Operators will record information whilst processing so it is normal to see changes in the content for
active_requests. - Zero values (times, counts) are omitted from the output.
system:all_collections
Reports all collections in all namespaces.system:collections
Reports all collections in thedefault namespace.
system:data_sync_info
Reports statistics related to data sync operations.system:indexes
Reports the user defined indexes (Ditto SDK only).Format Change in SDK 5.0+: The index information now includes the
direction field for each index key, indicating whether the index is sorted in ascending ("asc") or descending ("desc") order. Earlier versions did not include this field.system:metrics
Available in SDK version 5.0+. Uses timer driven observer refreshing. See: timer driven observers.
metrics_exporter_virtual_collection_enabled system parameter to true at startup. Changing it at runtime has no effect. If it isn’t set, the collection returns one document with a status of "disabled".
keyis the metric name.labelsholds the labels for this metric, such asremotefor a connection’s address.level,unitanddescriptiondescribe the metric.- Counters and gauges report three values:
currentis the current value.updatesis the number of updates recorded.deltais the change since the previous query.
- Histograms report
count,sample_rate,sumandabs_max. They also report one field for each quantile set inmetrics_exporter_virtual_collection_histogram_quantiles. By default these fields aremin,p50,p90,p95,p99andmax. dcount,dsum,dp50and other fields that start withdare histogram delta fields. A delta field covers only the values recorded since your previous query, and each histogram field has a delta version.
metrics_exporter_virtual_collection_level system parameter controls which metrics are included. The default is "info". Set it to "debug" or "trace" to include more detailed metrics.
system:network_quic
Available in SDK version 5.1+
- This collection doesn’t include runtime data such as round-trip time, discovered MTU or byte counts. To get these values, query system:metrics for keys under the
ditto.network.quicprefix. Each value’sremotelabel holds the remote address. layeris the network layer, based on the local address prefix. It isl2,l3orunknown.sideshows whether the local peer is theclientor theserverfor the connection.- The collection is empty unless next-generation networking (NGN) is enabled.
- Documents have no
_idfield.
system:network_routes
Available in SDK version 5.1+
ip -6 route command.
activeistruefor installed routes andfalsefor inactive (backup) routes.held_downmarks a route that the routing protocol has withdrawn but that stays installed until its hold-down timer ends. A held-down route still reportsactiveastrue.- The collection is empty unless next-generation networking (NGN) is enabled.
- Documents have no
_idfield.
system:peers
Available in SDK version 5.1+
peerKeyis the peer’s public key.peerMetadataholds metadata that your app attaches to this peer’s presence.identityServiceMetadataholds metadata from the identity service.
system:sync_degradations
Available in SDK version 5.2+
_idis a hash of the collection name and document ID.senderPeerKeyis the key of the peer that sent the update.reasonis currently alwaysdocumentTooBig.- Ditto Server doesn’t support this collection. Queries on Ditto Server return an error.
system:transport_connections
Available in SDK version 5.1+
stateis one ofdiscovered,connecting,connectedordisconnecting.sideshows whether the local peer is theclientor theserverfor the connection.typeisforcefor connections your app configures, such as WebSocket URLs, and Ditto always tries them.typeiscandidatefor peers that Ditto discovers, and Ditto connects to them only when the mesh needs them.remote_peeris only present once the remote peer’s key is known.- Ditto Server doesn’t support this collection. Queries on Ditto Server return an empty result.
system:shared_statements
Available in SDK version 5.0+
Uses timer driven observer refreshing. See: timer driven observers.
- Use the
#repreparedirective to bypass the cache for specific queries - Set
dql_default_directiveswith#reprepareto control caching behavior globally
system:support_bundles
Available in SDK version 5.2+. Not available in the JavaScript Web SDK.
EXECUTE FUNCTION system_request_support_bundle() (see Capturing Support Bundles).
_id.pk: the peer key of the device that generated the bundle;_id.created_at: the capture timestamp (RFC 3339).reason:USER_INITIATEDfor on-demand requests,CRASHfor crash-triggered captures.file: an attachment token for the bundle tar; pass it to the attachment-fetch API to retrieve the bytes.file.metadatacarries any caller-supplied metadata from the request.
support_bundle_request_rate,support_bundle_request_timeout, andsupport_bundle_retention_limitsystem parameters control admission, capture time, and on-device retention (ALTER SYSTEM)
system:dual
Uses timer driven observer refreshing. See: timer driven observers.
This collection’s content is static so an observer of it is not recommended.
system:request_history
Records information about requests that have completed. This is the same basic information as reported byactive_requests. Information is only recorded when some aspect of the request execution meets the user-configurable qualifiers. (See configuration.)
This is an in-memory cache and is not persisted across processes.
When creating an observer that selects from
system:request_history care should be taken that the observer statement itself is not captured by the qualifiers else an infinite loop may be established.For example, setting the threshold qualifier to 0 means the observer statement is guaranteed to be entered into the history cache triggering another invocation of the observer statement.In this situation it is important to note too that observer statement filtering (e.g. WHERE requestType != "observer") will only stop the callback from firing and not the infinite execution loop.request_history has a configurable maximum number of documents it will keep with oldest being dropped when attempting to add beyond the limit. (See configuration.)
Some points to note about the content:
- When executing in an environment with multiple subscription servers, only data from the servicing server is reported.
- There should not be any requests in
request_historyin “executing” state- The state should reflect the final state of the request: successful completion or failure.
- Zero values (times, counts) are omitted from the output.
DELETE statement.
See also: Diagnosing problems
SYSTEM:SYSTEM_INFO
Reports a system information dump’s contents.SYSTEM:TRANSPORTS_INFO
Reports available transports information.SYSTEM:VITALS
Uses timer driven observer refreshing. See: timer driven observers.
- The histograms decay over a 15 minute interval.
- “other” is all statements not covered by the individual metrics, e.g.
ALTER SYSTEM - All timings are in nanoseconds
Configuration
Where appropriate, the virtual collections content is determined by configuration settings allowing a user some control over what information is available.Request History
The system:request_history collection’s size is determined by theDQL_REQUEST_HISTORY_SIZE system parameter, defaulting to 4096, indicating the number of entries to retain. The larger the size and the larger the statements captured, the greater the memory use.
Qualifiers
Which statements qualify for inclusion in thesystem:request_history collection is determined by the DQL_REQUEST_HISTORY_QUALIFIERS system parameter which can be set to an object listing all desired qualifiers. Qualifiers not included in the object are removed (or reset).
The following qualifiers can be set:
threshold- the total execution time in milliseconds over which qualifies the request for inclusion. This is set by default to 1000 (1 second) and is always active. It can be set to 0 to include all statements.failed- if a statement’s execution fails.trueby default.system_delete- any delete statement run against asystemnamespace collection.trueby default.mutation- if a statement is a mutation of any sort. Not active (false) by default.
Slow Request Warnings
Ditto can periodically log a warning to the Ditto log for any DQL request that is still running after a configurable threshold, containing the same information reported by system:active_requests. This is particularly useful on Small Peers, where queryingsystem:active_requests live isn’t always practical, and helps distinguish a “hanging” request from one that’s simply processing a large or slow statement. This is controlled by the DQL_SLOW_REQUEST_WARN_SECONDS system parameter, defaulting to 60 seconds. The warning repeats at this interval for as long as the request keeps running.
0 to disable slow-request warnings.
For more information, see ALTER SYSTEM.
Request Timeout
TheDQL_REQUEST_TIMEOUT_SECONDS system parameter limits how long, in seconds, a DQL request is permitted to run before it is cancelled, defaulting to 0 (no limit).
Diagnosing problems
Profile analysis
The profile (#stats elements added to the plan) information permits assessment of where in the execution the time was spent and is therefore useful in directing next steps.
request_history entries show how many results the statement and each of its operators processed. This may give insight into what may need to be changed in a statement.
Analysis of the profile from multiple runs of the same statement may highlight things like selectivity-skew in filter values.
Some possible indications from profile analysis include:
- Stalled execution: If a request appears in the
active_requestsoutput but doesn’t appear to be making any progress it likely indicates an issue with the earliest (apart fromsequence) operator in the plan.- Commonly indicated by unchanging document counts anywhere in the profile.
- Excessive processing: An operator with a high document count potentially indicates this.
- Possibly a filter isn’t as selective as thought, etc.
- It may also show where “dam” operators are requiring all prior plan processing to complete before the remainder of the plan can run.
- e.g. a
GROUP BYneeds all input records before it can begin emitting the groups.
- e.g. a
- Data access issues: Underlying collection access issues may be indicated by high scan operator receive times.
- High
scanorfetchtimes are potentially indicative of collection access issues. - High index scan receive times are potentially indicative of index access issues.
- High
- Selectivity issues: Indications of index selectivity as compared to the entire predicate’s selectivity may be highlighted with a large number of documents being excluded by the
filteroperator.- The difference in document in & out counts for the
filteroperator indicate the filtering taking place.- A large amount of filtering may not indicate a problem if there are filters that cannot be applied by index scans.
- A large amount of filtering may indicate the query would benefit from different or additional indexes.
- The difference in document in & out counts for the
- Memory requirements: Document counts for operators that must process the entire prior pipeline - such as sorting and grouping - may give an indication of which statements will have larger memory requirements.
- Bottlenecks downstream in a plan may be indicated by a high operator
sendtime.- This may be observed for any operator and attention should be focused on later operators.
- Bottlenecks upstream in a plan may be indicated by a high operator
recvtime.- This may be observed for any operator and attention should be focused on earlier operators.
- Possible problems in an operator may be highlighted by a high
exectime.- This must be considered in conjunction with the number of documents being processed.
- Operator issues should be reported via support channels.
SYSTEM:ACTIVE_REQUESTS & SYSTEM:REQUEST_HISTORY
When initially developing DQL statements the explain statement is used to determine the execution plan a query will use. Once deployed it may not always be possible to revert to examination of such a plan or there might be concern that the plan has materially changed. Thesystem:active_requests and system:request_history collections contain1 instances of the actual execution plan used by a request, annotated with timing and document count information. These can be used to discover where in the execution a statement is taking time and can highlight things like unexpectedly low selectivity of filters or failure to find a suitable index.
Sometimes it may be possible to execute a statement directly when diagnosing issues. The
PROFILE keyword appends the request_history entry (without having to pass qualification) to the statement’s results. This allows dynamic access to all the same information as is included in request_history without having to capture a live instance of its execution.Producer operators are executed in parallel with their consumers and may show a much higher
"documentsOut" count than the consumer’s "documentsIn". For completed requests, this is an indication that the consumer has stopped processing early and has not processed all of the documents sent by the producer. This might be perfectly normal, for instance when a LIMIT operator has hit the document count and has therefore stopped processing, or it may be that a downstream operator has encountered an error and has stopped the statement execution as a result.1. Currently only available for select statements.
Timer driven observers
Store observers selecting from virtual collections with timer driven observer triggering are run in response to the timer trigger and not on collection content changes. The default timer interval is 500 milliseconds and can be changed with thelive_query_system_collection_refresh_interval system parameter.
The timer trigger only causes the observer DQL statement to be re-run. As with all observers, if the result set hasn’t changed since the previous run the observer callback is not invoked.
Virtual collections drive observers via the timer mechanism when driving based on changes is not feasible, for example system:actives gains an entry when the observer query itself runs, so driving on changes would result in an infinite loop.