Diagnostics Capture

Operator diagnostics capture is available from Operator version 0.16.0 onwards.

The Operator’s diagnostics capability captures a complete snapshot of a Big Peer deployment in a single command.

How it works

The Operator’s diagnostics capture gathers information about your Ditto Server deployment and Kubernetes environment to aid troubleshooting your deployment, giving you a single artifact to share when you contact Ditto support, speeding up issue resolution. The Ditto Operator is built for restricted environments. Collection happens entirely through the Kubernetes API and pod HTTP endpoints. It never needs pods/exec and makes no outbound network calls beyond the cluster’s API server, making it safe to run in air-gapped and compliance-sensitive clusters. Resources are discovered by enumerating everything in the namespaces you name. You specify the target namespaces, so you remain in control of what gets collected. Each run collects:

Kubernetes events
Container logs
Scraped HTTP endpoints (the built-in /metrics and /debug)
Ditto CR definitions
Core Kubernetes objects
Strimzi/Kafka resources
A handful of cluster-scoped resources such as Node, StorageClass, and PersistentVolume

Sensitive data is redacted by default.This includes:

Kubernetes Secrets
ConfigMap keys, labels and annotations whose keys appear sensitive, such as those containing token, password, secret, key, cred, etc.
Tokens spotted in event messages such as JWTs, long base64 blobs, and Authorization or Cookie headers

This redaction is best effort. In a support engagement with Ditto, we advise users to review the bundle prior to sharing with Ditto.

Running diagnostics capture

You capture diagnostics by creating a BigPeerManagerTask: the Operator runs the collection in-cluster (with the permissions it needs) and writes the archive to storage you control.

Creating a BigPeerManagerTask

A BigPeerManagerTask describes a maintenance task that the Operator runs for you. Today, diagnostics is the only supported task. The fields under spec.task.diagnostics configure what each run collects (namespaces, system, logMaxLines, maxArchives, and so on), and output configures where the archive is written. The task runs the Operator’s own image and version by default; pin them with spec.version and spec.image only if you need a specific build. You can trigger a task in two ways:

On demand: create the task suspended, then run it once whenever you need a bundle. Best for capturing state during active troubleshooting.
On a recurring schedule: give the task a cron schedule and the Operator runs it automatically. Best for capturing state continuously, so a bundle already exists when an incident happens.

The Operator provisions the identity each task’s underlying CronJob runs as. There’s a dedicated ServiceAccount and a ClusterRoleBinding to the cluster-wide read ClusterRole (ditto-bpmt-diagnostics), which the Ditto Operator Helm chart installs. A standard Operator Helm install therefore already has everything the task needs.

Run once on demand

To collect a single bundle without waiting for a schedule, create the task with suspend: true so it never fires on its own (a schedule is still required, but is ignored while suspended), and write to a PersistentVolumeClaim the Operator provisions and owns so the bundle persists once the run completes. namespaces is omitted, so only the task’s own namespace is collected (it is always included):

cat <<'EOF' | kubectl apply -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: diagnostics
spec:
  schedule: '0 0 * * *'
  suspend: true
  task:
    diagnostics:
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            provisioned:
              size: 1Gi
EOF

This will write the bundle to a 1Gi claim provisioned from the cluster’s default StorageClass. Retrieve it as described in Retrieving the archive.

Run on a recurring schedule

To collect a bundle continuously rather than on demand, drop suspend and set schedule to the cron expression you want. The Operator then runs the capture automatically on that cadence, with no manual Job trigger needed:

cat <<'EOF' | kubectl apply -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: diagnostics
spec:
  schedule: '0 */6 * * *'
  task:
    diagnostics:
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            provisioned:
              size: 1Gi
EOF

This runs every six hours. Because the storage outlives individual runs, older bundles are pruned per maxArchives (default 10) and you can fetch any that remain (see Retrieving the archive).

Collecting additional namespaces

To collect more than the task’s own namespace, list the extra ones under namespaces. Set system: true to also include kube-system:

  task:
    diagnostics:
      namespaces:
      - kafka
      - cert-manager
      system: true
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            emptyDir: {}

Choosing where the archive goes

The output.file.storage field selects the volume that backs the archive directory. There are three options, from least to most durable. An ephemeral emptyDir, discarded when the pod completes (mainly useful for testing the scheduling and execution of the task):

          storage:
            emptyDir: {}

A PersistentVolumeClaim the Operator creates, owns, and reuses across runs (and deletes when the task is deleted):

          storage:
            provisioned:
              size: 1Gi
              storageClass: gp3

An existing PersistentVolumeClaim that you manage yourself:

          storage:
            pvc:
              claimName: my-diagnostics-pvc

With provisioned and pvc, the bundles outlive individual runs, so you can fetch them later (see Retrieving the archive).

Retrieving the archive

With provisioned or pvc storage, the bundles persist on the pvc after each run. Because the pvc lives inside the cluster, you retrieve a bundle by mounting it into a short-lived pod and copying the file out with kubectl cp. Run these commands in the same namespace as the task.

Find the backing PVC

For a provisioned task, the Operator records the pvc it created under the task’s status. Read its name into a variable (for a pvc task, use your own claim name instead):

PVC=$(kubectl get bigpeermanagertask diagnostics -o jsonpath='{.status.pvcRef.name}')
echo "$PVC"

Start a temporary pod that mounts the claim

Create a small pod that mounts the pvc at the task’s output.file.path (/var/lib/diagnostics in these examples) and idles so you can copy from it:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: diagnostics-fetch
spec:
  restartPolicy: Never
  containers:
  - name: shell
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: archive
      mountPath: /var/lib/diagnostics
  volumes:
  - name: archive
    persistentVolumeClaim:
      claimName: $PVC
EOF

kubectl wait --for=condition=Ready pod/diagnostics-fetch

Copy the bundle to your machine

The most recent run is always available as latest_diagnostics.tar.xz. Copy it out:

kubectl cp diagnostics-fetch:/var/lib/diagnostics/latest_diagnostics.tar.xz ./latest_diagnostics.tar.xz

Unpack the archive

Extract the bundle to inspect or share its contents:

mkdir -p ./diagnostics-bundle
tar -xf latest_diagnostics.tar.xz -C ./diagnostics-bundle

Clean up the temporary pod

Delete the pod once you have the bundle. The claim and its archives are left untouched:

kubectl delete pod diagnostics-fetch

What’s in the archive

The archive is organised by scope, then namespace and resource, so you can browse it with tar tf before unpacking:

metadata.yaml
namespace/<namespace>/<group>_<version>_<kind>/<name>/manifest.yaml
namespace/<namespace>/<group>_<version>_<kind>/<name>/events.json
namespace/<namespace>/<group>_<version>_<kind>/<name>/logs/<container>.log
cluster_scoped/<group>_<version>_<kind>/<name>/manifest.yaml
endpoints/<namespace>/<pod>/metadata.json
endpoints/<namespace>/<pod>/<path>-<port>.txt

The metadata.yaml at the root records how the bundle was produced (the Operator version, the flags used, and the namespaces collected), along with a per-resource count of what was gathered. It’s the first thing to open. A run never aborts on a single failure. If a collector or an endpoint scrape fails, for example because of missing RBAC, the run records each failure under collector_errors in metadata.yaml and still writes whatever it managed to collect, so even a partial bundle is useful.

Advanced configuration

Scraping HTTP endpoints inline

By default the task scrapes the built-in /metrics and /debug endpoints on every pod labelled ditto.live/big-peer. You can add your own mappings under endpoints, each pairing a pod label selector with one or more target endpoints. The Operator renders the list into a controller-owned ConfigMap and mounts it into the pod:

cat <<'EOF' | kubectl apply -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: scraping-inline
spec:
  schedule: '*/5 * * * *'
  task:
    diagnostics:
      namespaces:
      - ditto
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            provisioned:
              size: 1Gi
      endpoints:
      - selector:
          matchLabels:
            strimzi.io/cluster: ditto
            strimzi.io/kind: Kafka
        targets:
        - path: /metrics
          port: 9404
      - selector:
          matchExpressions:
          - key: app.kubernetes.io/component
            operator: In
            values:
            - exporter
            - metrics
        targets:
        - path: /metrics
          port: 8080
        - path: /healthz
          port: http
EOF

A target’s port is either a number (9404) or the name of a container port (http).

Scraped response bodies are written to the archive verbatim and are not redacted. Review the bundle before sharing it.

Scraping HTTP endpoints via a ConfigMap

If you’d rather manage the scraping configuration yourself, put it in a ConfigMap as a diagnostics.yaml document and point the task at it with configMap. This is mutually exclusive with the inline endpoints field:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: scraping-cm
data:
  diagnostics.yaml: |
    endpoints:
    - selector:
        matchLabels:
          strimzi.io/cluster: ditto
          strimzi.io/kind: Kafka
      targets:
      - path: /metrics
        port: 9404
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: scraping-cm
spec:
  schedule: '*/5 * * * *'
  task:
    diagnostics:
      namespaces:
      - ditto
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            emptyDir: {}
      configMap:
        name: scraping-cm
EOF

The configMap.key field is optional; when omitted, the controller reads the diagnostics.yaml key (shown above).

Other task settings

The examples above cover the common fields. BigPeerManagerTask also exposes the standard Kubernetes CronJob controls: suspend to pause a schedule without deleting the task, concurrencyPolicy, successfulJobsHistoryLimit/failedJobsHistoryLimit, and a template for customising the generated CronJob, pod, and container (image pull secrets, tolerations, and so on). For the complete, version-accurate field reference, inspect the CRD on your own cluster:

kubectl explain bigpeermanagertask.spec

Ditto Operator

Connectors

Ditto Operator API

Reference

Diagnostics Capture

How it works

Running diagnostics capture

Creating a BigPeerManagerTask

Run once on demand

Run on a recurring schedule

Collecting additional namespaces

Choosing where the archive goes

Retrieving the archive

What’s in the archive

Advanced configuration

Scraping HTTP endpoints inline

Scraping HTTP endpoints via a ConfigMap

Other task settings

​How it works

​Running diagnostics capture

​Creating a BigPeerManagerTask

​Run once on demand

​Run on a recurring schedule

​Collecting additional namespaces

​Choosing where the archive goes

​Retrieving the archive

​What’s in the archive

​Advanced configuration

​Scraping HTTP endpoints inline

​Scraping HTTP endpoints via a ConfigMap

​Other task settings

How it works

Running diagnostics capture

Creating a BigPeerManagerTask

Run once on demand

Run on a recurring schedule

Collecting additional namespaces

Choosing where the archive goes

Retrieving the archive

What’s in the archive

Advanced configuration

Scraping HTTP endpoints inline

Scraping HTTP endpoints via a ConfigMap

Other task settings