> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ditto.live/llms.txt
> Use this file to discover all available pages before exploring further.

# Diagnostics Capture

<Info>
  Operator diagnostics capture is available from Operator version 0.16.0 onwards.
</Info>

The Operator's diagnostics capability captures a complete snapshot of a Big Peer deployment in
a single command.

## How it works

The Operator's diagnostics capture gathers information about your Ditto Server deployment and Kubernetes environment to
aid troubleshooting your deployment, giving you a single artifact to share when you contact Ditto support, speeding up
issue resolution.

The Ditto Operator is built for restricted environments. Collection happens entirely through the Kubernetes
API and pod HTTP endpoints. It never needs `pods/exec` and makes no outbound network calls
beyond the cluster's API server, making it safe to run in air-gapped and
compliance-sensitive clusters.

Resources are discovered by enumerating everything in the namespaces you name. You specify the target namespaces, so you
remain in control of what gets collected.

Each run collects:

* Kubernetes events
* Container logs
* Scraped HTTP endpoints (the built-in `/metrics` and `/debug`)
* Ditto CR definitions
* Core Kubernetes objects
* Strimzi/Kafka resources
* A handful of cluster-scoped resources such as `Node`, `StorageClass`, and `PersistentVolume`

<Note>
  **Sensitive data is redacted by default.**

  This includes:

  * Kubernetes Secrets
  * `ConfigMap` keys, labels and annotations whose keys appear sensitive, such as those containing `token`, `password`, `secret`, `key`, `cred`, etc.
  * Tokens spotted in event messages such as JWTs, long base64 blobs, and `Authorization` or `Cookie` headers

  This redaction is **best effort**. In a support engagement with Ditto, we advise users to review the bundle prior to sharing with Ditto.
</Note>

## Running diagnostics capture

You capture diagnostics by creating a `BigPeerManagerTask`: the Operator runs the collection
in-cluster (with the permissions it needs) and writes the archive to storage you control.

## Creating a BigPeerManagerTask

A `BigPeerManagerTask` describes a maintenance task that the Operator runs for you. Today,
`diagnostics` is the only supported task. The fields under `spec.task.diagnostics` configure what
each run collects (`namespaces`, `system`, `logMaxLines`, `maxArchives`, and so on), and `output`
configures where the archive is written. The task runs the Operator's own image and version by
default; pin them with `spec.version` and `spec.image` only if you need a specific build.

You can trigger a task in two ways:

* **On demand**: create the task suspended, then run it once whenever you need a bundle. Best for
  capturing state during active troubleshooting.
* **On a recurring schedule**: give the task a cron schedule and the Operator runs it
  automatically. Best for capturing state continuously, so a bundle already exists when an
  incident happens.

<Info>
  The Operator provisions the identity each task's underlying `CronJob` runs as. There's a dedicated
  `ServiceAccount` and a `ClusterRoleBinding` to the cluster-wide read `ClusterRole`
  (`ditto-bpmt-diagnostics`), which the Ditto Operator Helm chart installs. A standard Operator Helm
  install therefore already has everything the task needs.
</Info>

### Run once on demand

To collect a single bundle without waiting for a schedule, create the task with `suspend: true`
so it never fires on its own (a `schedule` is still required, but is ignored while suspended), and
write to a `PersistentVolumeClaim` the Operator provisions and owns so the bundle persists once
the run completes. `namespaces` is omitted, so only the task's own namespace is collected (it is
always included):

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: diagnostics
spec:
  schedule: '0 0 * * *'
  suspend: true
  task:
    diagnostics:
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            provisioned:
              size: 1Gi
EOF
```

This will write the bundle to a 1Gi claim provisioned from
the cluster's default `StorageClass`. Retrieve it as described in
[Retrieving the archive](#retrieving-the-archive).

### Run on a recurring schedule

To collect a bundle continuously rather than on demand, drop `suspend` and set `schedule` to
the cron expression you want. The Operator then runs the capture automatically on that cadence,
with no manual `Job` trigger needed:

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: diagnostics
spec:
  schedule: '0 */6 * * *'
  task:
    diagnostics:
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            provisioned:
              size: 1Gi
EOF
```

This runs every six hours. Because the storage outlives individual runs, older bundles are
pruned per `maxArchives` (default `10`) and you can
fetch any that remain (see [Retrieving the archive](#retrieving-the-archive)).

### Collecting additional namespaces

To collect more than the task's own namespace, list the extra ones under `namespaces`.

Set `system: true` to also include `kube-system`:

```yaml theme={null}
  task:
    diagnostics:
      namespaces:
      - kafka
      - cert-manager
      system: true
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            emptyDir: {}
```

### Choosing where the archive goes

The `output.file.storage` field selects the volume that backs the archive directory. There are
three options, from least to most durable.

An ephemeral `emptyDir`, discarded when the pod completes *(mainly useful for testing the scheduling and execution of the task)*:

```yaml theme={null}
          storage:
            emptyDir: {}
```

A `PersistentVolumeClaim` the Operator creates, owns, and reuses across runs (and deletes when
the task is deleted):

```yaml theme={null}
          storage:
            provisioned:
              size: 1Gi
              storageClass: gp3
```

An existing `PersistentVolumeClaim` that you manage yourself:

```yaml theme={null}
          storage:
            pvc:
              claimName: my-diagnostics-pvc
```

With `provisioned` and `pvc`, the bundles outlive individual runs, so you can fetch them later
(see [Retrieving the archive](#retrieving-the-archive)).

## Retrieving the archive

With `provisioned` or `pvc` storage, the bundles persist on the `pvc` after each run. Because the
`pvc` lives inside the cluster, you retrieve a bundle by mounting it into a short-lived pod and
copying the file out with `kubectl cp`. Run these commands in the same namespace as the task.

<Steps>
  <Step title="Find the backing PVC">
    For a `provisioned` task, the Operator records the `pvc` it created under the task's status.
    Read its name into a variable (for a `pvc` task, use your own claim name instead):

    ```bash theme={null}
    PVC=$(kubectl get bigpeermanagertask diagnostics -o jsonpath='{.status.pvcRef.name}')
    echo "$PVC"
    ```
  </Step>

  <Step title="Start a temporary pod that mounts the claim">
    Create a small pod that mounts the `pvc` at the task's `output.file.path`
    (`/var/lib/diagnostics` in these examples) and idles so you can copy from it:

    ```bash theme={null}
    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Pod
    metadata:
      name: diagnostics-fetch
    spec:
      restartPolicy: Never
      containers:
      - name: shell
        image: busybox
        command: ["sleep", "3600"]
        volumeMounts:
        - name: archive
          mountPath: /var/lib/diagnostics
      volumes:
      - name: archive
        persistentVolumeClaim:
          claimName: $PVC
    EOF

    kubectl wait --for=condition=Ready pod/diagnostics-fetch
    ```
  </Step>

  <Step title="Copy the bundle to your machine">
    The most recent run is always available as `latest_diagnostics.tar.xz`. Copy it out:

    ```bash theme={null}
    kubectl cp diagnostics-fetch:/var/lib/diagnostics/latest_diagnostics.tar.xz ./latest_diagnostics.tar.xz
    ```
  </Step>

  <Step title="Unpack the archive">
    Extract the bundle to inspect or share its contents:

    ```bash theme={null}
    mkdir -p ./diagnostics-bundle
    tar -xf latest_diagnostics.tar.xz -C ./diagnostics-bundle
    ```
  </Step>

  <Step title="Clean up the temporary pod">
    Delete the pod once you have the bundle. The claim and its archives are left untouched:

    ```bash theme={null}
    kubectl delete pod diagnostics-fetch
    ```
  </Step>
</Steps>

## What's in the archive

The archive is organised by scope, then namespace and resource, so you can browse it with
`tar tf` before unpacking:

```text theme={null}
metadata.yaml
namespace/<namespace>/<group>_<version>_<kind>/<name>/manifest.yaml
namespace/<namespace>/<group>_<version>_<kind>/<name>/events.json
namespace/<namespace>/<group>_<version>_<kind>/<name>/logs/<container>.log
cluster_scoped/<group>_<version>_<kind>/<name>/manifest.yaml
endpoints/<namespace>/<pod>/metadata.json
endpoints/<namespace>/<pod>/<path>-<port>.txt
```

The `metadata.yaml` at the root records how the bundle was produced (the Operator version,
the flags used, and the namespaces collected), along with a per-resource count of what was
gathered. It's the first thing to open.

A run never aborts on a single failure. If a collector or an endpoint scrape fails, for
example because of missing RBAC, the run records each failure under `collector_errors` in
`metadata.yaml` and still writes whatever it managed to collect, so even a partial bundle is
useful.

## Advanced configuration

### Scraping HTTP endpoints inline

By default the task scrapes the built-in `/metrics` and `/debug` endpoints on every pod
labelled `ditto.live/big-peer`. You can add your own mappings under `endpoints`, each pairing a
pod label selector with one or more target endpoints. The Operator renders the list into a
controller-owned `ConfigMap` and mounts it into the pod:

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: scraping-inline
spec:
  schedule: '*/5 * * * *'
  task:
    diagnostics:
      namespaces:
      - ditto
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            provisioned:
              size: 1Gi
      endpoints:
      - selector:
          matchLabels:
            strimzi.io/cluster: ditto
            strimzi.io/kind: Kafka
        targets:
        - path: /metrics
          port: 9404
      - selector:
          matchExpressions:
          - key: app.kubernetes.io/component
            operator: In
            values:
            - exporter
            - metrics
        targets:
        - path: /metrics
          port: 8080
        - path: /healthz
          port: http
EOF
```

A target's `port` is either a number (`9404`) or the name of a container port (`http`).

<Warning>
  Scraped response bodies are written to the archive verbatim and are **not** redacted.
  Review the bundle before sharing it.
</Warning>

### Scraping HTTP endpoints via a ConfigMap

If you'd rather manage the scraping configuration yourself, put it in a `ConfigMap` as a
`diagnostics.yaml` document and point the task at it with `configMap`. This is mutually
exclusive with the inline `endpoints` field:

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: scraping-cm
data:
  diagnostics.yaml: |
    endpoints:
    - selector:
        matchLabels:
          strimzi.io/cluster: ditto
          strimzi.io/kind: Kafka
      targets:
      - path: /metrics
        port: 9404
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerManagerTask
metadata:
  name: scraping-cm
spec:
  schedule: '*/5 * * * *'
  task:
    diagnostics:
      namespaces:
      - ditto
      output:
        file:
          path: /var/lib/diagnostics
          storage:
            emptyDir: {}
      configMap:
        name: scraping-cm
EOF
```

The `configMap.key` field is optional; when omitted, the controller reads the
`diagnostics.yaml` key (shown above).

### Other task settings

The examples above cover the common fields. `BigPeerManagerTask` also exposes the standard
Kubernetes `CronJob` controls: `suspend` to pause a schedule without deleting the task,
`concurrencyPolicy`, `successfulJobsHistoryLimit`/`failedJobsHistoryLimit`, and a `template`
for customising the generated `CronJob`, pod, and container (image pull secrets, tolerations,
and so on).

For the complete, version-accurate field reference, inspect the CRD on your own cluster:

```bash theme={null}
kubectl explain bigpeermanagertask.spec
```