> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ditto.live/llms.txt
> Use this file to discover all available pages before exploring further.

# Attachments

> Configure production attachment storage for a self-managed Big Peer on Amazon S3 or Azure Blob Storage.

<Info>
  The Amazon S3 and S3-compatible attachment backends are available from Operator version 0.1.4 onwards. The Azure Blob Storage backend is available from Operator version 0.17.1 onwards.
</Info>

[Attachments](/sdk/latest/crud/working-with-attachments) let your apps link large binary data — images, video, PDFs — to documents. Big Peer keeps those blobs in an object store called the **attachment backend**, and apps upload and fetch them through the SDK or the [HTTP API](/cloud/http-api/attachments). By default the backend is **in-memory**, which is ephemeral and intended only for local development — attachments are lost when a pod restarts. This guide sets up a durable backend: an Amazon S3 bucket, an Azure Blob Storage container, or another S3-compatible store such as MinIO or Google Cloud Storage.

<Warning>
  The default in-memory attachment backend is **not durable**. For any non-throwaway deployment, configure an external backend (Amazon S3 or Azure Blob Storage) as described below.
</Warning>

## Overview

This guide assumes you have already [deployed a Big Peer](./operator-quickstart#deploying-a-big-peer) and that it runs on Amazon EKS (for S3) or Azure AKS (for Azure Blob Storage). The steps below configure attachments on that existing Big Peer.

Attachment setup has two halves:

1. **You provision the cloud storage and identity** — the bucket or container, and the cloud identity your Big Peer will use to access it. The Operator does *not* create these.
2. **The Operator wires that identity into the Big Peer workloads** — based on `spec.attachments.backend`, it sets the right service account annotations, pod labels, and environment variables.

### Attachment Service Accounts

A single Big Peer runs three workloads that access attachments, each under its own service account. For a Big Peer named `example` in namespace `ditto`, they are:

* `ditto-example-api`
* `ditto-example-subscription`
* `ditto-example-replication`

The Operator applies the attachment identity to all three. Whatever cloud-side trust you configure (an AWS IAM role trust policy, or Azure federated credentials) must cover all three service accounts — this is the most common thing to get wrong. (The [S3-compatible](#s3-compatible-storage) path is the exception: it authenticates with a `Secret`, not a service account identity, so this doesn't apply.)

### Object Key Layout

Each attachment is stored as an object in your S3 bucket or Azure container. An object's **key** is its full path within that bucket or container. So that several Big Peers can share one bucket or container without colliding, the Operator gives every Big Peer its own key **prefix** — a folder-like path that all of its attachment objects' keys begin with.

The prefix is `<root>/<big-peer-name>/<namespace>`, and `<root>` defaults to `big-peer/attachments`. So the `example` Big Peer in the `ditto` namespace stores its attachment objects under keys that start with:

```
big-peer/attachments/example/ditto/
```

To use a different root, set the optional `prefix` field under `spec.attachments.backend.s3` (or `.azure`). For example, `prefix: my-team/blobs` stores objects under `my-team/blobs/example/ditto/`. The Operator always appends `/<big-peer-name>/<namespace>`, so each Big Peer stays isolated.

## Amazon S3 (EKS)

This path uses [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html): the Big Peer's service accounts assume an IAM role via your cluster's OIDC provider — no static credentials are stored in the cluster.

<Steps>
  <Step title="Confirm prerequisites">
    You need an EKS cluster with an **IAM OIDC provider associated**, the `aws` CLI configured for the target account, and your Big Peer's name (`example`) and namespace (`ditto`).

    Check whether the OIDC provider is associated:

    ```bash theme={null}
    aws eks describe-cluster --name "$CLUSTER" \
      --query "cluster.identity.oidc.issuer" --output text
    ```

    If your cluster has no OIDC provider yet, associate one:

    ```bash theme={null}
    eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER" --approve
    ```

    Set shared variables for the commands below:

    ```bash theme={null}
    export CLUSTER=my-eks-cluster
    export REGION=us-west-2
    export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    export BUCKET=ditto-example-attachments
    export NAMESPACE=ditto
    export BIGPEER=example
    export OIDC=$(aws eks describe-cluster --name "$CLUSTER" \
      --query "cluster.identity.oidc.issuer" --output text | sed 's#https://##')
    ```
  </Step>

  <Step title="Create the S3 bucket">
    ```bash theme={null}
    aws s3api create-bucket \
      --bucket "$BUCKET" \
      --region "$REGION" \
      --create-bucket-configuration LocationConstraint="$REGION"
    ```

    <Note>
      In `us-east-1`, omit `--create-bucket-configuration` (that region rejects a `LocationConstraint`). We recommend enabling default encryption and blocking public access on the bucket.
    </Note>
  </Step>

  <Step title="Create a least-privilege IAM policy">
    ```bash theme={null}
    cat > ditto-attachments-policy.json <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "ListBucket",
          "Effect": "Allow",
          "Action": ["s3:ListBucket"],
          "Resource": "arn:aws:s3:::$BUCKET"
        },
        {
          "Sid": "ReadWriteObjects",
          "Effect": "Allow",
          "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
          "Resource": "arn:aws:s3:::$BUCKET/*"
        }
      ]
    }
    EOF

    aws iam create-policy \
      --policy-name ditto-example-attachments \
      --policy-document file://ditto-attachments-policy.json
    ```
  </Step>

  <Step title="Create the IAM role with an IRSA trust policy">
    The trust policy allows all three Big Peer service accounts to assume the role. The `StringLike` wildcard `ditto-example-*` covers `-api`, `-subscription`, and `-replication` in one statement.

    ```bash theme={null}
    cat > ditto-attachments-trust.json <<EOF
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/$OIDC"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": { "$OIDC:aud": "sts.amazonaws.com" },
            "StringLike": {
              "$OIDC:sub": "system:serviceaccount:$NAMESPACE:ditto-$BIGPEER-*"
            }
          }
        }
      ]
    }
    EOF

    aws iam create-role \
      --role-name ditto-example-attachments \
      --assume-role-policy-document file://ditto-attachments-trust.json

    aws iam attach-role-policy \
      --role-name ditto-example-attachments \
      --policy-arn "arn:aws:iam::$ACCOUNT_ID:policy/ditto-example-attachments"
    ```

    <Note>
      Do **not** use `eksctl create iamserviceaccount` to create the service account — the Ditto Operator owns and manages those service accounts. Create the role directly (as above), or use `eksctl create iamserviceaccount --role-only`. The Operator applies the `eks.amazonaws.com/role-arn` annotation for you from the `iamRoleArn` you set on the Big Peer.
    </Note>
  </Step>

  <Step title="Configure the Big Peer">
    Add the S3 backend to your **existing** Big Peer with a merge patch. This leaves your `network`, `auth`, `version`, and other settings untouched. Replace the account ID in `iamRoleArn` with your own (the `$ACCOUNT_ID` from the first step):

    ```bash theme={null}
    kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
    spec:
      attachments:
        backend:
          s3:
            region: us-west-2
            bucket: ditto-example-attachments
            iamRoleArn: arn:aws:iam::<your-account-id>:role/ditto-example-attachments
    EOF
    ```

    <Note>
      If you manage your Big Peer from a YAML file instead, add the same `attachments` block to that file (alongside `network`, `auth`, and `transactions`) and re-apply it. Don't `kubectl apply` a spec that contains only `attachments` — that would remove your other settings.
    </Note>
  </Step>
</Steps>

Continue to [Verify Attachment Storage](#verify-attachment-storage).

## Azure Blob Storage (AKS)

The recommended path uses [Azure Workload Identity](https://learn.microsoft.com/azure/aks/workload-identity-overview): the Big Peer's service accounts federate with a user-assigned managed identity, so no storage credential is stored in the cluster. An account-key alternative is documented at the end of this section.

<Note>
  Azure Blob Storage support requires a Big Peer release that includes it. If your Big Peer predates it, raise `spec.version` to a supported release — check the [Big Peer release notes](/cloud/release-notes).
</Note>

<Steps>
  <Step title="Confirm prerequisites">
    You need an AKS cluster with the **OIDC issuer** and **Workload Identity** enabled, the `az` CLI logged in to the target subscription, and your Big Peer's name (`example`) and namespace (`ditto`).

    Enable the OIDC issuer and Workload Identity if they aren't already:

    ```bash theme={null}
    az aks update -g "$RG" -n "$CLUSTER" \
      --enable-oidc-issuer --enable-workload-identity
    ```

    Set shared variables:

    ```bash theme={null}
    export RG=my-resource-group
    export CLUSTER=my-aks-cluster
    export LOCATION=eastus
    export ACCOUNT=dittoexampleattach      # 3–24 lowercase letters/digits
    export CONTAINER=attachments           # 3–63 lowercase letters/digits/hyphens
    export IDENTITY=ditto-example-attachments
    export NAMESPACE=ditto
    export BIGPEER=example
    export OIDC=$(az aks show -g "$RG" -n "$CLUSTER" \
      --query oidcIssuerProfile.issuerUrl -o tsv)
    ```
  </Step>

  <Step title="Create the storage account and container">
    ```bash theme={null}
    az storage account create -g "$RG" -n "$ACCOUNT" -l "$LOCATION" --sku Standard_LRS
    az storage container create --account-name "$ACCOUNT" -n "$CONTAINER" --auth-mode login
    ```

    <Note>
      The Big Peer CRD validates these names: `account` must match `^[a-z0-9]{3,24}$` and `container` must match `^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$`.
    </Note>
  </Step>

  <Step title="Create a user-assigned managed identity">
    ```bash theme={null}
    az identity create -g "$RG" -n "$IDENTITY"

    export CLIENT_ID=$(az identity show -g "$RG" -n "$IDENTITY" --query clientId -o tsv)
    export PRINCIPAL_ID=$(az identity show -g "$RG" -n "$IDENTITY" --query principalId -o tsv)
    ```
  </Step>

  <Step title="Grant the identity data-plane access to the container">
    Assign the data-plane role **Storage Blob Data Contributor** (not account keys):

    ```bash theme={null}
    export SCOPE=$(az storage account show -g "$RG" -n "$ACCOUNT" --query id -o tsv)/blobServices/default/containers/$CONTAINER

    az role assignment create \
      --assignee-object-id "$PRINCIPAL_ID" \
      --assignee-principal-type ServicePrincipal \
      --role "Storage Blob Data Contributor" \
      --scope "$SCOPE"
    ```

    <Note>
      Role assignments can take a minute or two to propagate. Scope to the account instead of the container if you prefer broader access.
    </Note>
  </Step>

  <Step title="Create federated identity credentials — one per service account">
    Azure federated credentials do **not** support wildcards, so create one for each of the three Big Peer service accounts on the same identity:

    ```bash theme={null}
    for SA in api subscription replication; do
      az identity federated-credential create \
        --name "ditto-$BIGPEER-$SA" \
        --identity-name "$IDENTITY" \
        -g "$RG" \
        --issuer "$OIDC" \
        --subject "system:serviceaccount:$NAMESPACE:ditto-$BIGPEER-$SA" \
        --audiences api://AzureADTokenExchange
    done
    ```
  </Step>

  <Step title="Configure the Big Peer">
    Add the Azure backend to your **existing** Big Peer with a merge patch. This leaves your `network`, `auth`, `version`, and other settings untouched. Set `clientId` to the identity's client ID (the `$CLIENT_ID` from the managed-identity step):

    ```bash theme={null}
    kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
    spec:
      attachments:
        backend:
          azure:
            account: dittoexampleattach
            container: attachments
            auth:
              managedIdentity:
                clientId: <your-client-id>
    EOF
    ```

    <Note>
      If you manage your Big Peer from a YAML file instead, add the same `attachments` block to that file (alongside `network`, `auth`, and `transactions`) and re-apply it. Don't `kubectl apply` a spec that contains only `attachments` — that would remove your other settings.
    </Note>
  </Step>
</Steps>

<Accordion title="Alternative: authenticate with a storage-account access key">
  Workload Identity is the recommended path. Where it isn't available — for example a cluster without Workload Identity enabled, or a restricted / air-gapped environment — you can authenticate with a storage-account access key stored in a Kubernetes `Secret`.

  <Warning>
    An access key is a **long-lived, account-wide credential with no automatic rotation**. Restrict its scope, store it in a securely-managed `Secret`, and rotate it regularly. Prefer Workload Identity where you can.
  </Warning>

  First, read an account key and create the `Secret` in the Big Peer's namespace:

  ```bash theme={null}
  export KEY=$(az storage account keys list -g "$RG" -n "$ACCOUNT" \
    --query "[0].value" -o tsv)

  kubectl create secret generic azure-attach-creds \
    -n ditto --from-literal=AZURE_STORAGE_ACCESS_KEY="$KEY"
  ```

  Then reference the `Secret` from the Big Peer with a merge patch. The `accessKey` property defaults to `AZURE_STORAGE_ACCESS_KEY`:

  ```bash theme={null}
  kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
  spec:
    attachments:
      backend:
        azure:
          account: dittoexampleattach
          container: attachments
          auth:
            credentialsSecret:
              name: azure-attach-creds
  EOF
  ```
</Accordion>

Continue to [Verify Attachment Storage](#verify-attachment-storage).

## S3-Compatible Storage

Use this backend for any object store that speaks the S3 API but isn't AWS S3 accessed through IRSA — for example **MinIO**, **Google Cloud Storage** (through its S3-compatible endpoint), or AWS S3 itself with a static access key. It authenticates with an access key and secret held in a Kubernetes `Secret`, so — unlike the [Amazon S3 (EKS)](#amazon-s3-eks) and [Azure Blob Storage (AKS)](#azure-blob-storage-aks) paths — it **works on any Kubernetes cluster**, needs no EKS/AKS identity setup, and does not use the [three-service-account trust](#attachment-service-accounts).

<Warning>
  An access key and secret are long-lived credentials. Scope them to the bucket, keep them in a securely-managed `Secret`, and rotate them regularly. Where a workload-identity path is available — S3 on EKS or Azure on AKS — prefer it.
</Warning>

<Steps>
  <Step title="Create the bucket and get an access key">
    The bucket and credentials are specific to your provider:

    <Tabs>
      <Tab title="MinIO">
        Create a bucket (for example `mc mb myminio/ditto-example-attachments`) and an access key / secret key pair with read/write access. Set `endpoint` to your MinIO URL — for example an in-cluster `http://minio.ditto.svc.cluster.local:9000`.

        MinIO supports both S3 addressing styles; choose based on your DNS:

        * **Virtual-hosted** (`<bucket>.<endpoint>`) — the Big Peer's default, and works today. The bucket subdomain must resolve, so configure MinIO for virtual-host addressing (its `MINIO_DOMAIN` setting) with matching wildcard DNS. AWS S3 and Google Cloud Storage provide this automatically.
        * **Path-style** (`<endpoint>/<bucket>`) — coming in a future Ditto Operator release. When the `forcePathStyle` field ships, set `forcePathStyle: true` to drop the bucket-subdomain DNS requirement (the simplest option for a MinIO reached by an in-cluster service name or a plain hostname). Until then, use virtual-hosted addressing (above).
      </Tab>

      <Tab title="Google Cloud Storage">
        Create a bucket, then generate **HMAC keys** for a service account that can access it (Cloud Console → Cloud Storage → Settings → Interoperability, or `gcloud storage hmac create`). Use the endpoint `https://storage.googleapis.com`. See [Cloud Storage interoperability](https://cloud.google.com/storage/docs/interoperability).
      </Tab>

      <Tab title="Amazon S3">
        Create a bucket and an IAM user with an access key that has read/write access to it. Use the regional endpoint, e.g. `https://s3.us-west-2.amazonaws.com`. On EKS, prefer the credential-free [Amazon S3 (EKS)](#amazon-s3-eks) path instead.
      </Tab>
    </Tabs>
  </Step>

  <Step title="Store the credentials in a Secret">
    Create a `Secret` in the Big Peer's namespace holding the access key ID and secret:

    ```bash theme={null}
    kubectl create secret generic ditto-example-s3-creds -n ditto \
      --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
      --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
    ```
  </Step>

  <Step title="Configure the Big Peer">
    Add the `s3Compatible` backend to your existing Big Peer, using the endpoint, bucket, and `Secret` from the previous steps:

    ```bash theme={null}
    kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
    spec:
      attachments:
        backend:
          s3Compatible:
            endpoint: https://storage.googleapis.com
            region: auto
            bucket: ditto-example-attachments
            credentialsSecretRef:
              name: ditto-example-s3-creds
    EOF
    ```

    <Note>
      `region` is optional — use `auto` for Google Cloud Storage, or the bucket's region (for example `us-west-2`) for AWS S3 or MinIO deployments that require one. `credentialsSecretRef` reads the keys `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` by default; set `accessKeyId` / `secretAccessKey` if your `Secret` uses different keys.
    </Note>
  </Step>
</Steps>

Continue to [Verify Attachment Storage](#verify-attachment-storage).

## Verify Attachment Storage

<Note>
  The first two checks apply to the identity-based paths — S3 on EKS (IRSA) and Azure on AKS (Workload Identity). The [S3-compatible](#s3-compatible-storage) path authenticates with a `Secret` instead, so it has no service account annotation or injected identity token; for that path, skip to the upload-and-fetch check.
</Note>

<Steps>
  <Step title="Confirm the service account identity annotation">
    ```bash theme={null}
    kubectl get sa ditto-example-api -n ditto -o yaml
    ```

    Expected under `metadata.annotations`:

    * **AWS:** `eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/ditto-example-attachments`
    * **Azure (Workload Identity):** `azure.workload.identity/client-id: <clientId>`

    Repeat for `ditto-example-subscription` and `ditto-example-replication`.
  </Step>

  <Step title="Confirm credentials are wired into the pods">
    On **AWS**, the pods carry `AWS_ROLE_ARN` and `AWS_WEB_IDENTITY_TOKEN_FILE`:

    ```bash theme={null}
    kubectl get pod ditto-example-api-0 -n ditto -o yaml | grep -E "AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE"
    ```

    On **Azure**, the pods carry the `azure.workload.identity/use: "true"` label, and the AKS webhook injects the `AZURE_*` environment:

    ```bash theme={null}
    kubectl get pod ditto-example-api-0 -n ditto -o jsonpath='{.metadata.labels.azure\.workload\.identity/use}{"\n"}'
    kubectl get pod ditto-example-api-0 -n ditto -o yaml | grep -E "AZURE_CLIENT_ID|AZURE_FEDERATED_TOKEN_FILE"
    ```
  </Step>

  <Step title="Upload and fetch an attachment">
    Use an SDK `newAttachment` / `fetchAttachment` round-trip, or the [HTTP API attachment endpoints](/cloud/http-api/attachments), and confirm the blob round-trips. You can also confirm objects appear under your bucket or container (these commands reuse the `$BUCKET` / `$ACCOUNT` / `$CONTAINER` variables you exported earlier — set them again if you're in a new shell):

    ```bash theme={null}
    # AWS
    aws s3 ls "s3://$BUCKET/big-peer/attachments/example/ditto/" --recursive

    # Azure
    az storage blob list --account-name "$ACCOUNT" -c "$CONTAINER" \
      --prefix "big-peer/attachments/example/ditto/" --auth-mode login -o table
    ```
  </Step>
</Steps>

## Troubleshooting

| Symptom                                                                                | Likely cause and fix                                                                                                                                                                                                                                                |
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Service account has no identity annotation                                             | The `backend`/`auth` block is missing or malformed, or the Big Peer name/namespace differs from what you configured. Re-check `spec.attachments.backend`.                                                                                                           |
| `AccessDenied` on upload                                                               | AWS: the IAM policy or role scope doesn't cover the bucket/prefix. Azure: the **Storage Blob Data Contributor** assignment is missing or still propagating (wait a minute and retry).                                                                               |
| Credentials not injected (Azure)                                                       | The pod is missing `azure.workload.identity/use`, or Workload Identity / the OIDC issuer isn't enabled on the cluster. Re-run the `az aks update` from the prerequisites.                                                                                           |
| IRSA role assumption fails (AWS)                                                       | The pod's `sts:AssumeRoleWithWebIdentity` call is rejected — the IRSA trust policy `sub`/`aud` don't match, or the OIDC provider isn't associated with the cluster.                                                                                                 |
| Subject mismatch                                                                       | The trust-policy `sub` (AWS) or federated-credential `--subject` (Azure) must be exactly `system:serviceaccount:<namespace>:ditto-<big-peer-name>-{api,subscription,replication}`.                                                                                  |
| Wrong region/account                                                                   | AWS: `region` doesn't match the bucket. Azure: `account`/`container` don't match what you created.                                                                                                                                                                  |
| S3-compatible pods crash-loop at startup (`Error while dispatching request: io error`) | The bucket subdomain `<bucket>.<endpoint>` isn't resolving. Enable virtual-host addressing on the store (MinIO: `MINIO_DOMAIN` + matching wildcard DNS), or set `forcePathStyle: true` for path-style (`<endpoint>/<bucket>`).                                      |
| `403` / access denied on an S3-compatible endpoint                                     | Check the `endpoint` scheme, that the access key and secret are valid, and that the `credentialsSecretRef` `Secret` exists in the Big Peer's namespace with keys `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (or set `accessKeyId` / `secretAccessKey` to match). |