Attachments

The Amazon S3 and S3-compatible attachment backends are available from Operator version 0.1.4 onwards. The Azure Blob Storage backend is available from Operator version 0.17.1 onwards.

Attachments let your apps link large binary data — images, video, PDFs — to documents. Big Peer keeps those blobs in an object store called the attachment backend, and apps upload and fetch them through the SDK or the HTTP API. By default the backend is in-memory, which is ephemeral and intended only for local development — attachments are lost when a pod restarts. This guide sets up a durable backend: an Amazon S3 bucket, an Azure Blob Storage container, or another S3-compatible store such as MinIO or Google Cloud Storage.

The default in-memory attachment backend is not durable. For any non-throwaway deployment, configure an external backend (Amazon S3 or Azure Blob Storage) as described below.

Overview

This guide assumes you have already deployed a Big Peer and that it runs on Amazon EKS (for S3) or Azure AKS (for Azure Blob Storage). The steps below configure attachments on that existing Big Peer. Attachment setup has two halves:

You provision the cloud storage and identity — the bucket or container, and the cloud identity your Big Peer will use to access it. The Operator does not create these.
The Operator wires that identity into the Big Peer workloads — based on spec.attachments.backend, it sets the right service account annotations, pod labels, and environment variables.

Attachment Service Accounts

A single Big Peer runs three workloads that access attachments, each under its own service account. For a Big Peer named example in namespace ditto, they are:

ditto-example-api
ditto-example-subscription
ditto-example-replication

The Operator applies the attachment identity to all three. Whatever cloud-side trust you configure (an AWS IAM role trust policy, or Azure federated credentials) must cover all three service accounts — this is the most common thing to get wrong. (The S3-compatible path is the exception: it authenticates with a Secret, not a service account identity, so this doesn’t apply.)

Object Key Layout

Each attachment is stored as an object in your S3 bucket or Azure container. An object’s key is its full path within that bucket or container. So that several Big Peers can share one bucket or container without colliding, the Operator gives every Big Peer its own key prefix — a folder-like path that all of its attachment objects’ keys begin with. The prefix is <root>/<big-peer-name>/<namespace>, and <root> defaults to big-peer/attachments. So the example Big Peer in the ditto namespace stores its attachment objects under keys that start with:

big-peer/attachments/example/ditto/

To use a different root, set the optional prefix field under spec.attachments.backend.s3 (or .azure). For example, prefix: my-team/blobs stores objects under my-team/blobs/example/ditto/. The Operator always appends /<big-peer-name>/<namespace>, so each Big Peer stays isolated.

Amazon S3 (EKS)

This path uses IAM Roles for Service Accounts (IRSA): the Big Peer’s service accounts assume an IAM role via your cluster’s OIDC provider — no static credentials are stored in the cluster.

Confirm prerequisites

You need an EKS cluster with an IAM OIDC provider associated, the aws CLI configured for the target account, and your Big Peer’s name (example) and namespace (ditto).Check whether the OIDC provider is associated:

aws eks describe-cluster --name "$CLUSTER" \
  --query "cluster.identity.oidc.issuer" --output text

If your cluster has no OIDC provider yet, associate one:

eksctl utils associate-iam-oidc-provider --cluster "$CLUSTER" --approve

Set shared variables for the commands below:

export CLUSTER=my-eks-cluster
export REGION=us-west-2
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export BUCKET=ditto-example-attachments
export NAMESPACE=ditto
export BIGPEER=example
export OIDC=$(aws eks describe-cluster --name "$CLUSTER" \
  --query "cluster.identity.oidc.issuer" --output text | sed 's#https://##')

Create the S3 bucket

aws s3api create-bucket \
  --bucket "$BUCKET" \
  --region "$REGION" \
  --create-bucket-configuration LocationConstraint="$REGION"

In us-east-1, omit --create-bucket-configuration (that region rejects a LocationConstraint). We recommend enabling default encryption and blocking public access on the bucket.

Create a least-privilege IAM policy

cat > ditto-attachments-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::$BUCKET"
    },
    {
      "Sid": "ReadWriteObjects",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::$BUCKET/*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name ditto-example-attachments \
  --policy-document file://ditto-attachments-policy.json

Create the IAM role with an IRSA trust policy

The trust policy allows all three Big Peer service accounts to assume the role. The StringLike wildcard ditto-example-* covers -api, -subscription, and -replication in one statement.

cat > ditto-attachments-trust.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/$OIDC"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": { "$OIDC:aud": "sts.amazonaws.com" },
        "StringLike": {
          "$OIDC:sub": "system:serviceaccount:$NAMESPACE:ditto-$BIGPEER-*"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name ditto-example-attachments \
  --assume-role-policy-document file://ditto-attachments-trust.json

aws iam attach-role-policy \
  --role-name ditto-example-attachments \
  --policy-arn "arn:aws:iam::$ACCOUNT_ID:policy/ditto-example-attachments"

Do not use eksctl create iamserviceaccount to create the service account — the Ditto Operator owns and manages those service accounts. Create the role directly (as above), or use eksctl create iamserviceaccount --role-only. The Operator applies the eks.amazonaws.com/role-arn annotation for you from the iamRoleArn you set on the Big Peer.

Configure the Big Peer

Add the S3 backend to your existing Big Peer with a merge patch. This leaves your network, auth, version, and other settings untouched. Replace the account ID in iamRoleArn with your own (the $ACCOUNT_ID from the first step):

kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
spec:
  attachments:
    backend:
      s3:
        region: us-west-2
        bucket: ditto-example-attachments
        iamRoleArn: arn:aws:iam::<your-account-id>:role/ditto-example-attachments
EOF

If you manage your Big Peer from a YAML file instead, add the same attachments block to that file (alongside network, auth, and transactions) and re-apply it. Don’t kubectl apply a spec that contains only attachments — that would remove your other settings.

Continue to Verify Attachment Storage.

Azure Blob Storage (AKS)

The recommended path uses Azure Workload Identity: the Big Peer’s service accounts federate with a user-assigned managed identity, so no storage credential is stored in the cluster. An account-key alternative is documented at the end of this section.

Azure Blob Storage support requires a Big Peer release that includes it. If your Big Peer predates it, raise spec.version to a supported release — check the Big Peer release notes.

Confirm prerequisites

You need an AKS cluster with the OIDC issuer and Workload Identity enabled, the az CLI logged in to the target subscription, and your Big Peer’s name (example) and namespace (ditto).Enable the OIDC issuer and Workload Identity if they aren’t already:

az aks update -g "$RG" -n "$CLUSTER" \
  --enable-oidc-issuer --enable-workload-identity

Set shared variables:

export RG=my-resource-group
export CLUSTER=my-aks-cluster
export LOCATION=eastus
export ACCOUNT=dittoexampleattach      # 3–24 lowercase letters/digits
export CONTAINER=attachments           # 3–63 lowercase letters/digits/hyphens
export IDENTITY=ditto-example-attachments
export NAMESPACE=ditto
export BIGPEER=example
export OIDC=$(az aks show -g "$RG" -n "$CLUSTER" \
  --query oidcIssuerProfile.issuerUrl -o tsv)

Create the storage account and container

az storage account create -g "$RG" -n "$ACCOUNT" -l "$LOCATION" --sku Standard_LRS
az storage container create --account-name "$ACCOUNT" -n "$CONTAINER" --auth-mode login

The Big Peer CRD validates these names: account must match ^[a-z0-9]{3,24}$ and container must match ^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$.

Create a user-assigned managed identity

az identity create -g "$RG" -n "$IDENTITY"

export CLIENT_ID=$(az identity show -g "$RG" -n "$IDENTITY" --query clientId -o tsv)
export PRINCIPAL_ID=$(az identity show -g "$RG" -n "$IDENTITY" --query principalId -o tsv)

Grant the identity data-plane access to the container

Assign the data-plane role Storage Blob Data Contributor (not account keys):

export SCOPE=$(az storage account show -g "$RG" -n "$ACCOUNT" --query id -o tsv)/blobServices/default/containers/$CONTAINER

az role assignment create \
  --assignee-object-id "$PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope "$SCOPE"

Role assignments can take a minute or two to propagate. Scope to the account instead of the container if you prefer broader access.

Create federated identity credentials — one per service account

Azure federated credentials do not support wildcards, so create one for each of the three Big Peer service accounts on the same identity:

for SA in api subscription replication; do
  az identity federated-credential create \
    --name "ditto-$BIGPEER-$SA" \
    --identity-name "$IDENTITY" \
    -g "$RG" \
    --issuer "$OIDC" \
    --subject "system:serviceaccount:$NAMESPACE:ditto-$BIGPEER-$SA" \
    --audiences api://AzureADTokenExchange
done

Configure the Big Peer

Add the Azure backend to your existing Big Peer with a merge patch. This leaves your network, auth, version, and other settings untouched. Set clientId to the identity’s client ID (the $CLIENT_ID from the managed-identity step):

kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
spec:
  attachments:
    backend:
      azure:
        account: dittoexampleattach
        container: attachments
        auth:
          managedIdentity:
            clientId: <your-client-id>
EOF

Alternative: authenticate with a storage-account access key

Workload Identity is the recommended path. Where it isn’t available — for example a cluster without Workload Identity enabled, or a restricted / air-gapped environment — you can authenticate with a storage-account access key stored in a Kubernetes Secret.

An access key is a long-lived, account-wide credential with no automatic rotation. Restrict its scope, store it in a securely-managed Secret, and rotate it regularly. Prefer Workload Identity where you can.

First, read an account key and create the Secret in the Big Peer’s namespace:

export KEY=$(az storage account keys list -g "$RG" -n "$ACCOUNT" \
  --query "[0].value" -o tsv)

kubectl create secret generic azure-attach-creds \
  -n ditto --from-literal=AZURE_STORAGE_ACCESS_KEY="$KEY"

Then reference the Secret from the Big Peer with a merge patch. The accessKey property defaults to AZURE_STORAGE_ACCESS_KEY:

kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
spec:
  attachments:
    backend:
      azure:
        account: dittoexampleattach
        container: attachments
        auth:
          credentialsSecret:
            name: azure-attach-creds
EOF

Continue to Verify Attachment Storage.

S3-Compatible Storage

Use this backend for any object store that speaks the S3 API but isn’t AWS S3 accessed through IRSA — for example MinIO, Google Cloud Storage (through its S3-compatible endpoint), or AWS S3 itself with a static access key. It authenticates with an access key and secret held in a Kubernetes Secret, so — unlike the Amazon S3 (EKS) and Azure Blob Storage (AKS) paths — it works on any Kubernetes cluster, needs no EKS/AKS identity setup, and does not use the three-service-account trust.

An access key and secret are long-lived credentials. Scope them to the bucket, keep them in a securely-managed Secret, and rotate them regularly. Where a workload-identity path is available — S3 on EKS or Azure on AKS — prefer it.

Create the bucket and get an access key

The bucket and credentials are specific to your provider:

MinIO
Google Cloud Storage
Amazon S3

Create a bucket (for example mc mb myminio/ditto-example-attachments) and an access key / secret key pair with read/write access. Set endpoint to your MinIO URL — for example an in-cluster http://minio.ditto.svc.cluster.local:9000.MinIO supports both S3 addressing styles; choose based on your DNS:

Virtual-hosted (<bucket>.<endpoint>) — the Big Peer’s default, and works today. The bucket subdomain must resolve, so configure MinIO for virtual-host addressing (its MINIO_DOMAIN setting) with matching wildcard DNS. AWS S3 and Google Cloud Storage provide this automatically.
Path-style (<endpoint>/<bucket>) — coming in a future Ditto Operator release. When the forcePathStyle field ships, set forcePathStyle: true to drop the bucket-subdomain DNS requirement (the simplest option for a MinIO reached by an in-cluster service name or a plain hostname). Until then, use virtual-hosted addressing (above).

Create a bucket, then generate HMAC keys for a service account that can access it (Cloud Console → Cloud Storage → Settings → Interoperability, or gcloud storage hmac create). Use the endpoint https://storage.googleapis.com. See Cloud Storage interoperability.

Create a bucket and an IAM user with an access key that has read/write access to it. Use the regional endpoint, e.g. https://s3.us-west-2.amazonaws.com. On EKS, prefer the credential-free Amazon S3 (EKS) path instead.

Store the credentials in a Secret

Create a Secret in the Big Peer’s namespace holding the access key ID and secret:

kubectl create secret generic ditto-example-s3-creds -n ditto \
  --from-literal=AWS_ACCESS_KEY_ID=<your-access-key-id> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<your-secret-access-key>

Configure the Big Peer

Add the s3Compatible backend to your existing Big Peer, using the endpoint, bucket, and Secret from the previous steps:

kubectl patch bigpeer example -n ditto --type merge --patch-file /dev/stdin <<'EOF'
spec:
  attachments:
    backend:
      s3Compatible:
        endpoint: https://storage.googleapis.com
        region: auto
        bucket: ditto-example-attachments
        credentialsSecretRef:
          name: ditto-example-s3-creds
EOF

region is optional — use auto for Google Cloud Storage, or the bucket’s region (for example us-west-2) for AWS S3 or MinIO deployments that require one. credentialsSecretRef reads the keys AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY by default; set accessKeyId / secretAccessKey if your Secret uses different keys.

Continue to Verify Attachment Storage.

Verify Attachment Storage

The first two checks apply to the identity-based paths — S3 on EKS (IRSA) and Azure on AKS (Workload Identity). The S3-compatible path authenticates with a Secret instead, so it has no service account annotation or injected identity token; for that path, skip to the upload-and-fetch check.

Confirm the service account identity annotation

kubectl get sa ditto-example-api -n ditto -o yaml

Expected under metadata.annotations:

AWS: eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/ditto-example-attachments
Azure (Workload Identity): azure.workload.identity/client-id: <clientId>

Repeat for ditto-example-subscription and ditto-example-replication.

Confirm credentials are wired into the pods

On AWS, the pods carry AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE:

kubectl get pod ditto-example-api-0 -n ditto -o yaml | grep -E "AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE"

On Azure, the pods carry the azure.workload.identity/use: "true" label, and the AKS webhook injects the AZURE_* environment:

kubectl get pod ditto-example-api-0 -n ditto -o jsonpath='{.metadata.labels.azure\.workload\.identity/use}{"\n"}'
kubectl get pod ditto-example-api-0 -n ditto -o yaml | grep -E "AZURE_CLIENT_ID|AZURE_FEDERATED_TOKEN_FILE"

Upload and fetch an attachment

Use an SDK newAttachment / fetchAttachment round-trip, or the HTTP API attachment endpoints, and confirm the blob round-trips. You can also confirm objects appear under your bucket or container (these commands reuse the $BUCKET / $ACCOUNT / $CONTAINER variables you exported earlier — set them again if you’re in a new shell):

# AWS
aws s3 ls "s3://$BUCKET/big-peer/attachments/example/ditto/" --recursive

# Azure
az storage blob list --account-name "$ACCOUNT" -c "$CONTAINER" \
  --prefix "big-peer/attachments/example/ditto/" --auth-mode login -o table

Troubleshooting

Symptom	Likely cause and fix
Service account has no identity annotation	The `backend`/`auth` block is missing or malformed, or the Big Peer name/namespace differs from what you configured. Re-check `spec.attachments.backend`.
`AccessDenied` on upload	AWS: the IAM policy or role scope doesn’t cover the bucket/prefix. Azure: the Storage Blob Data Contributor assignment is missing or still propagating (wait a minute and retry).
Credentials not injected (Azure)	The pod is missing `azure.workload.identity/use`, or Workload Identity / the OIDC issuer isn’t enabled on the cluster. Re-run the `az aks update` from the prerequisites.
IRSA role assumption fails (AWS)	The pod’s `sts:AssumeRoleWithWebIdentity` call is rejected — the IRSA trust policy `sub`/`aud` don’t match, or the OIDC provider isn’t associated with the cluster.
Subject mismatch	The trust-policy `sub` (AWS) or federated-credential `--subject` (Azure) must be exactly `system:serviceaccount:<namespace>:ditto-<big-peer-name>-{api,subscription,replication}`.
Wrong region/account	AWS: `region` doesn’t match the bucket. Azure: `account`/`container` don’t match what you created.
S3-compatible pods crash-loop at startup (`Error while dispatching request: io error`)	The bucket subdomain `<bucket>.<endpoint>` isn’t resolving. Enable virtual-host addressing on the store (MinIO: `MINIO_DOMAIN` + matching wildcard DNS), or set `forcePathStyle: true` for path-style (`<endpoint>/<bucket>`).
`403` / access denied on an S3-compatible endpoint	Check the `endpoint` scheme, that the access key and secret are valid, and that the `credentialsSecretRef` `Secret` exists in the Big Peer’s namespace with keys `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` (or set `accessKeyId` / `secretAccessKey` to match).

Ditto Operator

Connectors

Ditto Operator API

Reference

Overview

Attachment Service Accounts

Object Key Layout

Amazon S3 (EKS)

Azure Blob Storage (AKS)

S3-Compatible Storage

Verify Attachment Storage

Troubleshooting

​Overview

​Attachment Service Accounts

​Object Key Layout

​Amazon S3 (EKS)

​Azure Blob Storage (AKS)

​S3-Compatible Storage

​Verify Attachment Storage

​Troubleshooting

Overview

Attachment Service Accounts

Object Key Layout

Amazon S3 (EKS)

Azure Blob Storage (AKS)

S3-Compatible Storage

Verify Attachment Storage

Troubleshooting