Introduction

The Ditto Operator allows you to deploy Big Peer (aka Ditto Server), Ditto’s cloud sync engine and data integration platform, to your own self-hosted Kubernetes environment.

Ditto Operator simplifies the configuration of the various components that form Big Peer, allowing you to focus on developing applications that get you connected to the cloud.

Prerequisites

A few prequisites need to be running before you install the Operator and deploy Big Peer.

Kubernetes Environment

The following steps assume you have a Kubernetes cluster, on at least version 1.31.

If you’re looking to try the Operator locally, we recommend using kind, a tool for running Kubernetes locally using Docker.

Example Kind Config

Here’s an example configuration you can use, which will expose the Big Peer Ingress on your local device, allowing you to follow the steps later in this guide:

cat <<'EOF' | kind create cluster --config -
---
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    metadata:
      name: config
    etcd:
      local:
        extraArgs:
          unsafe-no-fsync: "true"
    networking:
      serviceSubnet: 10.0.0.0/16
nodes:
  - role: control-plane
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP
      - containerPort: 4040
        hostPort: 4040
        protocol: TCP
EOF

Tools

You’ll also need to have kubectl and helm installed, with the appropriate KUBECONFIG current-context set.

Installing Certificate Manager

The Ditto Operator uses cert-manager to issue various auth-related certificates. You can install cert-manager into the target cluster using:

helm install cert-manager cert-manager    \
    --repo https://charts.jetstack.io     \
    --namespace cert-manager              \
    --create-namespace                    \
    --set 'crds.enabled=true'             \
    --set 'startupapicheck.enabled=false' \
    --set 'prometheus.enabled=false'

Installing an Ingress

The Ditto Operator manages ingresses to route traffic into the various Big Peer services.

We recommend using an ingress-nginx, as specific nginx specific annotations are used for optimal hashing. This guide will assume that nginx is being used.

The Ditto Operator can support other ingress controllers, but these may require different annotations. Please get in touch with Ditto support for guidance.

To install nginx:

If you opted to use Kind config provided to run the cluster locally, run the following:

kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml

If you opted to use Kind config provided to run the cluster locally, run the following:

kubectl apply -f https://kind.sigs.k8s.io/examples/ingress/deploy-ingress-nginx.yaml

If you are not installing the ingress-nginx in a kind cluster, you can run:

helm install ingress-nginx ingress-nginx              \
    --repo https://kubernetes.github.io/ingress-nginx \
    --namespace nginx                                 \
    --create-namespace

Installing Strimzi

The Ditto Operator uses strimzi to create a Kafka cluster and topic for the transaction log. You must install strimzi, a Kakfa Operator, into the target cluster using:

helm install strimzi strimzi-kafka-operator \
    --repo https://strimzi.io/charts        \
    --namespace kafka                       \
    --create-namespace                      \
    --set watchAnyNamespace=true

For convenience, this instructs Strimzi to watch all namespaces for Kafka resources.

If you wish to constrain this behaviour (for example you deploy Kafka for your other applications in another namespace) you can use --set watchNamespaces to specify the namespace you’re using for Ditto. Eg: --set watchNamespaces=ditto,another-ditto-namespace

Installing the Ditto Operator

With all prerequisites taken care of, the latest stable version of the Ditto Operator can be installed using the helm chart and image located at oci://quay.io/ditto-external/ditto-operator:

helm install ditto-operator                   \
  oci://quay.io/ditto-external/ditto-operator \
  --namespace ditto                           \
  --create-namespace

This installs the Operator into a newly created ditto namespace. All further steps will refer to this namespace.

Various configuration options are available in the chart’s values.yaml. You can view them with helm show values oci://quay.io/ditto-external/ditto-operator.

These values can be overriden using helm’s --set syntax. Further documentation for these values, and what they do, is coming soon.

You can now verify the Ditto Operator is installed and running with:

kubectl logs -f deployment/ditto-operator --namespace ditto

The various Ditto-related CRDs (Custom Resource Definitions) that were installed by the chart can be viewed using:

kubectl api-resources | grep ditto.live

Which will return output similar to:

NAME                       APIVERSION            NAMESPACED   KIND
bigpeers                   ditto.live/v1alpha1   true         BigPeer
bigpeerapps                ditto.live/v1alpha1   true         BigPeerApp
bigpeerstores              ditto.live/v1alpha1   true         BigPeerStore
bigpeersubscriptions       ditto.live/v1alpha1   true         BigPeerSubscription

Deploying a Big Peer

Here’s a minimal example to deploy Big Peer:

cat <<'EOF' | kubectl apply -f -
---
# Create the Big Peer deployment.
apiVersion: ditto.live/v1alpha1
kind: BigPeer
metadata:
  name: example
  namespace: ditto
spec:
  version: 1.43.0
  network:
    ingress:
      host: localhost
  auth:
    providers:
      onlinePlayground:
        anonymous:
          permission:
            read:
              everything: true
              queriesByCollection: {}
            write:
              everything: true
              queriesByCollection: {}
          sessionLength: 630000
          sharedToken: abc123
EOF

This will deploy a Big Peer which is:

Running on version 1.43.0
Defines an ingress on localhost
Enables Ditto SDK device auth via the onlinePlayground provider, with:
- Read and write access to all collections
- A shared token of abc123

OnlinePlayground is ideal for development and testing, as it allows you to quickly define a shared token that can be shared with devices for auth.

However, it should not be used for production. OnlineWithAuthentication is more appropriate for this.

If the BigPeer was applied successfully, the various components of the Big Peer can be observed by running:

kubectl get pods -l ditto.live/big-peer=example --namespace ditto

Which will match all running pods with the above BigPeer’s metadata.name, which is "example" in this case:

NAMESPACE  NAME                                                   READY  STATUS   RESTARTS  AGE
ditto      ditto-example-api-0                                    1/1    Running  0         1m20s
ditto      ditto-example-store-14d2fe44-0                         1/1    Running  0         1m19s
ditto      ditto-example-store-6a5d568b-0                         1/1    Running  0         1m19s
ditto      ditto-example-subscription-a475c814-7b459bfddb-brpr4   1/1    Running  0         1m20s*

Configuring Big Peer

Once Big Peer is deployed, you’ll need to configure it for first time usage.

The Operator API

The Ditto Operator provides a RESTful API for creating Apps and managing their configuration.

At this time, the Operator API does not have an authentication mechanism, and should not be exposed to the publicly. It is intended for use within the K8s cluster only.

To access the API, we recommend port-forwarding to ensure it remains secure and only accessible from your local machine.

For example:

kubectl port-forward deployments/ditto-operator 8080:8080

The following steps will assume you’ve portforwarded in this way.

Creating an App

In Ditto, an App is a logical container for your data. An App needs to be registered with Big Peer before you can start creating documents.

Each app has a unique ID that’s used to identify it across different Big Peers and devices. You’ll use this ID for all interactions with your App, including:

Connecting to the App from the Ditto SDK
Making HTTP API calls to the App
Setting up replication between Big Peers

Using the Operator API is the simplest way to create a new App.

Assuming you’ve port-forwarded to the Operator as described in the previous section, create an App using a POST request as follows:

curl -H 'Content-Type: application/json' \
     -X POST http://localhost:8080/namespace/ditto/bigPeer/example/app \
     -d '{"name":"example-app"}'

This will create an App called example-app, running on the example Big Peer, in the ditto namespace.

If successfil, you should receive a response like this:

{
  "name": "example-app",
  "appId": "2164bef3-37c0-489c-9ac6-c94b034525d7"
}

Take note of the appId from the response, you’ll need this later to use the HTTP API.

Using the Operator API is the simplest way to create a new App.

Assuming you’ve port-forwarded to the Operator as described in the previous section, create an App using a POST request as follows:

curl -H 'Content-Type: application/json' \
     -X POST http://localhost:8080/namespace/ditto/bigPeer/example/app \
     -d '{"name":"example-app"}'

This will create an App called example-app, running on the example Big Peer, in the ditto namespace.

If successfil, you should receive a response like this:

{
  "name": "example-app",
  "appId": "2164bef3-37c0-489c-9ac6-c94b034525d7"
}

Take note of the appId from the response, you’ll need this later to use the HTTP API.

Alternatively, you can create an app by declaring a BigPeerApp resource.

cat <<'EOF' | kubectl apply -f -
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerApp
metadata:
  name: example-app
  namespace: ditto
  labels:
    ditto.live/big-peer: example
spec:
  appId: 2164bef3-37c0-489c-9ac6-c94b034525d7
EOF

You’re responsible for choosing an App ID when creating an App in this way. It must be a valid UUID, and needs to be unique to each Big Peer.

This will create an App called example-app, running on the example Big Peer, in the ditto namespace.

You can verify that your app was created successfully by checking the status of the BigPeerApp resource:

kubectl get bigpeerapp example-app -n ditto

Using the Big Peer HTTP API

Using the HTTP API is the quickest way to verify your installation is running.

If you’ve deployed locally using the kind example config provided, these queries should work out of the box.

Otherwise, you may need to update the localhost according to the ingress you’ve configured on your K8s deployment.

Generate an API Key

To use the Big Peer HTTP API, you’ll need to generate an API key. This can only be done using the Operator API.

For this guide, we’ll create an API key that can has read and write access to all collections, with a very long expiry. For production scenarios, you may want to customise the scope of the API Keys access, and its expiry.

First, ensure you have port-forwarded to the Operator as described in the previous section.

Then, create an API key for your app using a POST request:

curl -H 'Content-Type: application/json' \
     -X POST http://localhost:8080/namespace/ditto/bigPeer/example/app/example-app/apiKey \
     -d '{
       "name": "my-key",
       "expiresAt": "2100-04-13T12:00:00Z",
       "permissions": {
         "remoteQuery": true,
         "read": {
           "everything": true,
           "queriesByCollection": {}
         },
         "write": {
           "everything": true,
           "queriesByCollection": {}
         }
       }
     }'

You’ll receive the raw API key as the response:

YlYr32ZAXotlf9zS1wh8JanJ2kRI2L38KnFHEh6nvaGiqKEqmAf8EsnZ9e0r

The raw API key is only shown once and is not stored anywhere. Make sure to save it securely, as you’ll need it for all HTTP API requests.

You can verify the API key was created by listing all keys for your app:

curl http://localhost:8080/namespace/ditto/bigPeer/example/app/example-app/apiKey

{
  "meta": {
    "page": 1,
    "size": 1,
    "totalPages": 1
  },
  "data": [
    {
      "name": "my-key",
      "expiresAt": "2100-04-13T12:00:00Z",
      "permissions": {
        "remoteQuery": true,
        "read": {
          "everything": true,
          "queriesByCollection": {}
        },
        "write": {
          "everything": true,
          "queriesByCollection": {}
        }
      },
      "description": null
    }
  ]
}

You can also verify this using kubectl as the Operator will make a BigPeerApiKey object to manage this key:

kubectl get bigpeerapikey my-key -n ditto -o yaml

kubectl get bigpeerapikey my-key -n ditto -o yaml
apiVersion: ditto.live/v1alpha1
kind: BigPeerApiKey
metadata:
  creationTimestamp: "2025-04-25T14:05:54Z"
  generation: 1
  labels:
    ditto.live/app: example-app
  name: my-key
  namespace: ditto
  resourceVersion: "10664"
  uid: 128351cf-4c44-4056-a96d-d0aa6a0b17b4
spec:
  expiresAt: "2100-04-13T12:00:00Z"
  hash: $scrypt$ln=15,r=8,p=1$DNpYWZdaH+yAgJbnIYu00Q$jshJ3PmHN6qjNwTd+3nlzrtKAcASBAGHazDFUM0agOE
  permissions:
    read:
      everything: true
      queriesByCollection: {}
    remoteQuery: true
    write:
      everything: true
      queriesByCollection: {}
  prefix: 9sNPfXzuBi
  version: 0

Insert an item into the `cars` collection

These examples assume you’ve deployed locally using the kind example config provided. If not, you may need to update localhost to the hostname you’ve configured as for your ingress.

Remember to replace YOUR_API_KEY with the API key generated earlier.

curl -X POST http://localhost/2164bef3-37c0-489c-9ac6-c94b034525d7/api/v4/store/execute \
    --header 'Authorization: bearer YOUR_API_KEY' \
    --header 'Content-Type: application/json' \
    --data-raw '{"statement":"INSERT INTO cars DOCUMENTS (:doc1)","args":{"doc1":{"_id":{"id":"777","locationId":"123456"},"color":"blue","timestamp":"1732192529"}}}'

This:

Uses the API key generated earlier
Inserts a new document into the cars collection, which will implicitly be created

List all items in the `cars` collection

We can then verify that the document was created with:

curl -X POST http://localhost/2164bef3-37c0-489c-9ac6-c94b034525d7/api/v4/store/execute \
    --header 'Authorization: bearer YOUR_API_KEY' \
    --header 'Content-Type: application/json' \
    --data-raw '{"statement":"SELECT * FROM cars"}'

Uninstalling using Helm

Before continuing, ensure any Big Peer resources that were created have been deleted. You can check for existing resources using the following command:

kubectl get bigpeers,bigpeerapps --namespace ditto

It is recommended you delete these resources before uninstalling the Ditto Operator. Otherwise, if you plan to reinstall the Ditto Operator again later the new installation will immediately discover the previous resources and proceed to create them anew.

You can now delete the helm release that was created in Installing the Ditto Operator:

helm delete ditto-operator

Ditto Operator

Ditto Operator API

Connectors

Ditto Operator (Private Preview)

Introduction

Prerequisites

Kubernetes Environment

Tools

Installing Certificate Manager

Installing an Ingress

Installing Strimzi

Installing the Ditto Operator

Deploying a Big Peer

Configuring Big Peer

The Operator API

Creating an App

Using the Big Peer HTTP API

Generate an API Key

Insert an item into the `cars` collection

List all items in the `cars` collection

Uninstalling using Helm

Ditto Operator

Ditto Operator API

Connectors

​Introduction

​Prerequisites

​Kubernetes Environment

​Tools

​Installing Certificate Manager

​Installing an Ingress

​Installing Strimzi

​Installing the Ditto Operator

​Deploying a Big Peer

​Configuring Big Peer

​The Operator API

​Creating an App

​Using the Big Peer HTTP API

​Generate an API Key

​Insert an item into the cars collection

​List all items in the cars collection

​Uninstalling using Helm

Introduction

Prerequisites

Kubernetes Environment

Tools

Installing Certificate Manager

Installing an Ingress

Installing Strimzi

Installing the Ditto Operator

Deploying a Big Peer

Configuring Big Peer

The Operator API

Creating an App

Using the Big Peer HTTP API

Generate an API Key

Insert an item into the `cars` collection

List all items in the `cars` collection

Uninstalling using Helm