Big Peer to Big Peer Replication (BP2BP for short) enables data synchronization between two or more Ditto Big Peers.

With BP2BP, you can configure fine-grained replication of data, allowing collections to be replicated unidirectionally or bidirectionally between clusters.

This is especially powerful for:

  • High availability: create fault tolerance by replicating data live, across multiple deployments and regions in the cloud
  • Edge sync: replicate from centralized systems to Big Peer peers deployed in, or near, the edge, bringing relevant data closer to edge applications
  • Data migrations: moving apps or workloads between environments by replicating data live

Replication is highly configurable, with support for DQL-based filtering to control exactly what data is synced between peers. You can replicate entire collections or narrow the scope to just the documents you care about — giving you flexibility in performance, consistency, and topology design.

​
Replication Scenarios

BP2BP gives you fine-grained control over your replication topology.

You can use it to build more complex replication patterns such as one-to-many and mesh topologies, by combining multiple unidirectional and bidirectional replication pairs.

You have full flexibility to configure the connections and filters in each replication between Big Peers. This gives you full control over data locality, resource usage and network usage.

Two common topologies are outlined below.

​
One to Many

In a one-to-many topology, a central Big Peer pushes data out to multiple others.

This is ideal for fanning out data, where a single Big Peer acts as the source of truth.

The central Big Peer may be connected to an external data source — such as a cloud application or a MongoDB database — allowing you to distribute centralized data to edge locations.

If the downstream Big Peers are in edge environments with unreliable connectivity, or if it’s difficult to establish inbound networking, we recommend configuring the connection address of the central Big Peer from the downstream Big Peers.

This ensures that edge Big Peers initiate the connection, and will automatically attempt to reconnect whenever connectivity is restored.

​
Mesh

In a mesh topology, every Big Peer connects to every other Big Peer. This is useful for high availability scenarios, allowing you to create a highly resilient mesh in the cloud.

Under the hood, BP2BP replication is powered by the same replication protocol that’s used by the Ditto SDK. This topology is therefore highly fault-tolerant, if the connection between two Big Peers is interrupted, changes are still able to successfully replicate across other connected Big Peers.

​
Getting Started

BP2BP is configured and managed using the Ditto Operator.

If you haven’t already done so, we recommend following the Operator Getting Started guide to get set up with the Ditto Operator.

To help explain the key concepts, the following guide will assume you’ve deployed the Operator using this guide, on a local kind cluster. Where applicable, there are considerations for other deployment scenarios.

​
Prerequisites

​
Deploy Ditto Operator

Version 0.3.0 or above is required.

See the Operator Getting Started guide to get the Operator deployed.

​
Deploy Big Peers

BP2BP supports replication between any number of Big Peers, deploying in the same or different Kubernetes clusters.

For more detail on how to deploy a Big Peer consult the Operator Getting Started guide.

For the purposes of this guide, we’ll deploy two Big Peers, bp1 and bp2, running in the ditto namespace:

cat <<'EOF' | kubectl apply -f -
---
# Create Big Peer 1
apiVersion: ditto.live/v1alpha1
kind: BigPeer
metadata:
  name: bp1
  namespace: ditto
spec:
  version: 1.43.0
  network:
    ingress:
      # A unique ingress is needed for each Big Peer
      host: bp1.localhost
  auth:
    providers:
      dummyProvider:
        tokenWebhook:
          url: http://dummy-auth-webhook.ditto.svc.cluster.local/validate_token
      onlinePlayground:
        anonymous:
          permission:
            read:
              everything: true
              queriesByCollection: {}
            write:
              everything: true
              queriesByCollection: {}
          sessionLength: 630000
          sharedToken: abc123
---
# Create Big Peer 2
apiVersion: ditto.live/v1alpha1
kind: BigPeer
metadata:
  name: bp2
  namespace: ditto
spec:
  version: 1.43.0
  network:
    ingress:
      # A unique ingress is needed for each Big Peer
      host: bp2.localhost
  auth:
    providers:
      dummyProvider:
        tokenWebhook:
          url: http://dummy-auth-webhook.ditto.svc.cluster.local/validate_token
      onlinePlayground:
        anonymous:
          permission:
            read:
              everything: true
              queriesByCollection: {}
            write:
              everything: true
              queriesByCollection: {}
          sessionLength: 630000
          sharedToken: abc123
EOF

This will create two Big Peers which:

  • Have unique names
  • Have unique ingresses on localhost (this will be useful later for our validation)
  • Support both OnlineWithAuthentication and OnlinePlayground auth types for SDK connectivity

​
Authenticate Big Peers

BP2BP replication uses mutual authentication to ensure that only trusted Big Peers can communicate with one another.

When a Big Peer is deployed, the Ditto Operator automatically issues it a self-signed CA certificate, which is stored in a secret named:

ditto-<big-peer-name>-auth-ca

To establish trust between peers, each Big Peer must be configured to trust the CA of the other.

This process varies a little depending on whether the Big Peers are deployed in the same K8s cluster, or in separate clusters.

For the purposes of this guide, we deploy the Big Peers in the same cluster.

Therefore we can directly reference the CA secrets that already exist in both BigPeer custom resources.

1

Patch BigPeer resources to trust each other's CA secrets

# On bp1, trust bp2
kubectl patch bigpeer bp1 -n ditto \
    --type='merge' \
    -p='
    spec:
      auth:
        trustedCaCertificates:
            - name: ditto-bp2-auth-ca
    '

# On bp2, trust bp1
kubectl patch bigpeer bp2 -n ditto \
    --type='merge' \
    -p='
    spec:
      auth:
        trustedCaCertificates:
            - name: ditto-bp1-auth-ca
    '

​
Apps

BP2BP is configured at the app level, with each connected Big Peer app forming part of a data mesh — similar to how edge devices participate in syncing when using the Ditto SDK.

For replication to work, each Big Peer must host an app with the same app ID.

We’ll create an app on each of the deployed Big Peers by creating BigPeerApp resources:

cat <<'EOF' | kubectl apply -n ditto -f -
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerApp
metadata:
  name: app-bp1
  namespace: ditto
  labels:
    ditto.live/big-peer: bp1
spec:
  appId: 2164bef3-37c0-489c-9ac6-c94b034525d7
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerApp
metadata:
  name: app-bp2
  namespace: ditto
  labels:
    ditto.live/big-peer: bp2
spec:
  appId: 2164bef3-37c0-489c-9ac6-c94b034525d7
EOF

In this example we’ve:

  • Created app app-bp1, on Big Peer bp1
  • Created app app-bp2, on Big Peer bp2

Crucially, these two apps share the same app ID: 2164bef3-37c0-489c-9ac6-c94b034525d7

​
Configuring Replication

Big Peer can replicate data either bidirectionally or unidirectionally.

To enable replication, each Big Peer app must be explicitly configured to receive data from other connected Big Peers. This is done by creating a BigPeerReplication resource for each app and specifying the data it should subscribe to.

In the following examples, we’ll refer to bp1 as the initiating Big Peer and bp2 as the target Big Peer.

“Initiating” refers to the Big Peer that initiates the connection — that is, the one which specifies the hostname of the big peer it’s connecting to. However, data can still be replicated in either direction (or both), depending on how subscriptions are defined.

​
Creating a Bidirectional Replication

To replicate data bidirectionally between two Big Peers, you’ll need to define a BigPeerReplication resource on each Big Peer for the same app.

In this example, we’ll sync documents from the cars collection where color='blue' — replicating them in both directions between Big Peers.

1

Create a BigPeerReplication on the target Big Peer

First, we need to prepare the app on the target Big Peer to receive a connection and subscribe to the documents we want to replicate from other Big Peers that connect to it.

cat <<'EOF' | kubectl apply -n ditto -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerReplication
metadata:
  name: blue-cars-bp2
  namespace: ditto
  labels:
    ditto.live/app: app-bp2
spec:
  subscriptions:
    queriesByCollection:
      cars:
        - "color='blue'"
EOF

This will:

  • Prepare app-bp2 on bp2 to accept a connection from other Big Peers
  • Configure it to subscribe to documents in the cars collection where color = 'blue' from any connected Big Peer — (in this case, from app-bp1 on bp1 when we connect the two in the next step)
2

Create a BigPeerReplication on the initiating Big Peer

Next, we need to configure replication on our initiating Big Peer. We’ll do this by again specifying which documents to subscribe to, in this case with an identical filter, to achieve bidirectional replication.

We must also provide the hostname of the target Big Peer app in the connections list.

In our example, both Big Peers live in the same K8s cluster and namespaces. Therefore we can use Kubernetes service DNS to reference the service directly as: ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040

If the Big Peers live in separate clusters, the BigPeerReplication services will need to be exposed such that they can be reached from the other cluster. See Configuring Ingresses for more.

cat <<'EOF' | kubectl apply -n ditto -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerReplication
metadata:
  name: blue-cars-bp1
  namespace: ditto
  labels:
    ditto.live/app: app-bp1
spec:
  connections:
    - ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040
  subscriptions:
    queriesByCollection:
      cars:
        - "color='blue'"
EOF

This will:

  • Connect app-bp1 on bp1 to app-bp2 on bp2, which we prepared in the previous step
  • Subscribe app-bp1 to all documents in the cars collection where color = 'blue' from any connected Big Peer — in this case, from app-bp2 on bp2
3

Validate configuration

If replication has been configured correctly, you should see two replication server pods running, one for each Big Peer:

kubectl get pods -n ditto -l 'ditto.live/replication-app-id=2164bef3-37c0-489c-9ac6-c94b034525d7'
NAME                                                   READY   STATUS    RESTARTS   AGE
ditto-bp1-replication-blue-cars-bp1-5d5497f6fb-ccvmb   1/1     Running   0          3m19s
ditto-bp2-replication-blue-cars-bp2-796d64c87-wdcrm    1/1     Running   0          8m36s

To confirm that documents are correctly syncing, first verify that documents replicate from initiating to target Big Peers:

  • Insert a document that matches the subscription on the initiating Big Peer
  • Query for the document in the target Big Peer to confirm it exists

Then, perform the inverse:

  • Insert a document that matches the subscription on the target Big Peer
  • Query for the document in the initiating Big Peer to confirm it exists

We recommend using the HTTP API to do this. See below a worked example to illustrate.

​
Creating a Unidirectional Replication

Configuring a unidirectional replication is very similar to bidirectional, but only one Big Peer subscribes to data.

For this example, we’ll replicate from the initiating bp1 to the target bp2.

1

Create a BigPeerReplication on the target Big Peer

First, we need to prepare the app on the target Big Peer to receive a connection and subscribe to the documents we want to replicate from the initiating Big Peer.

cat <<'EOF' | kubectl apply -n ditto -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerReplication
metadata:
  name: blue-cars-bp2
  namespace: ditto
  labels:
    ditto.live/app: app-bp2
spec:
  subscriptions:
    queriesByCollection:
      cars:
        - "color='blue'"
EOF

As before, this will:

  • Prepare app-bp2 on bp2 to accept a connection from other Big Peers
  • Configure it to subscribe to documents in the cars collection where color = 'blue' from any connected Big Peer — (in this case, from app-bp1 on bp1 when we connect the two in the next step)
2

Create a BigPeerReplication on the source app

Next, we need to configure replication on our initiating Big Peer. Since we only want to replicate to the target Big Peer, we won’t specify any subscriptions on the initiating Big Peer.

We do, however, need to provide the hostname of the target Big Peer in the connections list so that this Big Peer knows where to initiate the connection.

In our example, both Big Peers live in the same K8s cluster and namespaces. Therefore we can use Kubernetes service DNS to reference the service directly as: ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040

If the Big Peers live in seperate clusters, the BigPeerReplication services will need to be exposed such that they can be reached from the other cluster. See Configuring Ingresses for more.

cat <<'EOF' | kubectl apply -n ditto -f -
apiVersion: ditto.live/v1alpha1
kind: BigPeerReplication
metadata:
  name: blue-cars-bp1
  namespace: ditto
  labels:
    ditto.live/app: app-bp1
spec:
  connections:
    - ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040
EOF

This will:

  • Connect app-bp1 on bp1 to app-bp2 on bp2, which we prepared in the previous step
  • Not subscribe to any documents from connected Big Peers — since we have no subscriptions defined
  • Enable one-way replication to bp2, which does have an active subscription, ensuring that only bp2 receives data in this setup
3

Validate configuration

If replication has been configured correctly, you should see replication server pods running for both Big Peers.

kubectl get pods -n ditto -l 'ditto.live/replication-app-id=2164bef3-37c0-489c-9ac6-c94b034525d7'
NAME                                                   READY   STATUS    RESTARTS   AGE
ditto-bp1-replication-blue-cars-bp1-5d5497f6fb-ccvmb   1/1     Running   0          3m19s
ditto-bp2-replication-blue-cars-bp2-796d64c87-wdcrm    1/1     Running   0          8m36s

To confirm that documents are replicating:

  • Insert a document that matches the subscription on the initiating Big Peer
  • Query for the document in the target Big Peer to confirm it exists

Refer to the validation in the previous example for a step by step illustration of this (skipping the last two steps, as the replication is unidirectional)

​
Configuring Ingresses

Many production scenarios for BP2BP will involve replication between Big Peers deployed in separate Kubernetes clusters.

For these scenarios, ingresses may be required to allow Big Peers to reach each other for replication.

The exact solution will depend on how you’re managing the networking of your K8s cluster. The general steps required are:

  1. Ensure your ingress controller is configured to support TCP traffic (many only handle HTTP/HTTPS by default).
  2. Define TCP forwarding rules that map an external port to the BP2BP replication service, which listens on port 4040 internally.
    • The BP2BP service name follows the format: ditto-<bigpeer-name>-replication-<replication-name>.
  3. Update your ingress controller Service to expose TCP port 4040 externally.
  4. Ensure any network security groups, firewalls, or cloud load balancers allow TCP traffic on port 4040.
  5. Update your BigPeerReplication resources to use the external address and port for the connections field.

Below is an example of how to configure a TCP ingress using nginx.

For the purpose of this example, we’ll assume nginx has been configured per the Operator Getting Started guide, but these steps will look similar for any nginx deloyment.

1

Configure TCP Services

First, create or update the TCP services ConfigMap:

# Create the TCP services ConfigMap
kubectl create configmap tcp-services -n ingress-nginx \
    --from-literal=4040="ditto/ditto-bp1-replication-blue-cars-bp1:4040"

# Or if it already exists, patch it:
kubectl patch configmap tcp-services -n ingress-nginx \
    --patch '{"data":{"4040":"ditto/ditto-bp1-replication-blue-cars-bp1:4040"}}'
2

Update NGINX Ingress Controller

Patch the existing NGINX ingress controller deployment to add the TCP services configmap argument and port:

# Add the TCP services configmap argument to the existing args array
kubectl patch deployment ingress-nginx-controller -n ingress-nginx \
    --type=json \
    -p='[
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/args/-",
            "value": "--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services"
        }
    ]'

# Add the BP2BP port to the existing ports array
kubectl patch deployment ingress-nginx-controller -n ingress-nginx \
    --type=json \
    -p='[
        {
            "op": "add",
            "path": "/spec/template/spec/containers/0/ports/-",
            "value": {
                "containerPort": 4040,
                "hostPort": 4040,
                "name": "bp2bp",
                "protocol": "TCP"
            }
        }
    ]'
3

Update Service

Patch the ingress controller service to add the TCP port:

# Add the BP2BP port to the existing service ports array
kubectl patch service ingress-nginx-controller -n ingress-nginx \
    --type=json \
    -p='[
        {
            "op": "add",
            "path": "/spec/ports/-",
            "value": {
                "name": "bp2bp",
                "port": 4040,
                "protocol": "TCP",
                "targetPort": 4040
            }
        }
    ]'

After configuring TCP ingress, you’ll need to update your BigPeerReplication resources to use the external addresses of the ingress controllers when connecting to Big Peers in other clusters.

The examples above expose port 4040, which is the default BP2BP port. Adjust the port numbers if you’ve configured your Big Peers to use different ports.

​
Conflict Resolution

BP2BP uses the same underlying replication protocol as the Ditto SDK, meaning that all document synchronization is causally and eventually consistent, based on conflict-free, mergeable data structures.

When changes are made on different Big Peers while offline or disconnected, Ditto automatically resolves conflicts once peers reconnect. Each field within a document is versioned and merged independently using CRDTs (Conflict-Free Replicated Data Types) to ensure deterministic and predictable resolution.

For more detail on how conflicts are handled and how merges behave across data types, see Conflict Resolution in Ditto.

​
Deletion and Eviction

As BP2BP is powered by Ditto’s replication protocol, deletions and evictions behave much like they would in a Ditto SDK mesh at the edge.

There are two main methods that can be applied to delete data from connected Big Peers:

  • Delete the data first with a DELETE operation. This will create a tombstone, which will propagate to other connected Big Peers if the document covered falls within their subscription. As of Big Peer 1.42.0, tombstones will be automatically evicted from each Big Peer after a period (by default) of 30 days.

  • If you don’t wish to DELETE the document (for instance, when tombstones propagating to your applications is undesired because it has semantic meaning and would influence application behaviour), you can directly EVICT the document.

    However, you must first ensure that the Big Peer you’re evicting the document from does not have a subscription to another Big Peer which covers this document, causing it to “resurrect”. A common pattern is to design subscriptions such that they depend on a temporal property in the document - for example, subscribing to orders that are created within the last 5 days. This then allows you to evict the documents when conditions are such that it no longer falls within any active subscriptions.

The SDK DELETE documentation covers detailed eviction and deletion strategies that also apply here.