> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ditto.live/llms.txt
> Use this file to discover all available pages before exploring further.

# Big Peer to Big Peer Replication

Big Peer to Big Peer Replication (BP2BP for short) enables data synchronization between two or more Ditto Big Peers.

With BP2BP, you can configure fine-grained replication of data, allowing collections to be replicated unidirectionally or bidirectionally between clusters.

This is especially powerful for:

* **High availability**: create fault tolerance by replicating data live, across multiple deployments and regions in the cloud
* **Edge sync**: replicate from centralized systems to Big Peer peers deployed in, or near, the edge, bringing relevant data closer to edge applications
* **Data migrations**: moving apps or workloads between environments by replicating data live

Replication is **highly configurable**, with support for **DQL-based filtering** to control exactly what data is synced between peers. You can replicate entire collections or narrow the scope to just the documents you care about — giving you flexibility in performance, consistency, and topology design.

## Replication Scenarios

BP2BP gives you fine-grained control over your replication topology.

You can use it to build more complex replication patterns such as one-to-many and mesh topologies, by combining multiple unidirectional and bidirectional replication pairs.

You have full flexibility to configure the connections and filters in each replication between Big Peers. This gives you full control over data locality, resource usage and network usage.

Two common topologies are outlined below.

### One to Many

In a **one-to-many** topology, a central Big Peer pushes data out to multiple others.

This is ideal for fanning out data, where a single Big Peer acts as the source of truth.

The central Big Peer may be connected to an external data source — such as a cloud application or a MongoDB database — allowing you to distribute centralized data to edge locations.

```mermaid theme={null}
graph TD
    bp1(("bp1")) --> bp2(("bp2"))
    bp1 --> bp3(("bp3"))
    bp1 --> bp4(("bp4"))

```

If the downstream Big Peers are in edge environments with unreliable connectivity, or if it's difficult to establish inbound networking, we recommend configuring the connection address of the central Big Peer from the downstream Big Peers.

This ensures that edge Big Peers initiate the connection, and will automatically attempt to reconnect whenever connectivity is restored.

<Accordion title="Example BigPeerReplication specs">
  ```yaml theme={null}
  # bp1 "central" Big Peer
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp1
    namespace: ditto
    labels:
      ditto.live/app: app-bp1
  ---
  # bp2
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp2
    namespace: ditto
    labels:
      ditto.live/app: app-bp2
  spec:
    connections:
      - ditto-bp1-replication-replication-bp1.ditto.svc.cluster.local:4040
    subscriptions:
      queriesByCollection:
        cars:
          - "true"
  ---
  # bp3
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp3
    namespace: ditto
    labels:
      ditto.live/app: app-bp3
  spec:
    connections:
      - ditto-bp1-replication-replication-bp1.ditto.svc.cluster.local:4040
    subscriptions:
      queriesByCollection:
        cars:
          - "true"
  ---
  # bp4
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp4
    namespace: ditto
    labels:
      ditto.live/app: app-bp4
  spec:
    connections:
      - ditto-bp1-replication-replication-bp1.ditto.svc.cluster.local:4040
    subscriptions:
      queriesByCollection:
        cars:
          - "true"
  ```
</Accordion>

### Mesh

In a **mesh topology**, every Big Peer connects to every other Big Peer. This is useful for high availability scenarios, allowing you to create a highly resilient mesh in the cloud.

```mermaid theme={null}
graph LR
    bp1(("bp1")) <--> bp2(("bp2"))
    bp2 <--> bp3
    bp1 <--> bp3(("bp3"))
    bp2 <--> bp4
    bp1 <--> bp4(("bp4"))
    bp3 <--> bp4

```

Under the hood, BP2BP replication is powered by the same replication protocol that's used by the Ditto SDK. This topology is therefore highly fault-tolerant, if the connection between two Big Peers is interrupted, changes are still able to successfully replicate across other connected Big Peers.

<Accordion title="Example BigPeerReplication specs">
  ```yaml theme={null}
  # bp1 connects to bp2, bp3, bp4
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp1
    namespace: ditto
    labels:
      ditto.live/app: app-bp1
  spec:
    connections:
      - ditto-bp2-replication-bp2.ditto.svc.cluster.local:4040
      - ditto-bp3-replication-bp3.ditto.svc.cluster.local:4040
      - ditto-bp4-replication-bp4.ditto.svc.cluster.local:4040
  	subscriptions:
  	  queriesByCollection:
  	    cars:
  	      - "true"
  ---
  # bp2 connects to bp1, bp3, bp4
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp2
    namespace: ditto
    labels:
      ditto.live/app: app-bp2
  spec:
    connections:
      - ditto-bp1-replication-bp1.ditto.svc.cluster.local:4040
      - ditto-bp3-replication-bp3.ditto.svc.cluster.local:4040
      - ditto-bp4-replication-bp4.ditto.svc.cluster.local:4040
  	subscriptions:
  	  queriesByCollection:
  	    cars:
  	      - "true"
  ---
  # bp3 connects to bp1, bp2, bp4
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp3
    namespace: ditto
    labels:
      ditto.live/app: app-bp3
  spec:
    connections:
      - ditto-bp1-replication-bp1.ditto.svc.cluster.local:4040
      - ditto-bp2-replication-bp2.ditto.svc.cluster.local:4040
      - ditto-bp4-replication-bp4.ditto.svc.cluster.local:4040
  	subscriptions:
  	  queriesByCollection:
  	    cars:
  	      - "true"
  ---
  # bp4 connects to bp1, bp2, bp3
  apiVersion: ditto.live/v1alpha1
  kind: BigPeerReplication
  metadata:
    name: replication-bp4
    namespace: ditto
    labels:
      ditto.live/app: app-bp4
  spec:
    connections:
      - ditto-bp1-replication-bp1.ditto.svc.cluster.local:4040
      - ditto-bp2-replication-bp2.ditto.svc.cluster.local:4040
      - ditto-bp3-replication-bp3.ditto.svc.cluster.local:4040
  	subscriptions:
  	  queriesByCollection:
  	    cars:
  	      - "true"
  EOF
  ```
</Accordion>

# Getting Started

BP2BP is configured and managed using the Ditto Operator.

If you haven't already done so, we recommend following the [Operator Getting Started guide](../operator-quickstart) to get set up with the Ditto Operator.

To help explain the key concepts, the following guide will assume you've deployed the Operator using this guide, on a local `kind` cluster. Where applicable, there are considerations for other deployment scenarios.

## Prerequisites

### Deploy Ditto Operator

Version `0.3.0` or above is required.

See the [Operator Getting Started guide](../operator-quickstart) to get the Operator deployed.

### Deploy Big Peers

BP2BP supports replication between any number of Big Peers, deploying in the same or different Kubernetes clusters.

For more detail on how to deploy a Big Peer consult the [Operator Getting Started guide](../operator-quickstart#deploying-a-big-peer).

For the purposes of this guide, we'll deploy two Big Peers, `bp1` and `bp2`, running in the `ditto` namespace:

```bash theme={null}
cat <<'EOF' | kubectl apply -f -
---
# Create Big Peer 1
apiVersion: ditto.live/v1alpha1
kind: BigPeer
metadata:
  name: bp1
  namespace: ditto
spec:
  version: 1.43.0
  network:
    ingress:
      # A unique ingress is needed for each Big Peer
      host: bp1.localhost
  auth:
    providers:
      dummyProvider:
        tokenWebhook:
          url: http://dummy-auth-webhook.ditto.svc.cluster.local/validate_token
      onlinePlayground:
        anonymous:
          permission:
            read:
              everything: true
              queriesByCollection: {}
            write:
              everything: true
              queriesByCollection: {}
          sessionLength: 630000
          sharedToken: abc123
---
# Create Big Peer 2
apiVersion: ditto.live/v1alpha1
kind: BigPeer
metadata:
  name: bp2
  namespace: ditto
spec:
  version: 1.43.0
  network:
    ingress:
      # A unique ingress is needed for each Big Peer
      host: bp2.localhost
  auth:
    providers:
      dummyProvider:
        tokenWebhook:
          url: http://dummy-auth-webhook.ditto.svc.cluster.local/validate_token
      onlinePlayground:
        anonymous:
          permission:
            read:
              everything: true
              queriesByCollection: {}
            write:
              everything: true
              queriesByCollection: {}
          sessionLength: 630000
          sharedToken: abc123
EOF
```

This will create two Big Peers which:

* Have unique names
* Have unique ingresses on localhost (this will be useful later for our validation)
* Support both `OnlineWithAuthentication` and `OnlinePlayground` auth types for SDK connectivity

### Authenticate Big Peers

BP2BP replication uses mutual authentication to ensure that only trusted Big Peers can communicate with one another.

When a Big Peer is deployed, the Ditto Operator automatically issues it a self-signed CA certificate, which is stored in a secret named:

```bash theme={null}
ditto-<big-peer-name>-auth-ca
```

To establish trust between peers, each Big Peer must be configured to trust the CA of the other.

This process varies a little depending on whether the Big Peers are deployed in the same K8s cluster, or in separate clusters.

<Tabs>
  <Tab title="Same Cluster">
    For the purposes of this guide, we deploy the Big Peers in the same cluster.

    Therefore we can directly reference the CA secrets that already exist in both `BigPeer` custom resources.

    <Steps>
      <Step title="Patch BigPeer resources to trust each other's CA secrets">
        ```bash theme={null}
        # On bp1, trust bp2
        kubectl patch bigpeer bp1 -n ditto \
            --type='merge' \
            -p='
            spec:
              auth:
                trustedCaCertificates:
                    - name: ditto-bp2-auth-ca
            '

        # On bp2, trust bp1
        kubectl patch bigpeer bp2 -n ditto \
            --type='merge' \
            -p='
            spec:
              auth:
                trustedCaCertificates:
                    - name: ditto-bp1-auth-ca
            '
        ```
      </Step>
    </Steps>
  </Tab>

  <Tab title="Cross Cluster">
    In cross-cluster deployments, secrets must be manually exported from one cluster and applied to the other.

    You can do this however you see fit, the follow steps are one simple option.

    <Steps>
      <Step title="Export the public CA certificate from the first Big Peer">
        ```bash theme={null}
        # On cluster 1 (bp1), extract the public CA cert
        kubectl get secret ditto-bp1-auth-ca -n ditto \
            -o jsonpath='{.data.ca\.crt}' | base64 -d > bp1.crt
        ```
      </Step>

      <Step title="Transfer the cert to the second cluster and create a Secret">
        ```bash theme={null}
        # Copy bp1.crt to cluster 2, then run:
        kubectl create secret generic ditto-bp1-auth-ca \
            --from-file=ca.crt=bp1.crt \
            -n ditto
        ```
      </Step>

      <Step title="Reference the imported CA in the second BigPeer">
        ```bash theme={null}
        kubectl patch bigpeer bp2 -n ditto \
            --type='merge' \
            -p='
            spec:
                trustedCaCertificates:
                    - name: ditto-bp1-auth-ca
            '
        ```
      </Step>

      <Step title="Repeat in reverse for bp2 → bp1">
        On the second cluster (bp2):

        ```bash theme={null}
        # Extract the public CA cert
        kubectl get secret ditto-bp2-auth-ca -n ditto \
            -o jsonpath='{.data.ca\.crt}' | base64 -d > bp2.crt
        ```

        Transfer `bp2.crt` to the first cluster, then:

        ```bash theme={null}
        # Create the secret on cluster 1
        kubectl create secret generic ditto-bp2-auth-ca \
            --from-file=ca.crt=bp2.crt \
            -n ditto

        # Patch bp1 to trust bp2's CA
        kubectl patch bigpeer bp1 -n ditto \
            --type='merge' \
            -p='
            spec:
                trustedCaCertificates:
                    - name: ditto-bp2-auth-ca
            '
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

### Apps

BP2BP is configured at the app level, with each connected Big Peer app forming part of a data mesh — similar to how edge devices participate in syncing when using the Ditto SDK.

For replication to work, each Big Peer must host an app with the **same app ID**.

We'll create an app on each of the deployed Big Peers by creating `BigPeerApp` resources:

```bash theme={null}
cat <<'EOF' | kubectl apply -n ditto -f -
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerApp
metadata:
  name: app-bp1
  namespace: ditto
  labels:
    ditto.live/big-peer: bp1
spec:
  appId: 2164bef3-37c0-489c-9ac6-c94b034525d7
---
apiVersion: ditto.live/v1alpha1
kind: BigPeerApp
metadata:
  name: app-bp2
  namespace: ditto
  labels:
    ditto.live/big-peer: bp2
spec:
  appId: 2164bef3-37c0-489c-9ac6-c94b034525d7
EOF
```

In this example we've:

* Created app `app-bp1`, on Big Peer `bp1`
* Created app `app-bp2`, on Big Peer `bp2`

Crucially, these two apps share the same app ID: `2164bef3-37c0-489c-9ac6-c94b034525d7`

## Configuring Replication

Big Peer can replicate data either **bidirectionally** or **unidirectionally**.

To enable replication, each Big Peer app must be explicitly configured to receive data from other connected Big Peers. This is done by creating a `BigPeerReplication` resource for each app and specifying the data it should subscribe to.

<Info>
  In the following examples, we'll refer to `bp1` as the **initiating** Big Peer and `bp2` as the **target** Big Peer.

  "Initiating" refers to the Big Peer that initiates the connection — that is, the one which specifies the hostname of the big peer it's connecting to. However, data can still be replicated in either direction (or both), depending on how subscriptions are defined.
</Info>

### Creating a Bidirectional Replication

To replicate data bidirectionally between two Big Peers, you'll need to define a `BigPeerReplication` resource on **each** Big Peer for the **same app**.

In this example, we'll sync documents from the `cars` collection where `color='blue'` — replicating them in **both directions** between Big Peers.

<Steps>
  <Step title="Create a BigPeerReplication on the target Big Peer">
    First, we need to prepare the app on the target Big Peer to receive a connection and subscribe to the documents we want to replicate from other Big Peers that connect to it.

    ```yaml theme={null}
    cat <<'EOF' | kubectl apply -n ditto -f -
    apiVersion: ditto.live/v1alpha1
    kind: BigPeerReplication
    metadata:
      name: blue-cars-bp2
      namespace: ditto
      labels:
        ditto.live/app: app-bp2
    spec:
      subscriptions:
        queriesByCollection:
          cars:
            - "color='blue'"
    EOF
    ```

    This will:

    * Prepare `app-bp2` on `bp2` to accept a connection from other Big Peers
    * Configure it to subscribe to documents in the `cars` collection where `color = 'blue'` from any connected Big Peer — (in this case, from `app-bp1` on `bp1` when we connect the two in the next step)
  </Step>

  <Step title="Create a BigPeerReplication on the initiating Big Peer">
    Next, we need to configure replication on our initiating Big Peer. We'll do this by again specifying which documents to subscribe to, in this case with an identical filter, to achieve bidirectional replication.

    We must also provide the hostname of the target Big Peer app in the `connections` list.

    <Info>
      In our example, both Big Peers live in the same K8s cluster and namespaces. Therefore we can use Kubernetes service DNS to reference the service directly as:  `ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040`

      If the Big Peers live in separate clusters, the `BigPeerReplication` services will need to be exposed such that they can be reached from the other cluster. See [Configuring Ingresses](#configuring-ingresses) for more.
    </Info>

    ```yaml theme={null}
    cat <<'EOF' | kubectl apply -n ditto -f -
    apiVersion: ditto.live/v1alpha1
    kind: BigPeerReplication
    metadata:
      name: blue-cars-bp1
      namespace: ditto
      labels:
        ditto.live/app: app-bp1
    spec:
      connections:
        - ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040
      subscriptions:
        queriesByCollection:
          cars:
            - "color='blue'"
    EOF
    ```

    This will:

    * Connect `app-bp1` on `bp1` to `app-bp2` on `bp2`, which we prepared in the previous step
    * Subscribe `app-bp1` to all documents in the `cars` collection where `color = 'blue'` from any connected Big Peer — in this case, from `app-bp2` on `bp2`
  </Step>

  <Step title="Validate configuration">
    If replication has been configured correctly, you should see two replication server pods running, one for each Big Peer:

    ```bash theme={null}
    kubectl get pods -n ditto -l 'ditto.live/replication-app-id=2164bef3-37c0-489c-9ac6-c94b034525d7'
    ```

    ```yaml theme={null}
    NAME                                                   READY   STATUS    RESTARTS   AGE
    ditto-bp1-replication-blue-cars-bp1-5d5497f6fb-ccvmb   1/1     Running   0          3m19s
    ditto-bp2-replication-blue-cars-bp2-796d64c87-wdcrm    1/1     Running   0          8m36s
    ```

    To confirm that documents are correctly syncing, first verify that documents replicate from initiating to target Big Peers:

    * Insert a document that matches the subscription on the initiating Big Peer
    * Query for the document in the target Big Peer to confirm it exists

    Then, perform the inverse:

    * Insert a document that matches the subscription on the target Big Peer
    * Query for the document in the initiating Big Peer to confirm it exists

    We recommend using the HTTP API to do this. See below a worked example to illustrate.

    <Accordion title="Step by step example">
      <Info>
        This example assumes you've deployed the Big Peers in the [examples above](#deploy-big-peers), running in the same Kind Kubernetes cluster, using the recommended configuration from the [Operator Getting Started](../operator-quickstart) guide. This makes two separate ingresses for each Big Peer available over localhost for ease of validation.

        If you're using a different environment, you may need to update the `curl` commands below to match the hostname or IP address you've configured as your ingress.

        Alternatively, you can access the APIs by port-forwarding a Big Peer service manually. For example:

        ```bash theme={null}
        kubectl port-forward -n ditto service/bp1-api 8080:80
        ```
      </Info>

      <Steps>
        <Step title="Create a Big Peer HTTP API Key">
          If you haven't already, follow the steps in the [Operator Getting Started](../operator-quickstart#generate-an-api-key) guide to create an API Key for the App.
        </Step>

        <Step title="Insert a document into bp1">
          Use the HTTP API to create a new document in the `cars` collection on `bp1`:

          ```bash theme={null}
            curl -X POST http://bp1.localhost/2164bef3-37c0-489c-9ac6-c94b034525d7/api/v4/store/execute \
              --header "Authorization: bearer YOUR_API_KEY" \
              --header "Content-Type: application/json" \
              --data-raw '{
                "statement": "INSERT INTO cars DOCUMENTS (:doc1)",
                "args": {
                  "doc1": {
                    "_id": {"id": "001", "locationId": "2345"},
                    "color": "blue",
                    "type": "suv"
                  }
                }
              }'
          ```

          Expected response:

          ```yaml theme={null}
            {
              ...
              "mutatedDocumentIds":[{"id":"001","locationId":"2345"}],
              ...
            }
          ```
        </Step>

        <Step title="Read the document from bp2">
          After a short delay to allow replication, run:

          ```bash theme={null}
          curl -X POST http://bp2.localhost/2164bef3-37c0-489c-9ac6-c94b034525d7/api/v4/store/execute \
            --header "Authorization: bearer YOUR_API_KEY" \
            --header "Content-Type: application/json" \
            --data-raw '{
              "statement": "SELECT * FROM cars"
            }'
          ```

          Expected response:

          ```json theme={null}
          {
          ...
            "items": [
              {
                "_id": {"id": "001", "locationId": "2345"},
                "color": "blue",
                "timestamp": "1732192529"
              }
            ]
          ...
          }
          ```

          This confirms that replication from `bp1` to `bp2` is functioning correctly.
        </Step>

        <Step title="Insert a second document into bp2">
          Now confirm replication in the opposite direction:

          ```bash theme={null}
          curl -X POST http://bp2.localhost/2164bef3-37c0-489c-9ac6-c94b034525d7/api/v4/store/execute \
            --header "Authorization: bearer YOUR_API_KEY" \
            --header "Content-Type: application/json" \
            --data-raw '{
              "statement": "INSERT INTO cars DOCUMENTS (:doc2)",
              "args": {
                "doc2": {
                  "_id": {"id": "002", "locationId": "2345"},
                  "color": "blue",
                  "type": "hatchback"
                }
              }
            }'
          ```

          Expected response:

          ```yaml theme={null}
          {
            ...
            "mutatedDocumentIds":[{"id":"002","locationId":"2345"}],
            ...
          }
          ```
        </Step>

        <Step title="Read the second document from bp1">
          After a short delay to allow replication, run:

          ```bash theme={null}
          curl -X POST http://bp1.localhost/2164bef3-37c0-489c-9ac6-c94b034525d7/api/v4/store/execute \
            --header "Authorization: bearer YOUR_API_KEY" \
            --header "Content-Type: application/json" \
            --data-raw '{
              "statement": "SELECT * FROM cars"
            }'
          ```

          Expected output:

          ```json theme={null}
          {
            ...
            "items":[
              {
                "_id": {"id": "001", "locationId": "2345"},
                "color": "blue",
                "type": "suv",
                "timestamp": "1732192529"
              },
              {
                "_id": {"id": "002", "locationId": "2345"},
                "color": "blue",
                "type": "hatchback",
                "timestamp": "1732199999"
              }
            ]
            ...
          }
          ```
        </Step>
      </Steps>
    </Accordion>
  </Step>
</Steps>

### Creating a Unidirectional Replication

Configuring a unidirectional replication is very similar to bidirectional, but only one Big Peer subscribes to data.

For this example, we'll replicate from the initiating `bp1` to the target `bp2`.

<Steps>
  <Step title="Create a BigPeerReplication on the target Big Peer">
    First, we need to prepare the app on the target Big Peer to receive a connection and subscribe to the documents we want to replicate from the initiating Big Peer.

    ```yaml theme={null}
    cat <<'EOF' | kubectl apply -n ditto -f -
    apiVersion: ditto.live/v1alpha1
    kind: BigPeerReplication
    metadata:
      name: blue-cars-bp2
      namespace: ditto
      labels:
        ditto.live/app: app-bp2
    spec:
      subscriptions:
        queriesByCollection:
          cars:
            - "color='blue'"
    EOF
    ```

    As before, this will:

    * Prepare `app-bp2` on `bp2` to accept a connection from other Big Peers
    * Configure it to subscribe to documents in the `cars` collection where `color = 'blue'` from any connected Big Peer — (in this case, from `app-bp1` on `bp1` when we connect the two in the next step)
  </Step>

  <Step title="Create a BigPeerReplication on the source app">
    Next, we need to configure replication on our initiating Big Peer. Since we only want to replicate **to** the target Big Peer, we won't specify any subscriptions on the initiating Big Peer.

    We do, however, need to provide the hostname of the target Big Peer in the `connections` list so that this Big Peer knows where to initiate the connection.

    <Info>
      In our example, both Big Peers live in the same K8s cluster and namespaces. Therefore we can use Kubernetes service DNS to reference the service directly as:  `ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040`

      If the Big Peers live in separate clusters, the `BigPeerReplication` services will need to be exposed such that they can be reached from the other cluster. See [Configuring Ingresses](#configuring-ingresses) for more.
    </Info>

    ```yaml theme={null}
    cat <<'EOF' | kubectl apply -n ditto -f -
    apiVersion: ditto.live/v1alpha1
    kind: BigPeerReplication
    metadata:
      name: blue-cars-bp1
      namespace: ditto
      labels:
        ditto.live/app: app-bp1
    spec:
      connections:
        - ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040
    EOF
    ```

    This will:

    * Connect `app-bp1` on `bp1` to `app-bp2` on `bp2`, which we prepared in the previous step
    * Not subscribe to any documents from connected Big Peers — since we have no subscriptions defined
    * Enable one-way replication **to** `bp2`, which does have an active subscription, ensuring that only `bp2` receives data in this setup
  </Step>

  <Step title="Validate configuration">
    If replication has been configured correctly, you should see replication server pods running for both Big Peers.

    ```bash theme={null}
    kubectl get pods -n ditto -l 'ditto.live/replication-app-id=2164bef3-37c0-489c-9ac6-c94b034525d7'
    ```

    ```yaml theme={null}
    NAME                                                   READY   STATUS    RESTARTS   AGE
    ditto-bp1-replication-blue-cars-bp1-5d5497f6fb-ccvmb   1/1     Running   0          3m19s
    ditto-bp2-replication-blue-cars-bp2-796d64c87-wdcrm    1/1     Running   0          8m36s
    ```

    To confirm that documents are replicating:

    * Insert a document that matches the subscription on the initiating Big Peer
    * Query for the document in the target Big Peer to confirm it exists

    Refer to the validation in the [previous example](#creating-a-bidirectional-replication) for a step by step illustration of this (skipping the last two steps, as the replication is unidirectional)
  </Step>
</Steps>

# Configuring Ingresses

Many production scenarios for BP2BP will involve replication between Big Peers deployed in separate Kubernetes clusters.

For these scenarios, ingresses may be required to allow Big Peers to reach each other for replication.

The exact solution will depend on how you're managing the networking of your K8s cluster. The general steps required are:

1. Ensure your ingress controller is configured to support TCP traffic (many only handle HTTP/HTTPS by default).
2. Define TCP forwarding rules that map an external port to the BP2BP replication service, which listens on port `4040` internally.
   * The BP2BP service name follows the format: `ditto-<bigpeer-name>-replication-<replication-name>`.
3. Update your ingress controller Service to expose TCP port `4040` externally.
4. Ensure any network security groups, firewalls, or cloud load balancers allow TCP traffic on port `4040`.
5. Update your `BigPeerReplication` resources to use the external address and port for the `connections` field.

Below is an example of how to configure a TCP ingress using nginx.

<Tabs>
  <Tab title="NGINX Ingress Controller">
    For the purpose of this example, we'll assume `nginx` has been configured per the [Operator Getting Started guide](../operator-quickstart), but these steps will look similar for any nginx deloyment.

    <Steps>
      <Step title="Configure TCP Services">
        First, create or update the TCP services ConfigMap:

        ```bash theme={null}
        # Create the TCP services ConfigMap
        kubectl create configmap tcp-services -n ingress-nginx \
            --from-literal=4040="ditto/ditto-bp1-replication-blue-cars-bp1:4040"

        # Or if it already exists, patch it:
        kubectl patch configmap tcp-services -n ingress-nginx \
            --patch '{"data":{"4040":"ditto/ditto-bp1-replication-blue-cars-bp1:4040"}}'
        ```
      </Step>

      <Step title="Update NGINX Ingress Controller">
        Patch the existing NGINX ingress controller deployment to add the TCP services configmap argument and port:

        ```bash theme={null}
        # Add the TCP services configmap argument to the existing args array
        kubectl patch deployment ingress-nginx-controller -n ingress-nginx \
            --type=json \
            -p='[
                {
                    "op": "add",
                    "path": "/spec/template/spec/containers/0/args/-",
                    "value": "--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services"
                }
            ]'

        # Add the BP2BP port to the existing ports array
        kubectl patch deployment ingress-nginx-controller -n ingress-nginx \
            --type=json \
            -p='[
                {
                    "op": "add",
                    "path": "/spec/template/spec/containers/0/ports/-",
                    "value": {
                        "containerPort": 4040,
                        "hostPort": 4040,
                        "name": "bp2bp",
                        "protocol": "TCP"
                    }
                }
            ]'
        ```
      </Step>

      <Step title="Update Service">
        Patch the ingress controller service to add the TCP port:

        ```bash theme={null}
        # Add the BP2BP port to the existing service ports array
        kubectl patch service ingress-nginx-controller -n ingress-nginx \
            --type=json \
            -p='[
                {
                    "op": "add",
                    "path": "/spec/ports/-",
                    "value": {
                        "name": "bp2bp",
                        "port": 4040,
                        "protocol": "TCP",
                        "targetPort": 4040
                    }
                }
            ]'
        ```
      </Step>
    </Steps>
  </Tab>
</Tabs>

After configuring TCP ingress, you'll need to update your BigPeerReplication resources to use the external addresses of the ingress controllers when connecting to Big Peers in other clusters.

<Info>
  The examples above expose port 4040, which is the default BP2BP port. Adjust the port numbers if you've configured your Big Peers to use different ports.
</Info>

# Conflict Resolution

BP2BP uses the same underlying replication protocol as the Ditto SDK, meaning that all document synchronization is causally and eventually consistent, based on conflict-free, mergeable data structures.

When changes are made on different Big Peers while offline or disconnected, Ditto automatically resolves conflicts once peers reconnect. Each field within a document is versioned and merged independently using CRDTs (Conflict-Free Replicated Data Types) to ensure deterministic and predictable resolution.

For more detail on how conflicts are handled and how merges behave across data types, see [Conflict Resolution in Ditto](/key-concepts/syncing-data#conflict-resolution).

# Deletion and Eviction

As BP2BP is powered by Ditto's replication protocol, deletions and evictions behave much like they would in a Ditto SDK mesh at the edge.

There are two main methods that can be applied to delete data from connected Big Peers:

* Delete the data first with a `DELETE` operation. This will create a tombstone, which will propagate to other connected Big Peers if the document covered falls within their subscription. As of Big Peer `1.42.0`, tombstones will be automatically evicted from each Big Peer after a period (by default) of 30 days.
* If you don't wish to `DELETE` the document (for instance, when tombstones propagating to your applications is undesired because it has semantic meaning and would influence application behaviour), you can directly `EVICT` the document.

  However, you must first ensure that the Big Peer you're evicting the document from does not have a subscription to another Big Peer which covers this document, causing it to "resurrect".
  A common pattern is to design subscriptions such that they depend on a temporal property in the document - for example, subscribing to orders that are created within the last 5 days. This then allows you to evict the documents when conditions are such that it no longer falls within any active subscriptions.

The SDK [DELETE](/sdk/latest/crud/delete) documentation covers detailed eviction and deletion strategies that also apply here.