Example BigPeerReplication specs
Example BigPeerReplication specs
kind
cluster. Where applicable, there are considerations for other deployment scenarios.
0.3.0
or above is required.
See the Operator Getting Started guide to get the Operator deployed.
bp1
and bp2
, running in the ditto
namespace:
OnlineWithAuthentication
and OnlinePlayground
auth types for SDK connectivityBigPeer
custom resources.Patch BigPeer resources to trust each other's CA secrets
BigPeerApp
resources:
app-bp1
, on Big Peer bp1
app-bp2
, on Big Peer bp2
2164bef3-37c0-489c-9ac6-c94b034525d7
BigPeerReplication
resource for each app and specifying the data it should subscribe to.
bp1
as the initiating Big Peer and bp2
as the target Big Peer.“Initiating” refers to the Big Peer that initiates the connection — that is, the one which specifies the hostname of the big peer it’s connecting to. However, data can still be replicated in either direction (or both), depending on how subscriptions are defined.BigPeerReplication
resource on each Big Peer for the same app.
In this example, we’ll sync documents from the cars
collection where color='blue'
— replicating them in both directions between Big Peers.
Create a BigPeerReplication on the target Big Peer
app-bp2
on bp2
to accept a connection from other Big Peerscars
collection where color = 'blue'
from any connected Big Peer — (in this case, from app-bp1
on bp1
when we connect the two in the next step)Create a BigPeerReplication on the initiating Big Peer
connections
list.ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040
If the Big Peers live in separate clusters, the BigPeerReplication
services will need to be exposed such that they can be reached from the other cluster. See Configuring Ingresses for more.app-bp1
on bp1
to app-bp2
on bp2
, which we prepared in the previous stepapp-bp1
to all documents in the cars
collection where color = 'blue'
from any connected Big Peer — in this case, from app-bp2
on bp2
Validate configuration
Step by step example
curl
commands below to match the hostname or IP address you’ve configured as your ingress.Alternatively, you can access the APIs by port-forwarding a Big Peer service manually. For example:Create a Big Peer HTTP API Key
Insert a document into bp1
cars
collection on bp1
:Read the document from bp2
bp1
to bp2
is functioning correctly.Insert a second document into bp2
Read the second document from bp1
bp1
to the target bp2
.
Create a BigPeerReplication on the target Big Peer
app-bp2
on bp2
to accept a connection from other Big Peerscars
collection where color = 'blue'
from any connected Big Peer — (in this case, from app-bp1
on bp1
when we connect the two in the next step)Create a BigPeerReplication on the source app
connections
list so that this Big Peer knows where to initiate the connection.ditto-bp2-replication-blue-cars-bp2.ditto.svc.cluster.local:4040
If the Big Peers live in seperate clusters, the BigPeerReplication
services will need to be exposed such that they can be reached from the other cluster. See Configuring Ingresses for more.app-bp1
on bp1
to app-bp2
on bp2
, which we prepared in the previous stepbp2
, which does have an active subscription, ensuring that only bp2
receives data in this setupValidate configuration
4040
internally.
ditto-<bigpeer-name>-replication-<replication-name>
.4040
externally.4040
.BigPeerReplication
resources to use the external address and port for the connections
field.nginx
has been configured per the Operator Getting Started guide, but these steps will look similar for any nginx deloyment.Configure TCP Services
Update NGINX Ingress Controller
Update Service
DELETE
operation. This will create a tombstone, which will propagate to other connected Big Peers if the document covered falls within their subscription. As of Big Peer 1.42.0
, tombstones will be automatically evicted from each Big Peer after a period (by default) of 30 days.
DELETE
the document (for instance, when tombstones propagating to your applications is undesired because it has semantic meaning and would influence application behaviour), you can directly EVICT
the document.
However, you must first ensure that the Big Peer you’re evicting the document from does not have a subscription to another Big Peer which covers this document, causing it to “resurrect”.
A common pattern is to design subscriptions such that they depend on a temporal property in the document - for example, subscribing to orders that are created within the last 5 days. This then allows you to evict the documents when conditions are such that it no longer falls within any active subscriptions.