Ditto Operator (Private Preview)
Introduction
The Ditto Operator allows you to deploy Big Peer (aka Ditto Server), Ditto’s cloud sync engine and data integration platform, to your own self-hosted Kubernetes environment.
Ditto Operator simplifies the configuration of the various components that form Big Peer, allowing you to focus on developing applications that get you connected to the cloud.
Prerequisites
A few prequisites need to be running before you install the Operator and deploy Big Peer.
Kubernetes Environment
The following steps assume you have a Kubernetes cluster, on at least version 1.31.
If you’re looking to try the Operator locally, we recommend using kind, a tool for running Kubernetes locally using Docker.
Tools
You’ll also need to have kubectl and helm installed, with the appropriate KUBECONFIG current-context
set.
Installing Certificate Manager
The Ditto Operator uses cert-manager to issue various auth-related certificates. You can install cert-manager into the target cluster using:
Installing an Ingress
The Ditto Operator manages ingresses to route traffic into the various Big Peer services.
We recommend using an ingress-nginx, as specific nginx specific annotations are used for optimal hashing. This guide will assume that nginx is being used.
The Ditto Operator can support other ingress controllers, but these may require different annotations. Please get in touch with Ditto support for guidance.
To install nginx:
If you opted to use Kind config provided to run the cluster locally, run the following:
If you opted to use Kind config provided to run the cluster locally, run the following:
If you are not installing the ingress-nginx in a kind
cluster, you can run:
Installing Strimzi
The Ditto Operator uses strimzi to create a Kafka cluster and topic for the transaction log. You must install strimzi, a Kakfa Operator, into the target cluster using:
For convenience, this instructs Strimzi to watch all namespaces for Kafka resources.
If you wish to constrain this behaviour (for example you deploy Kafka for your other applications in another namespace) you can use --set watchNamespaces
to specify the namespace you’re using for Ditto. Eg: --set watchNamespaces=ditto,another-ditto-namespace
Installing the Ditto Operator
With all prerequisites taken care of, the latest stable version of the Ditto Operator can be installed using the helm chart and image located at oci://quay.io/ditto-external/ditto-operator
:
This installs the Operator into a newly created ditto
namespace. All further steps will refer to this namespace.
You can now verify the Ditto Operator is installed and running with:
The various Ditto-related CRDs (Custom Resource Definitions) that were installed by the chart can be viewed using:
Which will return output similar to:
Deploying a Big Peer
Here’s a minimal example to deploy Big Peer:
This will deploy a Big Peer which is:
- Running on version 1.43.0
- Defines an ingress on localhost
- Enables Ditto SDK device auth via the provider named
__playgroundProvider
:- Devices have read and write access to all collections
- A Small Peer can authenticate using an “Online Playground” identity with a shared token of
abc123
- This is a global authentication configuration which can be used to authenticate to any app on the Big Peer
OnlinePlayground is ideal for development and testing, as it allows you to quickly define a shared token that can be shared with devices for auth.
However, it should not be used for production. OnlineWithAuthentication is more appropriate for this.
Small Peers using an OnlinePlayground
or OnlineWithAuthentication
identity try to connect to Ditto’s cloud service by default. Extra configuration is required to point it to your Big Peer.
- In the parameters to
OnlinePlayground
, provide a custom authentication URL pointing to the ingress, e.g. “https://my-host” - In the parameters to
OnlinePlayground
, set “enable cloud sync” tofalse
- Set a
TransportConfig
with a websocket URL pointing to your ingress. In the Rust SDK, for example:transport_config.connect.websocket_urls.insert("wss://my-host".to_owned())
If the BigPeer
was applied successfully, the various components of the Big Peer can be observed by running:
Which will match all running pods with the above BigPeer
’s metadata.name
, which is "example"
in this case:
Configuring Big Peer
Once Big Peer is deployed, you’ll need to configure it for first time usage.
The Operator API
The Ditto Operator provides a RESTful API for creating Apps and managing their configuration.
At this time, the Operator API does not have an authentication mechanism, and should not be exposed to the publicly. It is intended for use within the K8s cluster only.
To access the API, we recommend port-forwarding to ensure it remains secure and only accessible from your local machine.
For example:
The following steps will assume you’ve portforwarded in this way.
Creating an App
In Ditto, an App is a logical container for your data. An App needs to be registered with Big Peer before you can start creating documents.
Each app has a unique ID that’s used to identify it across different Big Peers and devices. You’ll use this ID for all interactions with your App, including:
- Connecting to the App from the Ditto SDK
- Making HTTP API calls to the App
- Setting up replication between Big Peers
Using the Operator API is the simplest way to create a new App.
Assuming you’ve port-forwarded to the Operator as described in the previous section, create an App using a POST request as follows:
This will create an App called example-app
, running on the example
Big Peer, in the ditto
namespace.
If successfil, you should receive a response like this:
Take note of the appId
from the response, you’ll need this later to use the HTTP API.
Using the Operator API is the simplest way to create a new App.
Assuming you’ve port-forwarded to the Operator as described in the previous section, create an App using a POST request as follows:
This will create an App called example-app
, running on the example
Big Peer, in the ditto
namespace.
If successfil, you should receive a response like this:
Take note of the appId
from the response, you’ll need this later to use the HTTP API.
Alternatively, you can create an app by declaring a BigPeerApp
resource.
You’re responsible for choosing an App ID when creating an App in this way. It must be a valid UUID, and needs to be unique to each Big Peer.
This will create an App called example-app
, running on the example
Big Peer, in the ditto
namespace.
You can verify that your app was created successfully by checking the status of the BigPeerApp
resource:
Using the Big Peer HTTP API
Using the HTTP API is the quickest way to verify your installation is running.
If you’ve deployed locally using the kind
example config provided, these queries should work out of the box.
Otherwise, you may need to update the localhost
according to the ingress you’ve configured on your K8s deployment.
Generate an API Key
To use the Big Peer HTTP API, you’ll need to generate an API key. This can only be done using the Operator API.
For this guide, we’ll create an API key that can has read and write access to all collections, with a very long expiry. For production scenarios, you may want to customise the scope of the API Keys access, and its expiry.
First, ensure you have port-forwarded to the Operator as described in the previous section.
Then, create an API key for your app using a POST request:
You’ll receive the raw API key as the response:
The raw API key is only shown once and is not stored anywhere. Make sure to save it securely, as you’ll need it for all HTTP API requests.
You can verify the API key was created by listing all keys for your app:
You can also verify this using kubectl
as the Operator will make a BigPeerApiKey object to manage this key:
Insert an item into the cars
collection
These examples assume you’ve deployed locally using the kind
example config provided. If not, you may need to update localhost
to the hostname you’ve configured as for your ingress.
Remember to replace YOUR_API_KEY
with the API key generated earlier.
This:
- Uses the API key generated earlier
- Inserts a new document into the
cars
collection, which will implicitly be created
List all items in the cars
collection
We can then verify that the document was created with:
Uninstalling using Helm
Before continuing, ensure any Big Peer resources that were created have been deleted. You can check for existing resources using the following command:
It is recommended you delete these resources before uninstalling the Ditto Operator. Otherwise, if you plan to reinstall the Ditto Operator again later the new installation will immediately discover the previous resources and proceed to create them anew.
You can now delete the helm release that was created in Installing the Ditto Operator: