Introduction
The Ditto Operator allows you to deploy Big Peer (aka Ditto Server), Ditto’s cloud sync engine and data integration platform, to your own self-hosted Kubernetes environment. Ditto Operator simplifies the configuration of the various components that form Big Peer, allowing you to focus on developing applications that get you connected to the cloud.Prerequisites
A few prequisites need to be running before you install the Operator and deploy Big Peer.Kubernetes Environment
The following steps assume you have a Kubernetes cluster, on at least version 1.31. If you’re looking to try the Operator locally, we recommend using kind, a tool for running Kubernetes locally using Docker.Example Kind Config
Example Kind Config
Here’s an example configuration you can use, which will expose the Big Peer Ingress on your local device, allowing you to follow the steps later in this guide:
Tools
You’ll also need to have kubectl and helm installed, with the appropriate KUBECONFIGcurrent-context
set.
Installing Certificate Manager
The Ditto Operator uses cert-manager to issue various auth-related certificates. You can install cert-manager into the target cluster using:Installing an Ingress
The Ditto Operator manages ingresses to route traffic into the various Big Peer services. We recommend using an ingress-nginx, as specific nginx specific annotations are used for optimal hashing. This guide will assume that nginx is being used.The Ditto Operator can support other ingress controllers, but these may require different annotations. Please get in touch with Ditto support for guidance.
If you opted to use Kind config provided to run the cluster locally, run the following:
Installing Strimzi
The Ditto Operator uses strimzi to create a Kafka cluster and topic for the transaction log. You must install strimzi, a Kakfa Operator, into the target cluster using:For convenience, this instructs Strimzi to watch all namespaces for Kafka resources.If you wish to constrain this behaviour (for example you deploy Kafka for your other applications in another namespace) you can use
--set watchNamespaces
to specify the namespace you’re using for Ditto. Eg: --set watchNamespaces=ditto,another-ditto-namespace
Installing the Ditto Operator
With all prerequisites taken care of, the latest stable version of the Ditto Operator can be installed using the helm chart and image located atoci://quay.io/ditto-external/ditto-operator
:
ditto
namespace. All further steps will refer to this namespace.
You can now verify the Ditto Operator is installed and running with:
Deploying a Big Peer
Here’s a minimal example to deploy Big Peer:- Running on version 1.43.0
- Defines an ingress on localhost
- Enables Ditto SDK device auth via the provider named
__playgroundProvider
:- Devices have read and write access to all collections
- A Small Peer can authenticate using an “Online Playground” identity with a shared token of
abc123
- This is a global authentication configuration which can be used to authenticate to any app on the Big Peer
OnlinePlayground is ideal for development and testing, as it allows you to quickly define a shared token that can be shared with devices for auth.However, it should not be used for production. OnlineWithAuthentication is more appropriate for this.
Small Peers using an
OnlinePlayground
or OnlineWithAuthentication
identity try to connect to Ditto’s cloud service by default. Extra configuration is required to point it to your Big Peer.- In the parameters to
OnlinePlayground
, provide a custom authentication URL pointing to the ingress, e.g. “https://my-host” - In the parameters to
OnlinePlayground
, set “enable cloud sync” tofalse
- Set a
TransportConfig
with a websocket URL pointing to your ingress. In the Rust SDK, for example:transport_config.connect.websocket_urls.insert("wss://my-host".to_owned())
BigPeer
was applied successfully, the various components of the Big Peer can be observed by running:
BigPeer
’s metadata.name
, which is "example"
in this case:
Configuring Big Peer
Once Big Peer is deployed, you’ll need to configure it for first time usage.The Operator API
The Ditto Operator provides a RESTful API for creating Apps and managing their configuration. At this time, the Operator API does not have an authentication mechanism, and should not be exposed to the publicly. It is intended for use within the K8s cluster only. To access the API, we recommend port-forwarding to ensure it remains secure and only accessible from your local machine. For example:Creating an App
In Ditto, an App is a logical container for your data. An App needs to be registered with Big Peer before you can start creating documents. Each app has a unique ID that’s used to identify it across different Big Peers and devices. You’ll use this ID for all interactions with your App, including:- Connecting to the App from the Ditto SDK
- Making HTTP API calls to the App
- Setting up replication between Big Peers
Using the Operator API is the simplest way to create a new App.Assuming you’ve port-forwarded to the Operator as described in the previous section, create an App using a POST request as follows:This will create an App called Take note of the
example-app
, running on the example
Big Peer, in the ditto
namespace.If successfil, you should receive a response like this:appId
from the response, you’ll need this later to use the HTTP API.App-scoped Authentication Providers
The auth providers listed in theBigPeer
resource are available to all apps. You can also configure providers for a specific BigPeerApp
by including an auth
configuration. The following example defines an external webhook with the provider name my-provider
.
anonymous
provider may also be specified at the app level. Make sure to give it the name __playgroundProvider
so that it can be used by Ditto SDKs using an OnlinePlayground
identity.
Using the Big Peer HTTP API
Using the HTTP API is the quickest way to verify your installation is running.If you’ve deployed locally using the
kind
example config provided, these queries should work out of the box.Otherwise, you may need to update the localhost
according to the ingress you’ve configured on your K8s deployment.Generate an API Key
To use the Big Peer HTTP API, you’ll need to generate an API key. This can only be done using the Operator API. For this guide, we’ll create an API key that can has read and write access to all collections, with a very long expiry. For production scenarios, you may want to customise the scope of the API Keys access, and its expiry. First, ensure you have port-forwarded to the Operator as described in the previous section. Then, create an API key for your app using a POST request:The raw API key is only shown once and is not stored anywhere. Make sure to save it securely, as you’ll need it for all HTTP API requests.
kubectl
as the Operator will make a BigPeerApiKey object to manage this key:
Insert an item into the cars
collection
These examples assume you’ve deployed locally using the
kind
example config provided. If not, you may need to update localhost
to the hostname you’ve configured as for your ingress.Remember to replace YOUR_API_KEY
with the API key generated earlier.- Uses the API key generated earlier
- Inserts a new document into the
cars
collection, which will implicitly be created