MongoDB Connector

​

How It Works

​

Connector Restarts and Recovery

​

Data Modelling Considerations

​

ID Mapping Between Systems

{
    "_id": {
        "location": "LondonLiverpoolStreet",
        "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc"
    },
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    ...
}

// ✅ Correct: include the ID mapping field in the initial insert
const newId = new ObjectId();
await collection.insertOne({
  _id: newId,
  _mid: newId.toString(),
  // ... other fields
});

// ❌ Incorrect: adding the ID mapping field in a separate operation
await collection.insertOne({ _id: new ObjectId(), /* ... */ });
await collection.updateOne(
  { _id: insertedId },
  { $set: { _mid: insertedId.toString() } }
);
// The connector cannot sync this document correctly.

• 1:1 ID (i.e. the same ID is used in both Ditto and MongoDB)
• Single Document Field
• Multiple Document Fields

​

1:1 ID

• NULL cannot be present in the ID
• Arrays cannot be present in the ID
• The ID cannot be an object

​

Single Document Field

{
    "_id": ObjectId("abc123"),
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

{
    "_id": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

​

Multiple Document Fields

{
    "_id": ObjectId("abc123"),
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

{
    "_id": {
        "location": "LondonLiverpoolStreet",
        "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc"
    },
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

​

MongoDB Datatypes

​

EJSON Mode

{
  insertDate: ISODate( "{"$date":"2019-08-11T17:54:14.692Z"}" )
}

{
    "insertDate": { "$date": { "$numberLong": "1565546054692" } }
}

​

Native BSON Mapping

DQL Strict Mode

• Strict mode off (recommended): Objects are synchronized as maps, with scalar fields inside them stored as registers. Because each field is represented independently, the connector can synchronize a change to a single field without re-syncing the rest of the document, improving performance both at scale and over peer-to-peer connections. This also matches the default behavior of Ditto SDK 5.0 and later.
• Strict mode on: Each object is synchronized as a single register, so the entire object is replaced on every change using last-write-wins semantics.

​

Settable Counters

​

Setting Up the Connector

​

Pre-requisites

​

Create MongoDB Database

​

Create MongoDB Collections

db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: {"enabled": true} })

​

Create a MongoDB Database User

​

Add Ditto IPs to MongoDB Allowlist

​

Configuring the Connector

​

Step-by-Step Guide

1

Fill in the database field, with the MongoDB database that you wish to synchronize with Ditto.

2

Fill in the connection string field, with the connection string of the MongoDB database that you wish to synchronize with Ditto. You can find this connection string in the MongoDB Atlas UI, under the Connect button for your cluster, ensuring that you use the credentials of the database user that you created in the prerequisites.The connection string must be supplied in the format mongodb+srv://<username>:<password>@<cluster>.mongodb.net/. For example to connect to the cluster with the URL my_cluster.mongodb.net, using the username my_user with a password of my_password, you would provide mongodb+srv://my_user:my_password@my_cluster.mongodb.net/. Non-SRV-based connection strings cannot be used with the MongoDB connector.

While the password is passed into the Ditto Portal as part of the connection string, it is stored securely in Ditto’s systems and is not accessible to Ditto staff.

3

Fill in the ID mapping field, with the ID mapping that you wish to use for the connector.You can add new collections to the connector by clicking the Add Collection button, which opens the Add collection to sync form. Enter the collection name, toggle initial sync, then choose how your fields are mapped to Ditto.Enabling initial sync will trigger an initial sync of the collection from MongoDB to Ditto, this is rate limited and will be done in the background to avoid affecting the performance of the connector. Once the initial sync is complete, the connector will continue to synchronize the collection on an ongoing basis via MongoDB Change Streams.For very large collections, you may opt to not rely on the Connector to perform the initial sync, and instead perform the initial sync manually via the Ditto API so that you have finer-grained control over the process.The form also exposes the Enable DQL Strict Mode and Enable canonical Extended JSON v2 format options, which control how MongoDB datatypes are represented in Ditto. See MongoDB Datatypes and DQL Strict Mode for guidance on these options.To map your fields, choose one of the document schema options:If you select Auto-generated document schema, MongoDB fields are mapped to Ditto fields automatically, and the ID fields used between Ditto and MongoDB will be identical. This option is intended for development only.If you select Custom document schema, you can map specific MongoDB fields to Ditto types. Set the Ditto type of each field you want to use in the ID mapping to ID, and set any fields you want to treat as settable counters to Counter.Once done, click the Add collection button to add the collection to the connector configuration, repeat this step for each collection you wish to synchronize with Ditto.

ID Mapping is a critical part of configuring the connector, please make sure you read the ID Mapping section carefully and configure as necessary for your use case.

4

Once all of the fields have been added, click the Save button to create the connector.

5

Once the connector has been created, you will be able to see the status of the connector in the Ditto Portal. It will first show as Pending while the connector is being created, then once it is ready it will show as Running. In case of an error, the connector will show as Failed, and you will be able to see the error message in the Ditto Portal, see also Troubleshooting Connectivity.

​

Troubleshooting Connectivity

• Ensure that the MongoDB connection string is reachable
• Ensure that the MongoDB database user has the correct permissions
• Ensure that the MongoDB cluster has the correct IP addresses added to the allowlist (as shown in the Ditto Portal)
• Ensure that the MongoDB database exists
• Ensure that the MongoDB collections exist
• Ensure that the MongoDB collections have changeStreamPreAndPostImages set to true
• Ensure that it can create all of the necessary metadata collections

​

changeStreamPreAndPostImages Not Enabled

Collection <name> must be configured with changeStreamPreAndPostImages enabled

db.getCollectionInfos({name: "your_collection_name"})

db.runCommand({ collMod: "your_collection_name", changeStreamPreAndPostImages: {"enabled": true} })

​

Troubleshooting Schema Validation Errors

​

Documents Rejected Due to Null Values

db.runCommand({
  collMod: "your_collection",
  validator: {
    $jsonSchema: {
      bsonType: "object",
      properties: {
        // Allow both the expected type and null
        responseField: { bsonType: ["string", "null"] },
        completedAt: { bsonType: ["date", "null"] },
        assigneeId: { bsonType: ["string", "null"] }
      }
    }
  }
})

​

Troubleshooting Runtime Issues

• Pod restarts or rescheduling
• Certificate rotation events
• Cluster maintenance operations

1. Confirm the connector status still shows Running in the Ditto Portal under Settings > MongoDB Connector.
2. Make a test change to a document in a synchronized MongoDB collection and verify whether it appears in Ditto within a few seconds.
3. If the change does not appear, the connector is likely stuck and not consuming change stream events.

​

Connector Stops Consuming Events After Infrastructure Changes

• New documents written to MongoDB collections are not appearing in Ditto
• The MongoDB change stream event rate drops to zero
• The connector status in the Ditto Portal still shows as Running

​

Reconfiguring the Connector

• You need to add a new collection to the connector
• You need to remove a collection from the connector
• You need to change the ID mapping for a collection
• You need to change the credentials used by the connector

​

Behavior on Re-configuration

• If adding a new collection, the connector will automatically trigger an initial sync of the collection from MongoDB to Ditto, if initial sync is enabled in the collection configuration.
• If changing the ID mapping for a collection, the connector will automatically trigger an initial sync of the collection from MongoDB to Ditto if enabled in the collection configuration. All documents imported during this initial sync will be using the new ID mapping, but existing documents in Ditto will not be updated, you will need to manually delete these old documents in Ditto if desired.
• If toggling initial sync on a collection, the connector will not automatically trigger an initial sync of the collection from MongoDB to Ditto, unless there are other changes to the collection configuration (e.g. ID mapping).
• If removing a collection, the connector will stop synchronizing the collection between MongoDB and Ditto. Existing documents in this Ditto collection will not be deleted you will need to manually delete these documents in Ditto if desired.
• If changing the credentials used by the connector, the connector will automatically reconnect to MongoDB using the new credentials. No document changes will be missed as part of this rotation process.

​

Sync Direction

• Ditto to MongoDB only: Changes made on Ditto devices sync to MongoDB, but changes made directly in MongoDB do not sync back to Ditto.
• MongoDB to Ditto only: Changes made in MongoDB sync to Ditto devices, but changes made on Ditto devices do not sync back to MongoDB.

​

Targeted Resync

​

When to Use Targeted Resync

• You notice specific documents that exist in MongoDB are missing from Ditto.
• A connector outage occurred and some documents may not have been picked up by MongoDB Change Streams during the downtime.
• Transient errors caused individual documents to fail synchronization, and you want to retry them without resyncing the entire collection.

​

Specifying Document IDs

{
    "document_ids": [
        { "$oid": "69af67ebb85f7f5ffe09a257" },
        { "$oid": "69af67ebb85f7f5ffe09a258" }
    ]
}

{
    "document_ids": [
        "69af67ebb85f7f5ffe09a257",
        "69af67ebb85f7f5ffe09a258"
    ]
}

​

Monitoring Resync Progress

​

Internal Metadata

• Current MongoDB sessions, to avoid circular writes between the two systems. This is stored in the __ditto_connector_sessions internal collection within your MongoDB database. These are periodically cleaned up by the connector when the sessions are no longer needed.
• Documents that failed to synchronize from Ditto to MongoDB (for example they may have been rejected due to schema validation issues). These are stored in the __ditto_unsynced_documents internal collection within your MongoDB database. These documents serve as a record for you to know which documents have failed to synchronize, and can be used to manually resync the documents. This collection is not automatically cleaned up by the connector, you can choose to delete the documents in this collection if you no longer need to keep track of these documents.
• BSON field mappings for documents, currently stored as a metadata field (prefixed with _ within the Ditto document)

​

Monitoring Connector Health

​

Checking __ditto_unsynced_documents

​

Comparing Document Counts

​

Using Timestamps

​

Interaction with Deletes