MongoDB Connector

​

How It Works

​

Data Modelling Considerations

​

ID Mapping Between Systems

{
    "_id": {
        "location": "LondonLiverpoolStreet",
        "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc"
    },
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    ...
}

• 1:1 ID (i.e. the same ID is used in both Ditto and MongoDB)
• Single Document Field
• Multiple Document Fields

​

1:1 ID

• NULL cannot be present in the ID
• Arrays cannot be present in the ID
• The ID cannot be an object

​

Single Document Field

{
    "_id": ObjectId("abc123"),
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

{
    "_id": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

​

Multiple Document Fields

{
    "_id": ObjectId("abc123"),
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

{
    "_id": {
        "location": "LondonLiverpoolStreet",
        "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc"
    },
    "location": "LondonLiverpoolStreet",
    "orderId": "7c0c20ed-b285-48a6-80cd-6dcf06d52bcc",
    "items": [
        "fries",
        "burger",
        "soda"
    ]
}

​

MongoDB Datatypes

​

EJSON Mode

{
  insertDate: ISODate( "{"$date":"2019-08-11T17:54:14.692Z"}" )
}

{
    "insertDate": { "$date": { "$numberLong": "1565546054692" } }
}

​

Native BSON Mapping

​

Setting Up the Connector

​

Pre-requisites

​

Create MongoDB Database

​

Create MongoDB Collections

db.runCommand({ collMod: "orders", changeStreamPreAndPostImages: {"enabled": true} })

​

Create a MongoDB Database User

​

Add Ditto IPs to MongoDB Allowlist

3.147.233.88
52.15.232.117
3.130.255.9

​

Configuring the Connector

​

Step-by-Step Guide

1

Fill in the database field, with the MongoDB database that you wish to synchronize with Ditto.

2

Fill in the connection string field, with the connection string of the MongoDB database that you wish to synchronize with Ditto. You can find this connection string in the MongoDB Atlas UI, under the Connect button for your cluster, ensuring that you use the credentials of the database user that you created in the prerequisites.The connection string must be supplied in the format mongodb+srv://<username>:<password>@<cluster>.mongodb.net/. For example to connect to the cluster with the URL my_cluster.mongodb.net, using the username my_user with a password of my_password, you would provide mongodb+srv://my_user:my_password@my_cluster.mongodb.net/. Non-SRV-based connection strings cannot be used with the MongoDB connector.

While the password is passed into the Ditto Portal as part of the connection string, it is stored securely in Ditto’s systems and is not accessible to Ditto staff.

3

Fill in the ID mapping field, with the ID mapping that you wish to use for the connector.You can add new collections to the connector by clicking the Add Collection button, entering the collection name, toggling initial sync, then selecting how you would like your IDs mapped.Enabling initial sync will trigger an initial sync of the collection from MongoDB to Ditto, this is rate limited and will be done in the background to avoid affecting the performance of the connector. Once the initial sync is complete, the connector will continue to synchronize the collection on an ongoing basis via MongoDB Change Streams.For very large collections, you may opt to not rely on the Connector to perform the initial sync, and instead perform the initial sync manually via the Ditto API so that you have finer-grained control over the process.If you select the Match IDs option, then the ID fields used between Ditto and MongoDB will be identical.If you select the Map fields to Ditto ID option, then you will need to add all of the fields that you wish to map to the Ditto ID.Once done, click the Add collection button to add the collection to the connector configuration, repeat this step for each collection you wish to synchronize with Ditto.

ID Mapping is a critical part of configuring the connector, please makes sure you read the ID Mapping section carefully and configure as necessary for your use case.

4

Once all of the fields have been added, click the Save button to create the connector.

5

Once the connector has been created, you will be able to see the status of the connector in the Ditto Portal. It will first show as Pending while the connector is being created, then once it is ready it will show as Running. In case of an error, the connector will show as Failed, and you will be able to see the error message in the Ditto Portal, see also Troubleshooting Connectivity.

​

Troubleshooting Connectivity

• Ensure that the MongoDB connection string is reachable
• Ensure that the MongoDB database user has the correct permissions
• Ensure that the MongoDB cluster has the correct IP addresses added to the allowlist
• Ensure that the MongoDB database exists
• Ensure that the MongoDB collections exist
• Ensure that the MongoDB collections have changeStreamPreAndPostImages set to true
• Ensure that it can create all of the necessary metadata collections

​

Reconfiguring the Connector

• You need to add a new collection to the connector
• You need to remove a collection from the connector
• You need to change the ID mapping for a collection
• You need to change the credentials used by the connector

​

Behavior on Re-configuration

• If adding a new collection, the connector will automatically trigger an initial sync of the collection from MongoDB to Ditto, if initial sync is enabled in the collection configuration.
• If changing the ID mapping for a collection, the connector will automatically trigger an initial sync of the collection from MongoDB to Ditto if enabled in the collection configuration. All documents imported during this initial sync will be using the new ID mapping, but existing documents in Ditto will not be updated, you will need to manually delete these old documents in Ditto if desired.
• If toggling initial sync on a collection, the connector will not automatically trigger an initial sync of the collection from MongoDB to Ditto, unless there are other changes to the collection configuration (e.g. ID mapping).
• If removing a collection, the connector will stop synchronizing the collection between MongoDB and Ditto. Existing documents in this Ditto collection will not be deleted you will need to manually delete these documents in Ditto if desired.
• If changing the credentials used by the connector, the connector will automatically reconnect to MongoDB using the new credentials. No document changes will be missed as part of this rotation process.

​

Internal Metadata

• Current MongoDB sessions, to avoid circular writes between the two systems. This is stored in the __ditto_connector_sessions internal collection within your MongoDB database. These are periodically cleaned up by the connector when the sessions are no longer needed.
• Documents that failed to synchronize from Ditto to MongoDB (for example they may have been rejected due to schema validation issues). These are stored in the __ditto_unsynced_documents internal collection within your MongoDB database. These documents serve as a record for you to know which documents have failed to synchronize, and can be used to manually resync the documents. This collection is not automatically cleaned up by the connector, you can choose to delete the documents in this collection if you no longer need to keep track of these documents.
• BSON field mappings for documents, currently stored as a metadata field (prefixed with _ within the Ditto document)

​

Interaction with Deletes