Large Binary Files

See Syncing Data for more information on attachments in Ditto.

​

ATTACHMENT Object

The ATTACHMENT object stores your attachment outside the local device or object storage.

• The metadata component is a set of string values providing extra details about the file or complex document, such as name and size, among other identifying information.
• The blob store is an array of bytes representing your large file or complex document encoded as an ATTACHMENT data type. (See Blob Store)

​

Attachment Token

The attachment token encapsulates information Ditto uses to identify and retrieve the full attachment when needed; as in, you’ve explicitly fetched it in your app.

The following structure and corresponding table provide an overview of the schema the attachment token uses to model the data it contains:

{
  "id": "string",
  "len": 42,
  "metadata": {}
}

Here are the attachment token properties you can query to increase the efficiency of fetching operations in your app:

​

Example Document

Here is an attachment token structured within a document. In this structure, the picture attachment token acts as a placeholder for the full attachment storing a JPEG file named car_stock_photo.jpg:

{
    "_id": "abc123",
    "make": "Hyundai",
    "year": 2018,
    "color": "black",
    "picture": {
        "id": "<unique identifier>",
        "metadata": {
            "mime_type": "image/jpeg",
            "name": "car_stock_photo.jpg",
        },
        "len": 123
    }
}

​

Creating Attachments

There are two separate steps in the process of creating an attachment:

​

Initiating ATTACHMENT Objects

To create the ATTACHMENT object that encodes the actual contents of the file to store externally, call the newAttachment method on the store namespace.

let newAttachment = try await ditto.store.newAttachment(path: filePath)

​

Adding Optional Metadata

If desired, enclose extra information about the attachment, such as name and description. This metadata is useful for attachment fetching operations.

If you want to include information about the attachment, such as the attachment filename, as shown in the following snippet, a description, and other relevant details, enclose it in a metadata object as key-value pairs.

let metadata = ["name": "image.png"]
let newAttachmentWithMetadata = try await ditto.store.newAttachment(path: filePath, metadata: metadata)

For example, in the following snippet, the metadata property encapsulates the name of the attachment, as well as its description:

let metadata = ["name": "japan.png", "description": "This is an image of a sunset in Japan."];
let newAttachmentWithMetadata = try await ditto.store.newAttachment(path: filePath, metadata: metadata)

​

Inserting Attachments in a Document

After creating and storing a new attachment object in Ditto, you receive an attachment token as part of the response.

The following snippet demonstrates how to create a new document object containing a new attachment, and then insert it into the cars collection:

// First, create the attachment
let metadata = ["name": "image.png"]
let attachment = try await ditto.store.newAttachment(path: filePath, metadata: metadata)

// Then, create a new document object and store the attachment on the `my_attachment` field.
let newDocument: [String: Any] = ["_id": "123", "my_attachment": attachment]

// Finally, insert the new document into the collection.
// Note that the `my_attachment` field needs to be declared as an `ATTACHMENT` data type.
// When explicitly declaring types for your fields, the COLLECTION keyword is mandatory.
let result = await ditto.store.execute(
  query: "INSERT INTO COLLECTION docs (my_attachment ATTACHMENT) VALUES (:newDocument)",
  arguments: ["newDocument": newDocument]);

​

Fetching attachments

To fetch an attachment, you must first query the document containing the attachment token. The token is a pointer that references the full attachment stored externally.

The following snippet demonstrates how to fetch an attachment from a document with the ID 123 in the cars collection. The attachment token is stored in the my_attachment field of the document.

let result = try await ditto.store.execute(
    query: """
        SELECT * FROM COLLECTION cars (my_attachment ATTACHMENT)
        WHERE _id = "123"
    """,
)

let token = result.items.first?.value["my_attachment"] as? [String: Any]
if let token = token {
  let fetcher = try await ditto.store.fetchAttachment(token) { event in
      switch (event) {
          case .completed(let fetchedAttachment):
          case .progress:
          case .deleted:
      }
    }
}

​

Updating Attachment Tokens

Once an ATTACHMENT object is created and linked to a document, it becomes immutable, meaning its contents cannot be changed.

However, you can indirectly update attachments by performing an UPDATE operation on the document to replace the existing attachment token with a different one. The attachment token is the reference pointer linking the document to the attachment’s binary data stored separately.

// Updating a Ditto Document with an attachment
// Update the document "123" with a new attachment
try await ditto.store.execute(
    query: """
    UPDATE COLLECTION cars (my_attachment ATTACHMENT)
    SET my_attachment = :newAttachment
    WHERE _id = "123"
    """,
    arguments: ["newAttachment": newAttachment]
)

​

Deleting Attachments

Unlike documents, attachments cannot be explicitly deleted on their own. Instead, you modify the document containing the attachment token referencing it.

Attachment data stored within the Small Peer Ditto store is automatically garbage collected on a 10-minute cadence when no longer referenced. Currently, attachments can only be deleted by way of garbage collection.

The following table provides an overview of the various ways you can indirectly delete attachments:

The storage mechanism Small Peers use to store data, including blob data, depends on the platform:

• If running in the browser or a server-based system, data is stored in its Random Access Memory (RAM).
• If running on a mobile device like an iPhone, data is stored on its local filesystem.

​

Blob Store

A JSON blob stores binary data representing any file type. For example, it can store image files like JPEG, PNG, and TIFF; video files such as MP4; audio files, including MP3 and WAV; document files like PDF; and more.

The metadata is stored in the document, while the blob is stored in a dedicated location independent of documents. The following table provides an overview of blob storage locations:

​

Organizing by ID

Ditto organizes and stores blob data using the content-addressable network (CAN) paradigm. CAN is a distributed network architecture widely implemented in decentralized systems like Ditto’s where scalability, efficiency, and fault tolerance are essential requirements. (See the official Content-Addressable Network Wikipedia article.)

In more straightforward terms, Ditto organizes and accesses attachment data by the cryptographic hash table of its contents — for instance, attachment ID — rather than its physical location. A hash table is a data structure that stores key-value pairs.

By adhering to the CAN framework for attachment management, Ditto stores and retrieves data efficiently. For instance:

• Deduplication — If the same blob is created multiple times, Ditto stores only one copy. Similarly, if an attachment is added to multiple documents, they all reference the same blob.
• Conflict-free sync — Blobs are stored based on the cryptographic hash of their contents, ensuring that conflicts are effectively impossible.
• Decentralized — No central server is required to manage attachments. They can be created and accessed concurrently throughout the network, even with intermittent connectivity.

​

Purging Blob Data

Blob storage is managed internally using a reference-counting process known as garbage collection.

The garbage collection process running periodically in the background of Small Peers frees up space by removing blobs that no longer have any references — once an attachment has no remaining references, its blob is considered safe to remove, and the garbage collection process clears it from the device.

​

Syncing ATTACHMENT Objects

The two components comprising an ATTACHMENT object — metadata stored internally within a document and blob data storing the actual binary data of your attachment — sync in different ways:

• Metadata syncs just as any other document.
• Blob data syncs with an entirely different protocol than documents; one that is asynchronous and built on top of Ditto’s rainbow connection technology and optimizes for large binary data transfer. (See Mesh Networking)

Attachment blob sync being asynchronous provides three key advantages:

• Unblocking fetching operations so the document continues syncing without interruption when retrieving an attachment.
• Ditto multiplexes between these independent activities to balance needs for low latency and high throughput at the edge.
• Blob sync is resilient, so if connectivity is interrupted while transferring an attachment, progress is not lost. Once restored or if another peer possesses the attachment, Ditto resumes the transfer from the point of interruption.

Blob sync remains performant — Ditto uses data structures like Bloom filters to scale efficiently and remain light on system resources.

Bloom filters are randomized data structures designed to represent sets in a compressed form, making them highly effective in peer-to-peer environments where concerns such as memory usage and accuracy are prevalent. (See the official Bloom filter Wikipedia article.)

​

Garbage Collection

When an attachment token stored on a Small Peer end-user device is unused, meaning there are no remaining references to it, the garbage collection process running periodically in the background automatically deletes it to free up space.

Garbage collection is a reference-counting process that only runs on Small Peer devices (not on the Ditto Server). It involves scanning for objects, including blob objects, that are no longer needed or relevant and are therefore safe to delete from its device.

Ditto Server stores its data in Ditto’s third-party cloud object storage service provider, Amazon WebServices (AWS) Simple Storage Service (S3). For more information, see the official Amazon Simple Storage Service Documentation.

​

Lazy-Load Retrieval Pattern

Since the blob representing your large file is stored outside the local Ditto store, fetch it only when needed. This on-demand retrieval pattern, known as lazy loading, reduces resource usage on end-user devices.

For a realworld usage scenario, see either the demo chat app for iOS or Android in the getditto > demoapp-chat GitHub repository. For instance, in the iOS demo chat app, you can see a savvy implementation of ATTACHMENT with a full-resolution avatar image from a collection named User.