Working with Attachments

See Syncing Data for more information on attachments in Ditto.

​

ATTACHMENT Object

The ATTACHMENT object, shown on the right, stores your attachment outside the local device, is not directly visible in your code, and consists of metadata and a binary large object (blob) store.

• The metadata component is a set of string values providing extra details about the file or complex document, such as name and size, among other identifying information.
• The blob store is an array of bytes representing your large file or complex document encoded as an ATTACHMENT data type. (See Blob Store)

{
    "blob": [...], // array of bytes
    "metadata": {
        "mime_type": "image/jpeg",
        "name": "car_stock_photo.jpg"
    }
}

​

Attachment Token

The attachment token, shown on the right, encapsulates information Ditto uses to identify and retrieve the full attachment when needed; as in, you’ve explicitly fetched it in your app.

{
    "_id": "abc123",
    "make": "Hyundai",
    "year": 2018,
    "color": "black",
    "picture": {
        "id": "<unique identifier>",
        "metadata": {
            "mime_type": "image/jpeg",
            "name": "car_stock_photo.jpg",
        },
        "len": 123
    }
}

​

Schema

The following structure and corresponding table provide an overview of the schema the attachment token uses to model the data it contains:

{
  "id": "string",
  "len": 42,
  "metadata": {}
}

Here are the attachment token properties you can query to increase the efficiency of fetching operations in your app:

​

Blob Store

A JSON blob stores binary data representing any file type. For example, it can store image files like JPEG, PNG, and TIFF; video files such as MP4; audio files, including MP3 and WAV; document files like PDF; and more.

The metadata is stored in the document, while the blob is stored in a dedicated location independent of documents. The following table provides an overview of blob storage locations:

​

Organizing by ID

Ditto organizes and stores blob data using the content-addressable network (CAN) paradigm. CAN is a distributed network architecture widely implemented in decentralized systems like Ditto’s where scalability, efficiency, and fault tolerance are essential requirements. (See the official Content-Addressable Network Wikipedia article.)

In more straightforward terms, Ditto organizes and accesses attachment data by the cryptographic hash table of its contents — for instance, attachment ID — rather than its physical location. A hash table is a data structure that stores key-value pairs.

By adhering to the CAN framework for attachment management, Ditto stores and retrieves data efficiently. For instance:

• Deduplication — If the same blob is created multiple times, Ditto stores only one copy. Similarly, if an attachment is added to multiple documents, they all reference the same blob.
• Conflict-free sync — Blobs are stored based on the cryptographic hash of their contents, ensuring that conflicts are effectively impossible.
• Decentralized — No central server is required to manage attachments. They can be created and accessed concurrently throughout the network, even with intermittent connectivity.

​

Purging Blob Data

Blob storage is managed internally using a reference-counting process known as garbage collection.

The garbage collection process running periodically in the background of Small Peers frees up space by removing blobs that no longer have any references — once an attachment has no remaining references, its blob is considered safe to remove, and the garbage collection process clears it from the device.

​

Syncing ATTACHMENT Objects

The two components comprising an ATTACHMENT object — metadata stored internally within a document and blob data storing the actual binary data of your attachment — sync in different ways:

• Metadata syncs just as any other document.
• Blob data syncs with an entirely different protocol than documents; one that is asynchronous and built on top of Ditto’s rainbow connection technology and optimizes for large binary data transfer. (See Mesh Networking)

Attachment blob sync being asynchronous provides three key advantages:

• Unblocking fetching operations so the document continues syncing without interruption when retrieving an attachment.
• Ditto multiplexes between these independent activities to balance needs for low latency and high throughput at the edge.
• Blob sync is resilient, so if connectivity is interrupted while transferring an attachment, progress is not lost. Once restored or if another peer possesses the attachment, Ditto resumes the transfer from the point of interruption.

Blob sync remains performant — Ditto uses data structures like Bloom filters to scale efficiently and remain light on system resources.

Bloom filters are randomized data structures designed to represent sets in a compressed form, making them highly effective in peer-to-peer environments where concerns such as memory usage and accuracy are prevalent. (See the official Bloom filter Wikipedia article.)

​

Garbage Collection

When an attachment token stored on a Small Peer end-user device is unused, meaning there are no remaining references to it, the garbage collection process running periodically in the background automatically deletes it to free up space.

Garbage collection is a reference-counting process that only runs on Small Peer devices (not on the Big Peer cloud deployment). It involves scanning for objects, including blob objects, that are no longer needed or relevant and are therefore safe to delete from its device.

The Big Peer stores its data in Ditto’s third-party cloud object storage service provider, Amazon WebServices (AWS) Simple Storage Service (S3). For more information, see the official Amazon Simple Storage Service Documentation.

​

Lazy-Load Retrieval Pattern

Since the blob representing your large file is stored outside the local Ditto store, fetch it only when needed. This on-demand retrieval pattern, known as lazy loading, reduces resource usage on end-user devices.

For a realworld usage scenario, see either the demo chat app for iOS or Android in the getditto > demoapp-chat GitHub repository. For instance, in the iOS demo chat app, you can see a savvy implementation of ATTACHMENT with a full-resolution avatar image from a collection named User.

​

Creating Attachments

There are two separate steps in the process of creating an attachment:

​

Initiating ATTACHMENT Objects

To create the ATTACHMENT object that encodes the actual contents of the file to store externally, call the newAttachment method on the store namespace.

​

Adding Optional Metadata

If desired, enclose extra information about the attachment, such as name and description. This metadata is useful for attachment fetching operations.

If you want to include information about the attachment, such as the attachment filename, as shown in the following snippet, a description, and other relevant details, enclose it in a metadata object as key-value pairs.

For example, in the following snippet, the metadata property encapsulates the name of the attachment, as well as its description:

​

Inserting Attachments in a Document

After creating and storing a new attachment object in Ditto, you receive an attachment token as part of the response.

The following snippet demonstrates how to create a new document object containing a new attachment, and then insert it into the cars collection:

​

Deleting Attachments

Unlike documents, attachments cannot be explicitly deleted on their own. Instead, you modify the document containing the attachment token referencing it.

Attachment data stored within the Small Peer Ditto store is automatically garbage collected on a 10-minute cadence when no longer referenced. Currently, attachments can only be deleted by way of garbage collection.

The following table provides an overview of the various ways you can indirectly delete attachments:

The storage mechanism Small Peers use to store data, including blob data, depends on the platform:

• If running in the browser or a server-based system, data is stored in its Random Access Memory (RAM).
• If running on a mobile device like an iPhone, data is stored on its local filesystem.