Replacing Live Query Events with Store Observers

With Ditto SDK 4.11+, the new diffing APIs can be used with store observers to produce diffs between query results in consecutive invocations. This guide explains how to use the new diffing APIs to replace usage of the legacy live query APIs.

Overview

Live queries provide their callback with an event parameter that contains a diff of the current live query event against the previous one—a summary of which documents in the result have been:

Inserted: Indexes of new items
Updated: Indexes of items that existed previously, but have changed
Deleted: Indexes of items that were removed
Moved: Pairs of indexes showing items that changed position

Store observers don’t have a built-in feature that provides a diff, as generating it is computationally expensive. However, you can use the new Differ class to compute one when needed.

Key Differences

Legacy Live Query API

Provides automatic diffing through the event parameter
Maintains event.oldDocuments for deleted items (uses memory)
Includes event.isInitial flag for the first callback
Considers metadata changes as updates

Store Observer with Differ

Requires manual diffing using the Differ class
You must maintain previous results yourself
No built-in initial event detection
Ignores metadata-only changes when determining updates

Migration Example

Before: Using Legacy Live Query API

const appStateLegacy = {
  inserted: null,
  updated: null,
  deleted: null
};

ditto.store
  .collection('cars')
  .findAll()
  .observeLocal((documents, event) => {
    if (event.isInitial) {
      // Store all initial document IDs in app state
      appStateLegacy.inserted = documents.map(doc => doc.value._id);
      console.log("Initial live query event", appStateLegacy.inserted);
    } else {
      appStateLegacy.inserted = event.insertions?.map(
        (index) => documents[index].value._id
      );
      appStateLegacy.updated = event.updates?.map(
        (index) => documents[index].value._id
      );
      appStateLegacy.deleted = event.deletions?.map(
        // accessing `event.oldDocuments`
        (index) => event.oldDocuments[index].value._id
      );
      console.log("Live query event: ", JSON.stringify(appStateLegacy, null, 2));
    }
  });

After: Using Store Observer with Differ

Important Memory Management: QueryResults and QueryResultItems should be treated like database cursors. Always extract the data you need and then close/dematerialize them immediately. Never store QueryResultItems directly between observer emissions.

let differ = DittoDiffer()
var previousDocumentIds: [String] = [] // Store only extracted IDs

let observer = ditto.store.registerObserver(
  query: "SELECT * FROM cars") { queryResult in
    let diff = differ.diff(queryResult.items)

    // Extract current document IDs and dematerialize items
    let currentDocumentIds = queryResult.items.map { item in
        let id = item.value["_id"] as? String ?? "unknown"
        item.dematerialize() // Release memory after extracting data
        return id
    }

    // Handle deletions using stored IDs from previous emission
    for index in diff.deletions {
        let deletedId = previousDocumentIds[index]
        print("Deleted car with ID: \(deletedId)")
    }

    // Handle insertions using current IDs
    for index in diff.insertions {
        let insertedId = currentDocumentIds[index]
        print("Inserted car with ID: \(insertedId)")
    }

    // Handle updates using current IDs
    for index in diff.updates {
        let updatedId = currentDocumentIds[index]
        print("Updated car with ID: \(updatedId)")
    }

    // Store only the document IDs for next callback - no live references!
    previousDocumentIds = currentDocumentIds
}

// In v5 the differ is internal — the observer owns it. Use the (result, diff)
// overload of registerObserver. Closing the observer frees the differ. The
// suspend handler auto-closes the DittoQueryResult; never store or return
// result.items.
var previousDocumentIds: List<String> = emptyList()

val observer = ditto.store.registerObserver(
    "SELECT * FROM cars",
) { result, diff ->
    // Extract current document IDs and dematerialize items
    val currentDocumentIds = result.items.map { item ->
        val id = item.value["_id"].stringOrNull ?: "unknown"
        item.dematerialize() // Release memory after extracting data
        id
    }

    // Handle deletions using stored IDs from previous emission
    diff.deletions.forEach { index ->
        previousDocumentIds.getOrNull(index)?.let { deletedId ->
            println("Deleted car with ID: $deletedId")
        }
    }
    // Handle insertions using current IDs
    diff.insertions.forEach { index ->
        currentDocumentIds.getOrNull(index)?.let { insertedId ->
            println("Inserted car with ID: $insertedId")
        }
    }
    // Handle updates using current IDs
    diff.updates.forEach { index ->
        currentDocumentIds.getOrNull(index)?.let { updatedId ->
            println("Updated car with ID: $updatedId")
        }
    }

    // Store only the document IDs for next callback — no live references!
    previousDocumentIds = currentDocumentIds
}

const differ = new Differ();
let previousDocumentIds = []; // Store only extracted IDs

const changeHandler = (queryResult) => {
    const diff = differ.diff(queryResult.items);

    // Extract current document IDs and dematerialize items
    const currentDocumentIds = queryResult.items.map(item => {
        const id = item.value._id || 'unknown';
        item.dematerialize(); // Release memory after extracting data
        return id;
    });

    // Handle deletions using stored IDs from previous emission
    diff.deletions.forEach(index => {
        const deletedId = previousDocumentIds[index];
        console.log('Deleted car with ID:', deletedId);
    });

    // Handle insertions using current IDs
    diff.insertions.forEach(index => {
        const insertedId = currentDocumentIds[index];
        console.log('Inserted car with ID:', insertedId);
    });

    // Handle updates using current IDs
    diff.updates.forEach(index => {
        const updatedId = currentDocumentIds[index];
        console.log('Updated car with ID:', updatedId);
    });

    // Store only the document IDs for next callback - no live references!
    previousDocumentIds = currentDocumentIds;
}

const observer = ditto.store.registerObserver(
  "SELECT * FROM cars",
  changeHandler);

DittoDiffer differ = new DittoDiffer();
List<String> previousDocumentIds = new ArrayList<>(); // Store only extracted IDs

DittoStoreObserver observer = ditto.store.registerObserver(
    "SELECT * FROM cars",
    result -> {
        try (result) { // Auto-closes result when done
            DittoDiff diff = differ.diff(result.items);

            // Extract current document IDs and dematerialize items
            List<String> currentDocumentIds = new ArrayList<>();
            for (DittoQueryResultItem item : result.items) {
                String id = item.getValue().get("_id") != null ?
                            item.getValue().get("_id").toString() : "unknown";
                currentDocumentIds.add(id);
                item.dematerialize(); // Release memory after extracting data
            }

            // Handle deletions using stored IDs from previous emission
            for (int index : diff.getDeletions()) {
                if (index < previousDocumentIds.size()) {
                    String deletedId = previousDocumentIds.get(index);
                    System.out.println("Deleted car with ID: " + deletedId);
                }
            }

            // Handle insertions using current IDs
            for (int index : diff.getInsertions()) {
                String insertedId = currentDocumentIds.get(index);
                System.out.println("Inserted car with ID: " + insertedId);
            }

            // Handle updates using current IDs
            for (int index : diff.getUpdates()) {
                String updatedId = currentDocumentIds.get(index);
                System.out.println("Updated car with ID: " + updatedId);
            }

            // Store only the document IDs for next callback - no live references!
            previousDocumentIds = currentDocumentIds;
        } catch (Exception e) {
            // Handle any errors
            e.printStackTrace();
        }
    }
);

var differ = new DittoDiffer();
var previousDocumentIds = new List<string>(); // Store only extracted IDs

var observer = ditto.Store.RegisterObserver(
  "SELECT * FROM cars",
  (result) =>
  {
    using (result) // Disposes result when done
    {
        var diff = differ.Diff(result.Items);

        // Extract current document IDs and dematerialize items
        var currentDocumentIds = result.Items.Select(item =>
        {
            var id = item.Value.ContainsKey("_id") ?
                    item.Value["_id"].ToString() : "unknown";
            item.Dematerialize(); // Release memory after extracting data
            return id;
        }).ToList();

        // Handle deletions using stored IDs from previous emission
        foreach (var index in diff.Deletions)
        {
            if (index < previousDocumentIds.Count)
            {
                var deletedId = previousDocumentIds[index];
                Console.WriteLine($"Deleted car with ID: {deletedId}");
            }
        }

        // Handle insertions using current IDs
        foreach (var index in diff.Insertions)
        {
            var insertedId = currentDocumentIds[index];
            Console.WriteLine($"Inserted car with ID: {insertedId}");
        }

        // Handle updates using current IDs
        foreach (var index in diff.Updates)
        {
            var updatedId = currentDocumentIds[index];
            Console.WriteLine($"Updated car with ID: {updatedId}");
        }

        // Store only the document IDs for next callback - no live references!
        previousDocumentIds = currentDocumentIds;
    }
  });

DittoDiffer differ;
std::vector<std::string> previousDocumentIds; // Store only extracted IDs

auto observer = ditto.get_store().register_observer(
  "SELECT * FROM cars",
  [&](QueryResult result) {
      DittoDiff diff = differ.diff(result.items());

      // Extract current document IDs and dematerialize items
      std::vector<std::string> currentDocumentIds;
      for (auto& item : result.items()) {
          auto value = item.value();
          std::string id = value.contains("_id") ?
                          value["_id"].get<std::string>() : "unknown";
          currentDocumentIds.push_back(id);
          item.dematerialize(); // Release memory after extracting data
      }

      // Handle deletions using stored IDs from previous emission
      for (size_t index : diff.deletions()) {
          if (index < previousDocumentIds.size()) {
              std::string deletedId = previousDocumentIds[index];
              std::cout << "Deleted car with ID: " << deletedId << std::endl;
          }
      }

      // Handle insertions using current IDs
      for (size_t index : diff.insertions()) {
          std::string insertedId = currentDocumentIds[index];
          std::cout << "Inserted car with ID: " << insertedId << std::endl;
      }

      // Handle updates using current IDs
      for (size_t index : diff.updates()) {
          std::string updatedId = currentDocumentIds[index];
          std::cout << "Updated car with ID: " << updatedId << std::endl;
      }

      // Store only the document IDs for next callback - no live references!
      previousDocumentIds = currentDocumentIds;

      // C++ QueryResult uses RAII - destructor handles cleanup
  });

let mut differ = DittoDiffer::new();
let mut previous_document_ids: Vec<String> = Vec::new(); // Store only extracted IDs

let observer = ditto.store().register_observer_v2(
  "SELECT * FROM cars",
  move |result: QueryResult| {
      let diff = differ.diff(result.items());

      // Extract current document IDs and dematerialize items
      let mut current_document_ids: Vec<String> = Vec::new();
      for item in result.items() {
          let value = item.value();
          let id = value.get("_id")
              .and_then(|v| v.as_str())
              .unwrap_or("unknown")
              .to_string();
          current_document_ids.push(id);
          item.dematerialize(); // Release memory after extracting data
      }

      // Handle deletions using stored IDs from previous emission
      for &index in &diff.deletions {
          if let Some(deleted_id) = previous_document_ids.get(index) {
              println!("Deleted car with ID: {}", deleted_id);
          }
      }

      // Handle insertions using current IDs
      for &index in &diff.insertions {
          let inserted_id = &current_document_ids[index];
          println!("Inserted car with ID: {}", inserted_id);
      }

      // Handle updates using current IDs
      for &index in &diff.updates {
          let updated_id = &current_document_ids[index];
          println!("Updated car with ID: {}", updated_id);
      }

      // Store only the document IDs for next callback - no live references!
      previous_document_ids = current_document_ids;

      // Rust QueryResult uses RAII - Drop trait handles cleanup
  });

final differ = DittoDiffer();
final previousDocumentIds = <String>[]; // Store only extracted IDs

final observer = ditto.store.registerObserver(
  "SELECT * FROM cars",
  onChange: (queryResult) {
    final diff = differ.diff(queryResult.items);

    // Extract current document IDs
    // Note: Flutter SDK doesn't have explicit dematerialize
    final currentDocumentIds = queryResult.items.map((item) {
      final id = item.value['_id'] as String? ?? 'unknown';
      return id;
    }).toList();

    // Handle deletions using stored IDs from previous emission
    for (final index in diff.deletions) {
      if (index < previousDocumentIds.length) {
        final deletedId = previousDocumentIds[index];
        print('Deleted car with ID: $deletedId');
      }
    }

    // Handle insertions using current IDs
    for (final index in diff.insertions) {
      final insertedId = currentDocumentIds[index];
      print('Inserted car with ID: $insertedId');
    }

    // Handle updates using current IDs
    for (final index in diff.updates) {
      final updatedId = currentDocumentIds[index];
      print('Updated car with ID: $updatedId');
    }

    // Store only the document IDs for next callback - no live references!
    previousDocumentIds = currentDocumentIds;

    // Flutter handles cleanup automatically
  }
);

Implementation Details

Handling Deleted and Moved Documents

With store observers, results from the previous event are no longer provided directly. You need to:

Store the current results at the end of each callback:
```
previousItems = result.items;
```
Use the indexes provided in diff.deletions or diff.moves to access deleted and moved documents from your stored previousItems.

Detecting the Initial Event

The legacy API’s event.isInitial flag isn’t available with the Differ. To detect the first callback:

if (!previousItems) {
  // This is the initial callback
  // All items will appear as insertions in the diff
}

In the first callback, all items passed to differ.diff show up as insertions because the differ is initially empty.

Metadata Changes

The Differ ignores metadata changes when determining updates. For example:

Changing a document field and then setting it back to its original value will not be considered an update
This differs from the legacy API which would count any change as an update

Performance Considerations

Generating diffs may take significant time for very large result sets. Consider whether you truly need diff information for your use case.

Interactive Example

Try the interactive example on CodePen to see both approaches in action. Open your browser console to observe the results.

​Overview

​Key Differences

​Legacy Live Query API

​Store Observer with Differ

​Migration Example

​Before: Using Legacy Live Query API

​After: Using Store Observer with Differ

​Implementation Details

​Handling Deleted and Moved Documents

​Detecting the Initial Event

​Metadata Changes

​Performance Considerations

​Interactive Example

​Related Documentation

Overview

Key Differences

Legacy Live Query API

Store Observer with Differ

Migration Example

Before: Using Legacy Live Query API

After: Using Store Observer with Differ

Implementation Details

Handling Deleted and Moved Documents

Detecting the Initial Event

Metadata Changes

Performance Considerations

Interactive Example

Related Documentation