Ingesting changes

Webhook handlers and other trusted backend code can update a registered dataset with a primary key. This example uses the people dataset:

import { change } from "@sixb/core"
import { people } from "../datasets/people"

await sixb.datasets.ingest(people, {
  changes: [
    change.upsert({ id: "42", name: "Sam", updatedAt: "2026-09-09T10:00:00.123Z" }),
    change.delete({ id: "9" }, { sequence: "2026-09-09T10:01:00.000Z" }),
  ],
})

ingest validates the changes and stores them. It returns once they are stored, before they are in the dataset. A worker then writes the changes waiting for the dataset as one new version, usually within a second, so many webhooks give a few versions instead of one each.

Use sequenceBy when webhooks and syncs share a dataset.

Existing unkeyed dataset? Create a new keyed dataset → backfill → repoint consumers. Stored primary keys and sequenceBy are immutable.

For bulk loads, use a sync: ingest keeps each call in the database until a worker writes it.

React to new data

Code that needs the ingested data does not wait for ingest. Trigger it from the new version instead, here a sync that reads the dataset:

import { defineSchedule, events } from "@sixb/core"
import { people } from "../datasets/people"

export const peopleUpdated = defineSchedule("people-updated").on(events.dataset(people).updated())

Attach it to the sync, pipeline or workflow with .when(peopleUpdated). It fires for versions that changed rows, and a burst of them joins one waiting run.

Objects and relationships

Ingestion updates source data. Existing pipelines and projections determine the object result:

Where it runs and what can fail

• The lake worker writes ingested changes. sixb dev and sixb worker-group include it; a deployment that names its worker types must list lake, and sixb deploy warns when it does not.
• Changes reach the lake with no orchestrator running. Keep it on all the same: it reports changes that wait, and asks again for a write that was lost when no other ingest call comes.
• A server embedded with createSixbServer has no lake worker, so ingest writes the changes before it returns. See Custom server setup.
• Console › Lake health shows the changes still waiting per dataset, and since when, and sixb check names the datasets whose changes nothing has written for 10 minutes.
• A worker that stops while writing resumes where it stopped, and never writes changes twice.