Datasets

A dataset is a typed table of rows. Reach for one whenever you have table-shaped data: raw rows pulled from an external system, cleaned rows produced by a pipeline, or rows a projection turns into objects.

A dataset is a contract: the table's name, its columns, each column's type, and which columns may be null. Define it once and syncs, pipelines, projections, and storage all point at the same shape.

If an ontology is your object model, a dataset is your table model.

Define a dataset

A dataset has a stable id and a schema of columns. Build each column with col(name, type).

import { col, defineDataset } from "@sixb/core"

export const rawInvoicesDataset = defineDataset("erp.invoices", {
  schema: [
    col("id", "string"),
    col("number", "string"),
    col("amount", "decimal"),
    col("currency", "string"),
    col("status", "string"),
    col("issuedAt", "timestamp"),
    col("dueDate", "date"),
    col("customer_id", "string"),
    col("project_id", "string"),
  ],
  primaryKey: "id",
  description: "Raw invoice rows from the ERP.",
})

The id erp.invoices is the name every other part of the project references.

defineDataset options

Primary keys

Use one non-nullable string column for a single-column key, or two or more for a composite key:

export const invoices = defineDataset("erp.invoices", {
  schema: [col("id", "string"), col("status", "string")],
  primaryKey: "id",
})

export const invoiceLines = defineDataset("erp.invoice_lines", {
  schema: [
    col("invoiceId", "string"),
    col("lineItemId", "string"),
    col("description", "string"),
  ],
  primaryKey: ["invoiceId", "lineItemId"],
})

V1 primary keys have these constraints:

• Every key column must exist in the schema, have type string, and be non-nullable.
• Composite keys contain at least two unique columns. Column order is significant.
• A key cannot be added, removed, changed, or reordered after the dataset is created.
• Rows must be unique by key, and a row's key is immutable. Merge-capable lake providers enforce uniqueness for keyed snapshots, appends, transforms, and merges.

Use a keyed dataset with a merge sync when the source exposes ordered row changes and the dataset should stay current without replacing every row on each run.

col options

col("amount", "decimal")
col("project_id", "string", { nullable: true })

Pass { nullable: true } when a column may be missing or null. Use the json type to keep an intentionally unstructured payload:

col("raw", "json", { nullable: true })

Column types

Decimal columns reject JavaScript numbers because their exact source value may already have lost precision. Keep decimals as strings from the source, or construct typed values with decimal("...") or decimal(anExactBigInt). Do not convert a number with String(...) and assume precision is restored.

Use a dataset

A dataset on its own is just a shape. The pieces that produce and consume rows reference it.

A sync reads from a connector and writes into one dataset. It does not repeat the schema; it points at the definition:

import { defineSync } from "@sixb/core"
import { acmeErpConnector } from "../connectors/acme-erp"
import { rawInvoicesDataset } from "../datasets/erp"

export const syncErpInvoices = defineSync("sync-erp-invoices")
  .from(acmeErpConnector)
  .read((client) => client.listInvoices())
  .intoDataset(rawInvoicesDataset)

A pipeline reads datasets and writes new ones, keeping raw source rows separate from clean, app-ready rows:

import type { DatasetRow } from "@sixb/core"
import {
  col,
  defineDataset,
  definePipeline,
  definePipelineStep,
  defineSchedule,
  events,
} from "@sixb/core"
import { rawInvoicesDataset } from "../datasets/erp"

export const paidInvoicesDataset = defineDataset("invoices.paid", {
  schema: [
    col("id", "string"),
    col("amount", "decimal"),
    col("currency", "string"),
    col("customer_id", "string"),
  ],
})

async function* paidInvoices(rows: AsyncIterable<DatasetRow>) {
  for await (const row of rows) {
    if (row.status === "paid") {
      yield {
        id: row.id,
        amount: row.amount,
        currency: row.currency,
        customer_id: row.customer_id,
      }
    }
  }
}

export const paidInvoicesStep = definePipelineStep("paid-invoices")
  .inputs({ invoices: rawInvoicesDataset })
  .output(paidInvoicesDataset)
  .run(async ({ inputs, output }) => {
    await output.writeRows(paidInvoices(inputs.invoices.readRows()))
  })

export const rawInvoicesUpdated = defineSchedule("raw-invoices-updated").on(
  events.dataset(rawInvoicesDataset).updated()
)

export const paidInvoicesPipeline = definePipeline("paid-invoices")
  .when(rawInvoicesUpdated)
  .then(paidInvoicesStep)

Then a projection maps the clean rows onto Invoice objects.

Derive a dataset

Use .derive(parent) to copy a parent's schema, then narrow it with pick or extend it with add.

import { col, defineDataset } from "@sixb/core"
import { rawInvoicesDataset } from "./erp"

// Copy the full parent schema
export const invoicesArchive = defineDataset("invoices.archive").derive(rawInvoicesDataset)

// Narrow to a subset of columns and add new ones
export const invoicesSummary = defineDataset("invoices.summary").derive(rawInvoicesDataset, {
  pick: ["id", "amount", "currency", "customer_id"],
  add: [col("settled_at", "timestamp")],
})

Derived datasets do not inherit their parent's primary key. Declare primaryKey explicitly when the derived rows preserve the same identity contract.

Dataset vs ontology

Datasets and ontology types solve different problems.

The usual flow: source data lands in a raw dataset, pipelines shape it into a clean dataset, and projections turn it into ontology objects.

Register datasets

Export dataset definitions from datasets/ and createSixb() discovers them automatically.

your-project/
  datasets/
    erp.ts
  syncs/
    erp.ts
  pipelines/
    invoices.ts
  sixb.config.ts

You can also register them explicitly:

import { createSixb } from "@sixb/core"
import { rawInvoicesDataset } from "./datasets/erp"

export const sixb = await createSixb({
  datasets: [rawInvoicesDataset],
})

Row validation

Every row written to a dataset is validated against its schema:

• A row must be a plain object and may not contain unknown columns.
• Each value must match its column's declared type.
• A non-nullable column may not be missing, undefined, or null.
• A nullable column may be omitted or set to null.

Modeling tips

• Start from the source shape. Keep source column names (account_mgr_id, dept_id) on raw datasets — it makes debugging easier.
• Mark uncertain fields nullable. Reserve json for payloads you intentionally keep unstructured.
• Name datasets by layer and source: erp.invoices, erp.customers for raw, invoices.paid for derived.
• Model money as amount + currency, never as a unit type.
• Shape clean datasets with pipelines rather than overloading the raw dataset.

Related

• Connectors — external systems datasets read from
• Syncs — pull rows into a dataset
• Pipelines — transform one dataset into another
• Projections — turn rows into ontology objects