FormBD

image:[License,link="https://github.com/hyperpolymath/palimpsest-license"]

The database where the database is part of the story.

Overview

FormBD is a narrative-first, reversible, audit-grade database core. It treats schemas, constraints, migrations, blocks, and journals as narrative artefacts—the database is part of the story, not an opaque substrate.

Core Thesis

Schemas, constraints, migrations, blocks, and journals are narrative artefacts. The database is part of the story, not an opaque substrate.

Primary Values

Principle	Priority
Auditability	> Performance
Meaning	> Features
Reversibility	> Throughput
Agent Understanding	Required

Target Domains

• Investigative journalism

• Governance/compliance

• Agentic ecosystems + multi-repo handover

• Long-term cultural/institutional archives

Architecture

FormBD is structured in layers, each with a specific language and responsibility:

┌─────────────────────────────────────────────────────────────┐
│  Form.ControlPlane (Elixir/OTP - optional)                  │
│  Sessions, supervision, cluster edge                        │
├─────────────────────────────────────────────────────────────┤
│  Form.Normalizer (Factor + Lean 4)                          │
│  FD discovery, type encoding, proof-carrying normalization  │
├─────────────────────────────────────────────────────────────┤
│  Form.Runtime (Factor)                                      │
│  FQL parse/plan/exec, explain, introspection                │
├─────────────────────────────────────────────────────────────┤
│  Form.Bridge (Zig)                                          │
│  Stable Zig ABI (no C), safety governor                     │
├─────────────────────────────────────────────────────────────┤
│  Form.Model (Forth)                                         │
│  Multi-model logical layer: documents, edges, schemas       │
├─────────────────────────────────────────────────────────────┤
│  Form.Blocks (Forth)                                        │
│  Deterministic storage, journal, reversibility primitives   │
└─────────────────────────────────────────────────────────────┘

Layer Details

Form.Blocks (Forth)

• Fixed-size blocks with symbolic headers

• Append-only journal

• Crash recovery and integrity checks

• Repair guidance

Form.Model (Forth)

• Document collections

• Edge collections

• Schema + constraint metadata

• Migration artefacts

Form.Bridge (Zig)

• Stable Zig ABI for runtimes (no C dependency)

• Opaque handles + byte buffers + explicit error codes

• Marshalling only; no business logic duplication

• Uses callconv(.C) for FFI compatibility without requiring C toolchain

Form.Runtime (Factor)

• FQL minimal subset for PoC

• Planner steps introspection

• Constraint explanation surfaces

• Provenance surfaces

Form.Normalizer (Factor + Lean 4)

• Automatic functional dependency discovery (DFD/TANE/FDHits)

• Type encoding of FDs in FQL-dt

• Normal form predicates (1NF through BCNF)

• Proof-carrying schema evolution

• Narrative explanations for all normalization decisions

• See Self-Normalizing Specification

Form.ControlPlane (Elixir/OTP, optional)

• Out-of-process core engine via port

• Sessions and supervision

• Cluster edge coordination

Core Invariants

These are non-negotiable guarantees:

Truth Ownership	On-disk truth is owned by the block/journal layer; higher layers do not bypass it.
Journal-First	Every mutating operation is journaled before being considered committed.
Reversibility	Every committed operation MUST have a defined inverse OR be explicitly marked irreversible-with-story.
Renderability	Blocks and journal entries MUST be renderable deterministically into human/agent-readable form.
Provenance	All query results can optionally include provenance pointers to journal and blocks.
Constraints as Ethics	Constraints are explainable: rejections must return reasons + pointers + narrative rationale.

FQL (FormBD Query Language)

FQL is the query language for FormBD—not a "forms" query language, but the native query interface for this narrative-first database.

PoC subset capabilities:

• INSERT document into collection

• INSERT edge (from, to, type, props)

• SELECT with simple predicates

• EXPLAIN (returns plan + reasons)

• INTROSPECT schema/constraints

• Optional provenance output

Non-Goals

• Be a drop-in Postgres replacement

• Win microbenchmarks

• Ship full distributed consensus in the first PoC

Repository Structure

formbd/
├── spec/              # Format specs + rationale (AsciiDoc)
├── core-forth/        # Form.Blocks + Form.Model (truth core)
├── core-zig/          # Form.Bridge (ABI + port framing)
├── core-factor/       # Form.Runtime (FQL + introspection)
├── control-plane/     # Elixir/OTP gateway (optional)
├── tools/             # Render/inspect/doctor utilities
├── test-vectors/      # Golden bytes + golden renders
└── stories/           # Narrative examples + onboarding artefacts

PoC Acceptance Criteria

• Single-node db open/close

• Append-only journal with deterministic rendering

• Document + edge insert/select

• Constraint rejection returns explain payload

• Migration artefact recorded + reversible

• Golden test vectors pass

• Seam checks (B↔M, M↔R, B↔R) pass at freeze

Open Questions

See formbd.scm for structured open questions with acceptance criteria:

Core PoC

• Q-BLOCK-HEADER-001: Minimal block header layout

• Q-JOURNAL-ENTRY-001: Minimal journal entry schema

• Q-ABI-BLOBS-001: ABI blob encoding choice

• Q-FQL-POC-001: FQL PoC grammar

• Q-CTRL-PLANE-001: Elixir/OTP introduction timing

Self-Normalizing (Form.Normalizer)

• Q-NORM-001: Default FD discovery algorithm (DFD vs TANE vs FDHits)

• Q-NORM-002: Approximate FD handling policy

• Q-NORM-003: Denormalization support scope

• Q-NORM-004: FQL-dt proof integration interface

• Q-NORM-005: Query rewriting for affected queries

Documentation

Getting Started

• Quickstart Guide - Get running in 5 minutes

• Versioning & Stability - API stability guarantees

• Changelog - Release history

Architecture & Design

• Architecture - Technical architecture deep-dive

• Roadmap - Development roadmap

• Philosophy - Design philosophy and principles

• formbd.scm - Machine-readable specification

Specifications

• FQL Specification - Query language reference

• FQL Dependent Types - Compile-time verified queries

• Self-Normalizing Database - Automatic FD discovery, proof-carrying schema evolution

• Block Format - Storage layer specification

• Journal Format - Audit log specification

Operations

• Deployment Guide - Run FormBD in production

• Observability - Monitoring, logging, tracing

• Security & Auth - Authentication and hardening

Integration

• API Reference - Programmatic interfaces

• Migration from RDBMS - Moving from PostgreSQL/MySQL

• Integration Patterns - Queues, search, analytics

Domain-Specific

• WP05 - FormBD for Interactive Documentary/Journalism

Related Projects

• FQLdt - Dependently-typed FQL (same database, compile-time proofs)

• FormBD Studio - Zero-friction GUI for non-technical users

• BoFIG - Evidence graph for investigative journalism

• Zotero-FormBD - Post-truth reference manager with PROMPT scores

• FormBD Debugger - Proof-carrying database debugger (Lean 4 + Idris 2)

• FormBase - Open-source Airtable alternative with provenance

License

Palimpsest-MPL 1.0 (PMPL-1.0)

Contributing

See CONTRIBUTING.adoc for guidelines.