docs: DataJoint 2.1 Documentation - PostgreSQL Multi-Backend Support

[dimitri-yatsenko]

Summary

This PR adds documentation for DataJoint 2.1 features, primarily PostgreSQL multi-backend support and updated tutorials that work identically on both MySQL and PostgreSQL.

Changes

New Documentation Files

• src/reference/specs/database-backends.md — Specification for supported database backends
  • Supported backends table (MySQL 8.0+, PostgreSQL 15+)
  • Adapter architecture diagram
  • Backend compatibility matrix
  • Type mapping reference (core types → native types)
  • Connection management patterns
  • Migration guidance

Updated Tutorials

All tutorials have been verified to work on both MySQL and PostgreSQL:

• src/tutorials/advanced/json-type.ipynb — Simplified to focus on storage patterns
  • Insert and fetch entire JSON objects (backend-agnostic)
  • Filtering in Python after fetching (no backend-specific SQL)
  • Clear guidelines for JSON vs normalized tables
  • Works identically on MySQL and PostgreSQL

Modified Reference Files

• src/about/versioning.md

  • Added v2.1 to version history table with "PostgreSQL multi-backend support"

• src/how-to/configure-database.md

  • Added database.backend setting to configuration table
  • Added new "PostgreSQL Backend" section with:
    • Configuration file examples
    • Environment variable setup (DJ_BACKEND)
    • Programmatic configuration
    • Docker Compose example for local development
    • Backend compatibility note

• src/how-to/migrate-to-v20.md

  • NEW: Added "Column Comment Format (Critical for Blob Migration)" section explaining:
    • The :<type>: comment format used by DataJoint 2.0
    • Why blob columns need :<blob>: markers (without them, blobs return raw bytes)
    • Usage examples for check_migration_status(), migrate_columns(), migrate_blob_columns()
  • UPDATED: Expanded "Option B: In-Place Migration" with step-by-step guide using actual datajoint.migrate functions:
    • backup_schema(), migrate_columns(), rebuild_lineage(), migrate_external(), verify_schema_v20()

• src/reference/specs/type-system.md

  • Added version indicator for PostgreSQL type mappings

• src/explanation/type-system.md

  • Added version indicator noting PostgreSQL support in 2.1

• src/reference/specs/index.md

  • Added Database Backends to Foundation section in reading order

• mkdocs.yaml

  • Added navigation entry for database-backends.md under Specifications > Foundation
  • Updated datajoint_version to "2.1" (indicates latest documented version)

Testing Infrastructure

• docker-compose.yaml — MySQL 8.0, PostgreSQL 15, MinIO containers for testing
• scripts/execute_notebooks.py — Notebook execution script with backend selection

Version Indicators

All PostgreSQL-related content includes proper version admonitions per about/versioning.md:

!!! version-added "New in 2.1"
    PostgreSQL is now supported as an alternative database backend.

Backend-Agnostic Design

The tutorials follow these principles for backend compatibility:

1. JSON fields: Insert/fetch entire objects; filter in Python (no SQL restrictions on JSON)
2. Standard SQL: Use only SQL features supported by both MySQL and PostgreSQL
3. Type definitions: Use DataJoint core types that map correctly to both backends

Related

• datajoint/datajoint-python PR #1339 (PostgreSQL implementation)

Test Plan

• Build docs locally and verify new pages render correctly
• Verify version admonitions display properly
• Check navigation links work
• Verify Docker Compose example syntax is correct
• Run all tutorials on MySQL backend
• Run all tutorials on PostgreSQL backend

---

🤖 Generated with Claude Code

[MilagrosMarin]

mkdocs.yaml:217 - Should datajoint_version be updated to "2.1" to reflect the PostgreSQL content in this PR?

extra:
  datajoint_version: "2.1"  # Updated from "2.0"

Or is this intentionally kept at "2.0" since that's the documentation baseline (per about/versioning.md)?

Just want to clarify the intent here.

[MilagrosMarin]

query-algebra.md Section 13.6 - The backend-specific SQL transpilation details (identifier quoting, HAVING alias handling, function translation) are thorough and valuable.

Question: Is this the intended location for these details, or would they fit better in database-backends.md alongside the other backend-specific behavior documentation?

Either location works—just checking if this was a deliberate choice to keep all SQL generation details together in the query-algebra spec.

[dimitri-yatsenko]

Re: mkdocs.yaml:217 (datajoint_version)

The datajoint_version: "2.0" setting indicates the documentation baseline, not the newest feature version. Per our versioning strategy, 2.0 is the baseline version where features are documented without markers. Features added in 2.1 (like PostgreSQL) are explicitly marked with !!! version-added "New in 2.1" admonitions throughout.

This approach lets users on 2.0 use the documentation while clearly seeing which features require upgrading to 2.1.

[dimitri-yatsenko]

Re: query-algebra.md Section 13.6 location

This was a deliberate choice following our Diataxis-based organization. Both files are specification pages (Reference category), but they serve different purposes:

query-algebra.md specifies the query algebra—how operators work and how they translate to SQL. Section 13.6 is integral to this spec because it documents how query expressions generate backend-specific SQL. This is relevant to developers understanding DataJoint's internals or debugging complex queries.

database-backends.md specifies backend configuration and user-facing compatibility patterns—what backends are supported, how to configure them, and how to write portable code.

The overlap in function translation is intentional: query-algebra.md documents the internal translation mechanism, while database-backends.md shows users how to leverage it. Users looking for "how do I write portable aggregations" go to database-backends.md; developers asking "how does DataJoint generate SQL for different backends" find it in the query algebra spec.

[dimitri-yatsenko]

Update: Changed datajoint_version to "2.1"

On reflection, the displayed version should indicate what the documentation covers, not the baseline. Since this PR documents 2.1 features (PostgreSQL), displaying "2.1" is more accurate for users.

The baseline concept (2.0 features unmarked, 2.1+ with admonitions) remains—it's about how features are marked, not what version to display.

I've updated mkdocs.yaml to set datajoint_version: "2.1".