Data Architecture Overview

This page provides a high-level map of the entire data system powering the Longterm Wiki. The system is organized into three data layers, each with its own schema, storage, CLI tools, and automations. Data flows from external resources through verification into the wiki's core knowledge bases.

For the detailed naming guide and PG table reference, see Data Architecture: Three Bases and Naming GuideE1334Explains the wiki's three data layers (TableBase, FactBase, WikiBase), how they map to PG tables, and clarifies naming confusions around entities, things, and facts.. For the overall system architecture (Next.js, Crux CLI, deployment), see System ArchitectureE734Technical architecture of the Longterm Wiki — data pipeline, clever design patterns, and the ideas that make it workQuality: 73/100.

The Three Data Layers

flowchart TB
  subgraph L1["Resources Layer"]
      L1A["External content archive
citation_content,
resource_content_versions"]
      L1B["News collection
25 RSS/web sources"]
  end
  subgraph L2["Source-Check Layer"]
      L2A["Verification pipeline
claim extraction + LLM check"]
      L2B["Evidence + Verdicts
2 PG tables"]
  end
  subgraph L3["Three Bases Layer"]
      L3A["TableBase
entities, PG-primary tables"]
      L3B["FactBase
structured facts"]
      L3C["WikiBase
MDX prose pages"]
  end
  L1 -->|"cached content"| L2
  L2 -->|"verdicts inform"| L3
  L1 -->|"news triggers
page improvements"| L3
  L3 -.->|"claims to verify"| L2
  L2 -.->|"verdict confidence
filters news priority"| L1

Each layer is a self-contained data system with its own schema, storage tables, CLI commands, and health checks. The layers feed into each other: the Resources layer collects and archives external content, the Source-Check layer verifies claims against that content, and the Three Bases layer holds the wiki's own knowledge — informed by verification results.

Resources Layer — External Content

The Resources layer is the wiki's interface with the outside world. It collects, archives, and monitors external content — everything from RSS news items to cached web pages to tracked website snapshots. Other layers read from this archive instead of re-fetching live URLs.

What It Stores

News Collection (Auto-Update)

The auto-update system is the primary way new content enters the Resources layer. It scrapes 25 configured sources and routes news to wiki pages.

flowchart TB
  subgraph Sources["25 Configured Sources"]
      RSS["RSS/Atom Feeds"]
      WEB["Web Searches"]
  end
  subgraph Pipeline["Auto-Update Pipeline"]
      FETCH["Feed Fetcher"]
      DIGEST["Dedup + Digest"]
      ROUTER["Page Router"]
      FILTER["Source-Check Filter"]
  end
  subgraph Output["Outputs"]
      ARCHIVE["Resource Archive
content cached for
future verification"]
      IMPROVE["Page Improve Pipeline"]
  end
  RSS --> FETCH
  WEB --> FETCH
  FETCH --> DIGEST
  DIGEST --> ROUTER
  ROUTER --> FILTER
  FILTER --> ARCHIVE
  FILTER --> IMPROVE

CLI

# Auto-update (news collection)
pnpm crux w auto-update plan              # Preview what would be updated
pnpm crux w auto-update run --budget=30   # Execute with $30 budget cap
pnpm crux w auto-update digest            # Fetch and display news digest
pnpm crux w auto-update sources           # List configured sources
pnpm crux w auto-update history           # Show past runs

# Link health
pnpm crux w check-links                   # Check external URL health

Automation

• GitHub Actions: .github/workflows/auto-update.yml runs daily at 06:00 UTC
• State tracking: data/auto-update/state.yaml tracks last-seen items per source
• Watchlist: data/auto-update/watchlist.yaml identifies pages due for scheduled updates
• Dashboards: Auto-Update RunsE914History of news-driven auto-update runs, budget tracking, and source health0, Auto-Update NewsE915Browse news items found by the auto-update system and see how they were routed to wiki pages0, Data SourcesE2131Structured data sources for grants, personnel, and investments — tracked with versioned snapshots and freshness monitoring

Key Files

Source-Check Layer — Verification

The Source-Check layer verifies that claims in wiki pages and FactBase facts are actually supported by their cited sources. It reads cached content from the Resources layer (instead of re-fetching live URLs) and produces verdicts that inform the Three Bases layer.

How It Works

flowchart TB
  subgraph Input["What Gets Checked"]
      FB_FACTS["FactBase Facts"]
      WIKI_CLAIMS["Wiki Page Claims"]
      TB_RECORDS["TableBase Records
personnel, grants,
investments, etc."]
  end
  subgraph Archive["Resources Layer"]
      CACHED["Cached source content"]
  end
  subgraph Check["Verification Pipeline"]
      COLLECT["Item Collector"]
      VERIFY["LLM Verifier"]
  end
  subgraph Store["Storage"]
      EVIDENCE["source_check_evidence"]
      VERDICTS["source_check_verdicts"]
  end
  FB_FACTS --> COLLECT
  WIKI_CLAIMS --> COLLECT
  TB_RECORDS --> COLLECT
  CACHED --> VERIFY
  COLLECT --> VERIFY
  VERIFY --> EVIDENCE
  EVIDENCE --> VERDICTS

What It Stores

Verdict Categories

Each verdict carries a confidence score (0.0 to 1.0) and is timestamped so stale checks can be detected.

CLI

# FactBase facts
pnpm crux fb sourcing --entity=anthropic      # Check all facts for an entity
pnpm crux fb sourcing --fact=f_dW5cR9mJ8q     # Check a single fact

# Wiki page claims
pnpm crux w sourcing-wiki-pages --page=anthropic        # Check one page
pnpm crux w sourcing-wiki-pages --limit=5 --budget=2    # Batch with budget

Key Files

Three Bases Layer — The Wiki's Knowledge

The Three Bases layer is the wiki's own knowledge — the entities, facts, and prose pages that readers see. It's organized into three conceptual "Bases," each with its own source of truth, plus a set of PG-primary tables for high-volume relational data.

The Three Bases

Each base has:

• A source of truth (YAML or MDX files in git)
• A PG mirror (synced at build time for API queries)
• A build artifact (JSON file consumed by Next.js at build time)
• A CLI command group (for querying, validating, and enriching)

YAML-Primary vs. PG-Primary

Beyond the three bases, this layer also includes PG-primary tables — data that lives in PostgreSQL directly with no YAML backing. This is one of the most common points of confusion.

PG-primary tables include: personnel, grants, funding_rounds, investments, divisions, funding_programs, benchmarks, benchmark_results, jobs, research_areas, political_votes, political_scores, bluesky_posts, website_sources, prediction_market_questions.

Build Pipeline

The central transformation is apps/web/scripts/build-data.mjs, which runs 20 sequential phases to compile YAML + MDX + PG data into the JSON artifacts the frontend reads:

flowchart TB
  subgraph Input["Inputs"]
      YAML["YAML Entities"]
      FB_YAML2["FactBase YAML"]
      MDX2["MDX Pages"]
      API["Wiki-server API
facts, resources,
assessments"]
  end
  subgraph Phases["build-data.mjs (20 phases)"]
      direction TB
      P1["1-3. Load YAML, IDs, MDX"]
      P2["4-7. Derived, KB, pages, links"]
      P3["8-18. Risk, resources, refs,
coverage, rankings, etc."]
      P4["19-20. Transform + write"]
      P1 --> P2 --> P3 --> P4
  end
  subgraph Output["Outputs"]
      DB_JSON2["database.json"]
      FB_JSON2["factbase-data.json"]
      PG_SYNC["PostgreSQL sync"]
  end
  YAML --> P1
  FB_YAML2 --> P1
  MDX2 --> P1
  API --> P2
  P4 --> DB_JSON2
  P4 --> FB_JSON2
  P4 --> PG_SYNC

ID Schemes

Different parts of this layer use different ID formats. A single entity (e.g., Anthropic) might have all of these:

The entity_ids table and factbase-data.json's slugToEntityId mapping bridge between these ID systems.

Cross-Base Index: The things Table

The things table is a universal search index that gives every identifiable item in the system a single row — regardless of which base it comes from. This enables cross-domain search and a unified browse UI.

How Each Base Gets Populated

Most data entry is automated. The goal is a defensive pipeline: data enters through verification gates so records are born with green sourcing dots, rather than being verified after the fact. See Discussion #3958 for the full strategy.

WikiBase: Page Creation and Improvement

WikiBase has two pipeline engines:

Page creation (crux w create "Title" --tier=standard):

• Tiers: budget ($8-12), standard ($15-25), premium ($30-50)
• Multi-phase: web research via Firecrawl → LLM drafts page → adversarial review
• Auto-creates YAML entity stubs if the entity doesn't exist

Page improvement (crux w improve <id> --tier=standard --apply):

• Tiers: polish ($2-3, style only), standard ($5-8, light research), deep ($15-25, full research)
• V2 batch mode: crux w improve --engine=v2 --batch=anthropic,miri --apply
• Post-processing: citation audit, semantic diff safety check, auto-enrichment with FactBase references
• Semantic diff blocks changes that exceed tier scope (exit code 75)

Auto-update pipeline (daily, automated):

1. Fetches RSS/web sources → builds news digest
2. Routes news items to relevant pages (LLM-based matching)
3. Source-check filter prioritizes by verdict confidence
4. Runs improve pipeline on matched pages (budget-capped, default $50/run)

TableBase: Enrichment Loop

TableBase uses a scan → rank → agent loop to systematically fill gaps in PG-primary tables:

1. Scanner queries wiki-server for completeness per entity per table
2. Task ranker scores gaps: (100 - completeness) * taskWeight * importance
3. Agent runs web search + LLM to fill specific fields for one entity
4. Pre-submit verification checks each record against its source URL before writing — records are born with verdicts

pnpm crux tb tablebase scan        # Per-table completeness scores
pnpm crux tb tablebase gaps        # Ranked gap list
pnpm crux tb tablebase improve     # Fill one gap (~$0.50-1.50/task)
pnpm crux tb tablebase loop        # Autonomous loop with --budget

People discovery scans 5 data sources (experts, org keyPeople, FactBase, entity refs, paper authors) and creates entity stubs for frequently-mentioned people:

pnpm crux tb people discover       # Find new people candidates
pnpm crux tb people enrich --source=wikidata  # Add Wikidata facts

FactBase: Fact Entry

FactBase facts are currently the least automated — most enter via manual addition or Wikidata enrichment:

pnpm crux fb add-fact anthropic revenue 5e9 --asOf=2025-06 --source=URL
pnpm crux fb wikidata-enrich --entity=anthropic   # Import from Wikidata
pnpm crux fb show anthropic                        # View all facts
pnpm crux fb validate                              # 40+ validation rules

The improve pipeline extracts some facts as a side effect (wrapping claims in <FBFactValue> tags). See Known Gaps for the lack of automated fact discovery.

Naming Confusions

The same words mean different things in different contexts. This is the single biggest source of confusion when working in the codebase.

"Entity" Across Contexts

"Things" Across Contexts

These are completely unrelated despite sharing a name. The FactBase directory predates the PG table.

"Facts" Across Contexts

Health and Monitoring

Each layer has its own health infrastructure:

CI-Blocking Gate Checks

The gate (pnpm crux w validate gate --fix) runs ~50 checks. Most are blocking — 17 are advisory (warnings only). The blocking checks include:

• Unified content rules (13 rules in one pass): comparison-operators, dollar-signs, frontmatter-schema, wiki-id-integrity, prefer-entitylink, entitylink-ids, footnote-integrity, kbf-refs, no-deprecated-components, pipeline-artifacts, resource-ref-integrity, url-safety, no-quoted-subcategory
• Code quality: TypeScript type checks, .returning() guard, no untyped row casts, no console.log in server, prompt escaping, dangerous patterns, conflict markers
• Data integrity: YAML schema, FactBase stableId usage, KB schema, entity reference integrity, temporal invariants, controlled vocab, cross-base consistency
• Build: tests, build-data, MDX compilation smoke-test

How Data Flows End-to-End

Putting it all together — here's the auto-update path (the most common flow) from a real-world event to a reader seeing updated information. Other paths exist: TableBase enrichment (scan → rank → agent → pre-submit verify → PG), on-demand sourcing, and manual edits.

1. Resources: Auto-update pipeline fetches RSS feeds, discovers "Anthropic announces new model"
2. Resources: Source fetcher downloads the blog post content and caches it in resource_content_versions
3. Resources: Page router matches the news item to the Anthropic wiki page
4. Three Bases: The page improve pipeline updates the Anthropic MDX page with new information
5. Source-Check: Source-check reads the cached content, compares new claims against it, stores verdicts
6. Three Bases: build-data.mjs compiles updated MDX + YAML into database.json
7. Deploy: Next.js rebuilds, PostgreSQL synced, frontend serves updated page

Lifecycle of a Single Entity

To make the architecture concrete, here's how a single entity (Anthropic) exists across all three layers:

flowchart TB
  subgraph Resources["Resources Layer"]
      NEWS["Auto-update finds
new Anthropic blog post"]
      CACHED["Blog post cached in
resource_content_versions"]
  end
  subgraph Sourcing["Source-Check Layer"]
      CHECK["Verifies revenue claim
against cached Reuters article"]
  end
  subgraph ThreeBases["Three Bases Layer"]
      YAML_E["TableBase: entities/
organizations.yaml"]
      FB_E["FactBase: things/
anthropic.yaml"]
      MDX_E["WikiBase: content/docs/
anthropic.mdx"]
      BUILD_E["build-data.mjs →
database.json → /wiki/E22"]
  end
  NEWS --> CACHED
  CACHED --> CHECK
  NEWS -->|"triggers improve
pipeline for"| MDX_E
  CHECK -->|"verdicts for"| FB_E
  YAML_E --> BUILD_E
  FB_E --> BUILD_E
  MDX_E --> BUILD_E

IDs for Anthropic across systems:

What Queries What (Runtime vs. Build-Time)

This is a critical distinction: content pages make zero runtime API calls, while internal dashboards call the wiki-server API at request time.

Key implication: A wiki-server outage does NOT break the public-facing wiki. Content pages are fully static. Only internal dashboards and CLI tools are affected.

ISR Caching

Internal dashboard pages use Next.js Incremental Static Regeneration. Most pages cache for 300 seconds (5 minutes); data-source pages cache for 60 seconds. Pages that fail to fetch from the wiki-server fall back to local static files via withApiFallback.

Common Tasks Cheat Sheet

PG Table Relationship Map

The entities table is the hub — most PG-primary tables reference it via stable_id. Simplified view of major foreign key relationships:

flowchart TB
  ENTITIES["entities
(stable_id)"]
  FACTS["facts"]
  WIKI["wiki_pages"]
  THINGS["things"]
  PERSONNEL["personnel"]
  GRANTS["grants"]
  FUNDING_R["funding_rounds"]
  INVESTMENTS["investments"]
  DIVISIONS["divisions"]
  BENCHMARKS["benchmark_results"]
  RESOURCES["resources"]
  RES_CIT["resource_citations"]
  PAGE_LINKS["page_links"]
  SOURCE_CHK["source_check_evidence"]
  EDIT_LOGS["edit_logs"]
  FACTS -->|"entity_id"| ENTITIES
  PERSONNEL -->|"person + org
entity_id"| ENTITIES
  GRANTS -->|"org + grantee
entity_id"| ENTITIES
  FUNDING_R -->|"company
entity_id"| ENTITIES
  INVESTMENTS -->|"company + investor
entity_id"| ENTITIES
  DIVISIONS -->|"parent_org_id"| ENTITIES
  BENCHMARKS -->|"model_id"| ENTITIES
  RES_CIT -->|"page_id"| WIKI
  RES_CIT -->|"resource_id"| RESOURCES
  PAGE_LINKS -->|"source + target"| WIKI
  EDIT_LOGS -->|"page_id"| WIKI
  SOURCE_CHK -->|"entity_id"| ENTITIES
  THINGS -->|"parent_thing_id"| THINGS

Key patterns:

• entities.stable_id is the universal join key — personnel, grants, funding rounds, investments, divisions, benchmark results, and facts all FK to it
• wiki_pages.id is the hub for content relationships — resource citations, page links, edit logs, hallucination risk snapshots all FK to it
• things is self-referential (parent_thing_id) and indexes all other tables via source_table + source_id
• resources connects to wiki pages via resource_citations and to facts via source URLs

CLI Command Map

The Crux CLI is organized into domain groups. Here's the full tree:

flowchart TB
  CRUX["pnpm crux"]
  W["w (wiki)"]
  FB["fb (factbase)"]
  TB["tb (tablebase)"]
  GH["gh (github)"]
  SYS["sys (system)"]
  QUERY["query"]
  CTX["context"]
  CRUX --> W
  CRUX --> FB
  CRUX --> TB
  CRUX --> GH
  CRUX --> SYS
  CRUX --> QUERY
  CRUX --> CTX

What's Not Here (Known Gaps)

The architecture has known limitations and missing pieces. Documenting them prevents wasted effort trying to find features that don't exist.

Comparison to Conventional Architectures

For contributors coming from other systems, here's how this architecture maps to more common patterns:

The unusual parts:

• YAML as source of truth instead of a database — enables git review workflows but makes queries harder
• Three separate data models (TableBase, FactBase, WikiBase) instead of one unified schema — each optimized for its domain but creating naming confusion
• Build-time data compilation into a single JSON file — enables zero-API content pages but requires a full rebuild for any data change
• LLM-powered automation at every layer — content creation, fact verification, data enrichment, news routing all use LLM calls

Related Documents

• Data Architecture: Three Bases and Naming GuideE1334Explains the wiki's three data layers (TableBase, FactBase, WikiBase), how they map to PG tables, and clarifies naming confusions around entities, things, and facts. — Detailed PG table reference and naming confusions
• System ArchitectureE734Technical architecture of the Longterm Wiki — data pipeline, clever design patterns, and the ideas that make it workQuality: 73/100 — Overall technical architecture
• Knowledge Base ArchitectureE904Architecture document for the KB system — YAML-based structured facts, property taxonomy, record collections, and how they relate to the wiki's data pipeline.Quality: 50/100 — Deep dive into FactBase internals
• DB Schema OverviewE1054Complete reference for the wiki-server PostgreSQL schema — tables, relationships, ER diagrams, and migration history0 — Full ER diagrams and migration history
• Data System Authority RulesE1018Defines which data system is authoritative for each entity type, resolving overlaps between KB YAML and old YAML facts.Quality: 47/100 — Which system is authoritative for each entity