Architecture

llmwiki has two overlapping structures:

  1. The Karpathy three-layer wiki (conceptual): raw/ → wiki/ → site/
  2. The eight-layer build (implementation): how responsibilities are distributed across Python modules, HTML templates, scripts, CI, etc.

This document covers both. For why the product is shaped this way (MCP for agents, site for humans, scriptable add, human candidate gate), see Product principles.

Layer 1: Karpathy's three-layer wiki

From the original LLM Wiki gist:

raw/           IMMUTABLE source documents
    ↓          (llmwiki converts .jsonl → .md here)
wiki/sources/  synthesized session summaries (LLM)
    ↓          (llmwiki synth — default also harvests candidates)
wiki/candidates/  PENDING review stubs for entities/concepts
    ↓          (agent /wiki-candidates → promote|merge|discard; site UI in #97)
wiki/entities/ concepts/   TRUSTED hubs (after review — never auto-promoted)
    ↓
site/          GENERATED static HTML
               (llmwiki builds here via `llmwiki build` — not part of `synth`)

Trusted entity/concept hubs require human-or-agent review of candidates. Synthesis alone can leave Home looking “finished” (Raw → Synthesized) while the knowledge layer is still empty — Home’s Knowledge layer table (Candidates | Entities | Concepts) and Analytics Candidates to review make that backlog visible (#84).

raw/ — immutable layer

Everything under raw/ is treated as source-of-truth. The converter writes to it; nothing else should. If a source is wrong, fix the converter, not the output.

The converter writes one markdown file per session under raw/sessions/<project>/<date>-<slug>.md. Each file has YAML frontmatter (project, started, model, tools_used, gitBranch, etc.) and a Conversation body rendered turn-by-turn.

wiki/ — LLM-maintained layer

Your coding agent owns this layer entirely. It writes via the Ingest Workflow in CLAUDE.md:

wiki/
├── index.md          catalog of all pages, updated on every ingest
├── log.md            append-only chronological record
├── overview.md       living synthesis across all sources
├── sources/          one summary page per raw source (kebab-case slug)
├── candidates/       pending entity/concept stubs (harvest; review before promote)
├── entities/         people, products, tools (TitleCase.md)
├── concepts/         ideas, frameworks, patterns (TitleCase.md)
├── projects/         codebases and work streams (kebab-case slug)
└── syntheses/        saved query answers (kebab-case slug)

Pages interlink via [[wikilinks]]. Contradictions are recorded, not silently overwritten. Pages compound over time — every new source makes the wiki richer.

site/ — generated static layer

llmwiki build reads raw/ (and wiki/ if populated) and renders a complete static HTML site. Nothing here is hand-maintained. Safe to delete and regenerate any time.

Layer 2: The eight-layer build

Internally the code is organised into eight functional layers. Each layer has one clear responsibility.

┌──────────────────────────────────────────────────────┐
│  L7  CI / ops          .github/workflows/            │
│  L6  Adapters          llmwiki/adapters/             │
│  L5  Schema / docs     CLAUDE.md, AGENTS.md, docs/   │
│  L4  Distribution      setup.sh, .bat, .claude/      │
│  L3  Viewer            script.js in build.py         │
│  L2  Site              build.py (HTML + CSS)         │
│  L1  Wiki              CLAUDE.md workflows           │
│  L0  Raw               llmwiki/convert.py            │
└──────────────────────────────────────────────────────┘

L0 — Raw

Owner: llmwiki/convert.py

Reads .jsonl from the agent's session store (via an adapter), filters out noise records, runs redaction, normalises the output into markdown, and writes to raw/sessions/.

Key properties:

  • Idempotent — mtime tracked in <vault>/llmwiki-state.json (unified queue + sync + synth + quarantine state)
  • Privacy-first — API keys + tokens + emails redacted by default; home-path usernames redacted when redaction.redact_username: true
  • Live-session safe — skips files with a record younger than 60 minutes
  • Agent-agnostic — delegates discovery to the adapter registry

Vault state (v1.4+): one active llmwiki-state.json per process, configured at the CLI border (apply_default_vault → configure_state_file). Library modules call resolve_state_file() — they never re-read config.json for the state path. Import-time constants like DEFAULT_STATE_FILE were removed so tests and library callers cannot accidentally write into a developer's configured vault.

L1 — Wiki

Owner: your coding agent, following CLAUDE.md / AGENTS.md

llmwiki does NOT write to wiki/ directly. The agent does, via slash commands (/wiki-ingest, /wiki-query, /wiki-lint) that execute the workflows in the schema file.

L2 — Site (HTML generator)

Owner: llmwiki/build.py

Converts every file under raw/sessions/ (and any hand-authored files under wiki/) into static HTML. Uses python-markdown (the only runtime dep) — syntax highlighting runs in the browser via highlight.js loaded from a pinned jsdelivr CDN (v0.5, #73), so the build pipeline itself stays stdlib-only. Writes to site/.

Pages rendered (v0.9 surface):

  • site/index.html — home with hero + 365-day activity heatmap + token-usage stat grid + recently-updated card + project grid with topic chips
  • site/projects/index.html — project grid with freshness badges
  • site/projects/<project>.html — per-project page with topics strip, 365-day heatmap (scoped), tool-calling bar chart, token timeline, main sessions + sub-agents
  • site/sessions/index.html — sortable sessions table with filter bar
  • site/sessions/<project>/<slug>.html — per-session transcript with tool chart + token card + full conversation
  • site/models/index.html — sortable AI-model directory (v0.7, #55)
  • site/models/<slug>.html — per-model info card + changelog timeline + pricing sparkline (v0.7, #56)
  • site/index.html — pipeline State dashboard (recent raw docs)
  • site/raw.html — raw-docs file-tree browser (tree loaded from documents-tree.js)
  • site/documents/<path>.html — one page per raw document file; shared tree sidebar loaded once via JS
  • site/documents-tree.json (+ .js sidecar) — single shared doctree payload for Raw + document pages
  • site/recent.html — newest raw documents, one row per logical document
  • site/analytics.html — activity heatmap, token stats, project grid
  • site/search-index.json — pre-built client-side search index
  • site/sources/<project>/<slug>.md — copies of raw session markdown for download / agents
  • Plus AI-consumable exports: llms.txt, llms-full.txt, graph.jsonld, sitemap.xml, rss.xml

Documents enter raw/docs/ either via the asynchronous producer queue path or synchronously via llmwiki add (llmwiki/add_doc.py, #16) — both produce the same dir-per-doc, section-chunked layout.

L3 — Viewer (browser JS)

Owner: script.js (a string constant inside build.py)

Everything that happens in the browser, in vanilla JS:

  • Theme toggle with data-theme attribute + localStorage + system preference
  • Reading progress bar (scroll-linked CSS)
  • Copy-as-markdown + copy-code buttons (Clipboard API + document.execCommand fallback for HTTP)
  • Auto-collapse of long tool-result sections into <details>
  • Cmd+K command palette (fuzzy search over search-index.json)
  • Keyboard shortcuts: /, g h, g p, g s, j, k, ?
  • Sessions-table filter bar (project, model, date range, slug text)

Zero dependencies. No bundler. No framework. One file.

L4 — Distribution

Owner: the installable llmwiki package

How users install and run llmwiki:

  • setup.sh / setup.bat — one-click install from a source checkout
  • sync.sh / sync.bat — wrappers around python3 -m llmwiki sync
  • build.sh / build.bat — wrappers around python3 -m llmwiki build
  • upgrade.sh / upgrade.bat — git pull + re-run setup
  • llmwiki/agent_kit/ — packaged /wiki-* slash commands and user skills; llmwiki install-agent-kit --dest PATH copies them into an agent directory
  • .claude/ — contributor-only commands and skills (not shipped)
  • llmwiki/mcp/ — MCP server stub

L5 — Schema / docs

Owner: root-level markdown + docs/

Tells humans and agents how the system works:

  • CLAUDE.md — Claude Code schema with Ingest / Query / Lint workflows
  • AGENTS.md — Codex / OpenCode / Gemini mirror of the same
  • .kiro/steering/ — always-loaded contribution / format / verification rules
  • docs/framework.md — Open Source Framework v4.1 adapted for llmwiki
  • docs/research.md — Phase 1.25 research report
  • docs/feature-matrix.md — 161 features across 16 categories

L6 — Adapters

Owner: llmwiki/adapters/

One file per agent. Each subclass of BaseAdapter does three things:

  1. Knows where the agent writes its session store
  2. Walks that store to discover .jsonl files
  3. Derives a friendly project slug from the path

Everything else (record parsing, filtering, redaction, rendering) is shared in convert.py.

L7 — CI / ops

Owner: .github/workflows/ + tests/

  • ci.yml — lint + tests + build smoke on every push + PR (Python 3.12 only)
  • gitleaks.yml — secret scan
  • pages.yml — build + deploy to GitHub Pages on tag push (Phase 6.5 Self-Demo)
  • tests/fixtures/<agent>/ — synthetic fixtures
  • tests/snapshots/<agent>/ — expected markdown outputs
  • tests/test_*.py — pytest unit + snapshot tests

Adding an adapter

See framework.md §5.25 Adapter Flow for the full contract. TL;DR: one new file at llmwiki/adapters/<agent>.py, one fixture, one snapshot test, one doc page, one README line, one CHANGELOG entry.

Design principles

  1. Stdlib first. Runtime dep: markdown only. Nothing else. Syntax highlighting runs client-side via a CDN-loaded highlight.js (v0.5, #73) so the build stays deterministic and offline-capable.
  2. Privacy by default. Redact everything sensitive before it hits disk.
  3. Idempotent everything. Re-running any command is safe and cheap.
  4. Localhost only. No network, no telemetry, no cloud. The user controls if/when to publish.
  5. One file per concern. build.py is one file, not a folder of templates. The whole HTML rendering lives there including CSS + JS.
  6. Agent-agnostic core. convert.py doesn't know which agent produced the .jsonl. Adapters translate.

Keyboard shortcuts

⌘K / Ctrl+KOpen command palette
/Focus search
g hGo to home
g pGo to projects
g sGo to sessions
j / kNext / prev row (tables)
?Show this help
EscClose dialogs

Structured queries

Mix key:value filters with free text in the palette:

type:sessionOnly session pages
project:llm-wikiFilter by project name (substring)
model:claudeFilter by model name (substring)
date:>2026-03-01Sessions after a date
date:<2026-04-01Sessions before a date
tags:rustPages mentioning a tag/topic
sort:dateSort results by date (newest first)

Example: type:session project:llm-wiki date:>2026-04 sort:date