Architecture
llmwiki has two overlapping structures:
- The Karpathy three-layer wiki (conceptual):
raw/→wiki/→site/ - The eight-layer build (implementation): how responsibilities are distributed across Python modules, HTML templates, scripts, CI, etc.
This document covers both. For why the product is shaped this way (MCP for agents, site for humans, scriptable add, human candidate gate), see Product principles.
Layer 1: Karpathy's three-layer wiki
From the original LLM Wiki gist:
raw/ IMMUTABLE source documents
↓ (llmwiki converts .jsonl → .md here)
wiki/sources/ synthesized session summaries (LLM)
↓ (llmwiki synth — default also harvests candidates)
wiki/candidates/ PENDING review stubs for entities/concepts
↓ (agent /wiki-candidates → promote|merge|discard; site UI in #97)
wiki/entities/ concepts/ TRUSTED hubs (after review — never auto-promoted)
↓
site/ GENERATED static HTML
(llmwiki builds here via `llmwiki build` — not part of `synth`)
Trusted entity/concept hubs require human-or-agent review of candidates. Synthesis alone can leave Home looking “finished” (Raw → Synthesized) while the knowledge layer is still empty — Home’s Knowledge layer table (Candidates | Entities | Concepts) and Analytics Candidates to review make that backlog visible (#84).
raw/ — immutable layer
Everything under raw/ is treated as source-of-truth. The converter writes to it; nothing else should. If a source is wrong, fix the converter, not the output.
The converter writes one markdown file per session under raw/sessions/<project>/<date>-<slug>.md. Each file has YAML frontmatter (project, started, model, tools_used, gitBranch, etc.) and a Conversation body rendered turn-by-turn.
wiki/ — LLM-maintained layer
Your coding agent owns this layer entirely. It writes via the Ingest Workflow in CLAUDE.md:
wiki/
├── index.md catalog of all pages, updated on every ingest
├── log.md append-only chronological record
├── overview.md living synthesis across all sources
├── sources/ one summary page per raw source (kebab-case slug)
├── candidates/ pending entity/concept stubs (harvest; review before promote)
├── entities/ people, products, tools (TitleCase.md)
├── concepts/ ideas, frameworks, patterns (TitleCase.md)
├── projects/ codebases and work streams (kebab-case slug)
└── syntheses/ saved query answers (kebab-case slug)
Pages interlink via [[wikilinks]]. Contradictions are recorded, not silently overwritten. Pages compound over time — every new source makes the wiki richer.
site/ — generated static layer
llmwiki build reads raw/ (and wiki/ if populated) and renders a complete static HTML site. Nothing here is hand-maintained. Safe to delete and regenerate any time.
Layer 2: The eight-layer build
Internally the code is organised into eight functional layers. Each layer has one clear responsibility.
┌──────────────────────────────────────────────────────┐
│ L7 CI / ops .github/workflows/ │
│ L6 Adapters llmwiki/adapters/ │
│ L5 Schema / docs CLAUDE.md, AGENTS.md, docs/ │
│ L4 Distribution setup.sh, .bat, .claude/ │
│ L3 Viewer script.js in build.py │
│ L2 Site build.py (HTML + CSS) │
│ L1 Wiki CLAUDE.md workflows │
│ L0 Raw llmwiki/convert.py │
└──────────────────────────────────────────────────────┘
L0 — Raw
Owner: llmwiki/convert.py
Reads .jsonl from the agent's session store (via an adapter), filters out noise records, runs redaction, normalises the output into markdown, and writes to raw/sessions/.
Key properties:
- Idempotent — mtime tracked in
<vault>/llmwiki-state.json(unified queue + sync + synth + quarantine state) - Privacy-first — API keys + tokens + emails redacted by default; home-path usernames redacted when
redaction.redact_username: true - Live-session safe — skips files with a record younger than 60 minutes
- Agent-agnostic — delegates discovery to the adapter registry
Vault state (v1.4+): one active llmwiki-state.json per process, configured at the CLI border (apply_default_vault → configure_state_file). Library modules call resolve_state_file() — they never re-read config.json for the state path. Import-time constants like DEFAULT_STATE_FILE were removed so tests and library callers cannot accidentally write into a developer's configured vault.
L1 — Wiki
Owner: your coding agent, following CLAUDE.md / AGENTS.md
llmwiki does NOT write to wiki/ directly. The agent does, via slash commands (/wiki-ingest, /wiki-query, /wiki-lint) that execute the workflows in the schema file.
L2 — Site (HTML generator)
Owner: llmwiki/build.py
Converts every file under raw/sessions/ (and any hand-authored files under wiki/) into static HTML. Uses python-markdown (the only runtime dep) — syntax highlighting runs in the browser via highlight.js loaded from a pinned jsdelivr CDN (v0.5, #73), so the build pipeline itself stays stdlib-only. Writes to site/.
Pages rendered (v0.9 surface):
site/index.html— home with hero + 365-day activity heatmap + token-usage stat grid + recently-updated card + project grid with topic chipssite/projects/index.html— project grid with freshness badgessite/projects/<project>.html— per-project page with topics strip, 365-day heatmap (scoped), tool-calling bar chart, token timeline, main sessions + sub-agentssite/sessions/index.html— sortable sessions table with filter barsite/sessions/<project>/<slug>.html— per-session transcript with tool chart + token card + full conversationsite/models/index.html— sortable AI-model directory (v0.7, #55)site/models/<slug>.html— per-model info card + changelog timeline + pricing sparkline (v0.7, #56)site/index.html— pipeline State dashboard (recent raw docs)site/raw.html— Documents sidebar (logical-doc tree + quick filter) loaded fromdocuments-tree.jssite/documents/<path>.html— one reader page per logical document (#305): multi-part chunks assemble on the canonical URL; non-canonical…/<slug>-NN.htmlpaths are redirect stubs to#part-NN; shared tree sidebar loaded once via JSsite/documents-tree.json(+.jssidecar) — nested layout of the same logical-document leaves assearch-indextype:"document"entries (id / title / href parity)site/recent.html— newest raw documents, one row per logical documentsite/analytics.html— activity heatmap, token stats, project gridsite/search-index.json— pre-built client-side search index (onetype:"document"meta entry per logical doc)site/sources/<project>/<slug>.md— copies of raw session markdown for download / agents- Plus AI-consumable exports:
llms.txt,llms-full.txt,graph.jsonld,sitemap.xml,rss.xml
Documents enter raw/docs/ either via the asynchronous producer queue path or synchronously via llmwiki add (llmwiki/add_doc.py, #16 / #311) — each import writes one complete Markdown file per document. Synthesis chunks long bodies in memory to the active backend budget and stitches one wiki source page; length-driven -NN series on disk are a legacy layout. The site build (#305) still groups any remaining multi-piece raw files into logical documents for reading, search, and the sidebar without rewriting raw/ or wiki/; operators merge legacy pieces offline with llmwiki migrate whole-document-storage (no mandatory mass re-synth).
L3 — Viewer (browser JS)
Owner: script.js (a string constant inside build.py)
Everything that happens in the browser, in vanilla JS:
- Theme toggle with
data-themeattribute + localStorage + system preference - Reading progress bar (scroll-linked CSS)
- Copy-as-markdown + copy-code buttons (Clipboard API +
document.execCommandfallback for HTTP) - Auto-collapse of long tool-result sections into
<details> - Cmd+K command palette (fuzzy search over
search-index.json) - Keyboard shortcuts:
/,g h,g p,g s,j,k,? - Sessions-table filter bar (project, model, date range, slug text)
Zero dependencies. No bundler. No framework. One file.
L4 — Distribution
Owner: the installable llmwiki package
How users install and run llmwiki:
setup.sh/setup.bat— one-click install from a source checkoutsync.sh/sync.bat— wrappers aroundpython3 -m llmwiki syncbuild.sh/build.bat— wrappers aroundpython3 -m llmwiki buildupgrade.sh/upgrade.bat—git pull+ re-run setupllmwiki/agent_kit/— packaged/wiki-*slash commands and user skills;llmwiki install-agent-kit --dest PATHcopies them into an agent directory.claude/— contributor-only commands and skills (not shipped)llmwiki/mcp/— MCP server stub
L5 — Schema / docs
Owner: root-level markdown + docs/
Tells humans and agents how the system works:
CLAUDE.md— Claude Code schema with Ingest / Query / Lint workflowsAGENTS.md— Codex / OpenCode / Gemini mirror of the same.kiro/steering/— always-loaded contribution / format / verification rulesdocs/framework.md— Open Source Framework v4.1 adapted for llmwikidocs/research.md— Phase 1.25 research reportdocs/feature-matrix.md— 161 features across 16 categories
L6 — Adapters
Owner: llmwiki/adapters/
One file per agent. Each subclass of BaseAdapter does three things:
- Knows where the agent writes its session store
- Walks that store to discover
.jsonlfiles - Derives a friendly project slug from the path
Everything else (record parsing, filtering, redaction, rendering) is shared in convert.py.
L7 — CI / ops
Owner: .github/workflows/ + tests/
ci.yml— lint + tests + build smoke on every push + PR (Python 3.12 only)gitleaks.yml— secret scanpages.yml— build + deploy to GitHub Pages on tag push (Phase 6.5 Self-Demo)tests/fixtures/<agent>/— synthetic fixturestests/snapshots/<agent>/— expected markdown outputstests/test_*.py— pytest unit + snapshot tests
Adding an adapter
See framework.md §5.25 Adapter Flow for the full contract. TL;DR: one new file at llmwiki/adapters/<agent>.py, one fixture, one snapshot test, one doc page, one README line, one CHANGELOG entry.
Design principles
- Stdlib first. Runtime dep:
markdownonly. Nothing else. Syntax highlighting runs client-side via a CDN-loaded highlight.js (v0.5, #73) so the build stays deterministic and offline-capable. - Privacy by default. Redact everything sensitive before it hits disk.
- Idempotent everything. Re-running any command is safe and cheap.
- Localhost only. No network, no telemetry, no cloud. The user controls if/when to publish.
- One file per concern. build.py is one file, not a folder of templates. The whole HTML rendering lives there including CSS + JS.
- Agent-agnostic core.
convert.pydoesn't know which agent produced the .jsonl. Adapters translate.