I’ve asked an AI about my own documents plenty of times. Upload a PDF, ask a question, get a solid answer. Upload a related PDF next week and ask about that one, and it starts over: re-reading everything from scratch, as if the first PDF never existed. Do that fifty times over a month, feed it fifty related documents, and you get fifty disconnected answers, because nothing about document twelve ever gets folded into what it already worked out from document one.

That gap is what pulled me into the LLM Wiki pattern. Andrej Karpathy described it as a break from the usual upload-and-ask loop. Instead of an AI that re-derives an answer from raw documents every time you ask, what if it kept notes? What if it read something once, filed what it learned into a structured, cross-linked wiki, and started every future answer from there instead of from zero?

“LLMs don’t get bored, don’t forget to update a cross-reference, and can touch 15 files in one pass.”
– Andrej Karpathy, LLM Wiki

I wanted to see if the idea held up outside a whiteboard, so I built one, then pointed it at The Hound of the Baskervilles. It’s public domain, so no one’s copyrighted material ends up in a public write-up, and low stakes enough that I can find the rough edges without risking proprietary knowledge. It also happens to have a genuine plot twist, which made it a good stress test for whether the wiki could actually update its knowledge. And since it’s public domain, any reader can pull the same text and follow along.

None of that is really about a novel, though. Think about asking your own team’s agent why a specific requirement went into the spec six months ago, and getting an answer instead of a shrug. Or picture being three months into a new dean’s job, and a faculty member asks why the core curriculum requirement changed in 2019. The person who made that call is long gone. Ask around and you’ll get three different stories, because institutional memory drifts, and good luck finding the actual email or meeting summary from years ago to check. An agent that had actually ingested the committee minutes and budget memos as they happened, and revised its own notes each time a later document contradicted an earlier one, would hand you the version that’s closest to what was actually decided and why.

Retrieval works, but synthesis doesn’t stick

RAG (retrieval-augmented generation) is a very good open-book search. A question comes in, the system finds the most relevant chunks of text, hands them to the model, and the model writes an answer. It’s fast, it’s cheap per query, and it always reads current data, which is exactly what you want for a question like “what are the side effects of this medication?” You want the recent label, not a synthesis of every label this drug has ever had.

That same design, treating every query as independent, starts to cost you somewhere else. Query #1 and query #1,000 run through the identical loop: search, retrieve, generate, forget. Neither one leaves anything behind for the other to build on. That’s the right trade-off for fact lookup over a huge, constantly changing corpus (customer support tickets, legal filings, enterprise knowledge bases), where you want the current answer, not the history. But in a dynamic work environment, a project where each new document builds on the last, the connections between documents matter as much as the documents themselves. Re-deriving that whole picture from scratch every time someone asks means paying the same reasoning cost, and risking a slightly different answer, again and again. It starts to feel like reading the same five chapters of a novel every time you want to know what happens in chapter six.

Three layers, three jobs

What I ended up building keeps three things clearly separate:

• • Raw sources: the original material, untouched. The AI reads from here but never edits it.

• • The wiki: markdown pages the AI writes and owns: summaries, entity pages, concept pages, comparisons. This is the layer that gets richer every time something new comes in.

• • The schema: a document describing folder structure, naming conventions, and required metadata. This is what turns the AI into a disciplined maintainer instead of just another chatbot.

And three operations run against those layers:

• • Ingest: a new source arrives, the AI reads it, and updates or creates whatever wiki pages the new information touches. One source might touch ten or fifteen pages.

• • Query: instead of re-reading raw documents, the AI searches the wiki, reads the relevant pages, and answers from what’s already been synthesized.

• • Lint: a periodic pass that checks for contradictions between pages, stale claims, and pages nothing else links to.

In practice each of those is a named skill I invoke directly: /llm-wiki-ingest, /llm-wiki-query, /llm-wiki-lint, plus a fourth, /llm-wiki-maintain, for periodic health checks outside of an ingest session. Typing /llm-wiki-ingest chapter-08-first-report-of-dr-watson is the entire interface: the skill reads the schema, decides what pages a chapter touches, and only then reaches for the wiki_tool.py CLI underneath to actually index and validate the result.

None of this is complicated. The intelligence isn’t the trick here, the bookkeeping discipline is.

Chapter 7 says Barrymore. Chapter 13 says otherwise.

Here’s where the Hound test earned its keep. After chapter 1, the wiki had exactly what you’d expect: stub pages for Holmes and Watson, Baskerville Hall mentioned in passing. Nothing to connect yet.

Chapter 7 is where it commits to a suspicion. Sir Henry confronts his butler Barrymore about a telegram meant to confirm his alibi, and it unravels: the telegram went to Mrs. Barrymore, not to him, and he’d already lied to Henry’s face about hearing his wife sob at night. Ingesting the chapter didn’t just summarize the scene; it updated barrymore.md: “Telegram alibi not confirmed… Watson now holds open the possibility he was the London spy.” A hypothesis, built from evidence, filed on the page.

Chapter 8 makes it worse. Watson catches Barrymore at 2 a.m., signaling out at the moor with a candle, then snuffing it and creeping away. By this point the graph view alone tells the story: Barrymore sits at the center of a dense cluster of suspicion.

Chapter 13 is the reveal, and it doesn’t just correct Barrymore’s page; it goes to one nobody was watching. mr-stapleton.md, the mild-mannered naturalist next door, gets rewritten in place: “Major: Vandeleur alias added; St. Oliver’s school named; portrait = Hugo’s face; dictated the L.L. letter; Laura Lyons mechanism exposed.” the-ll-letter.md, tracked quietly since chapter 11, gets its own correction: “Author attribution revised: Lyons wrote it but Stapleton dictated it.” Same files, conclusions rewritten in place, not new pages arguing with old ones.

Ask the wiki afterward who the suspects are, and it answers cleanly from its own pages: Barrymore, cleared; Laura Lyons, cleared; Stapleton, the murderer. A RAG system asked the same question would retrieve chunks from chapters 7, 8, and 13 and reconcile that contradiction fresh, every single time someone asked. Here, the reconciliation happened once, at ingest.

The part that isn’t just prompting

It would be easy to read all of this as “give an LLM some folders and a good system prompt,” and there’s truth to that. What actually made it trustworthy enough to rely on were the deterministic pieces underneath the AI’s judgment calls.

I built a small CLI, wiki_tool.py, with commands for each job: build regenerates the wiki’s table of contents and search catalog from whatever pages exist. lint validates every page’s metadata and flags orphan pages. source-scan and source-lint hash every raw source file and flag it if a page cites a source that’s since changed, and search-catalog gives ingest and query a real ranked-retrieval layer over the compiled wiki instead of a full-text guess. A real run after an ingest looks like this:

$ python3 scripts/wiki_tool.py build && python3 scripts/wiki_tool.py lint | tail -4 && \
  python3 scripts/wiki_tool.py source-scan --update --accept-covered && python3 scripts/wiki_tool.py source-lint

Built - 23 concept(s), index.md and catalog.jsonl updated
Wiki/Locations/northumberland-hotel.md: orphan - no inbound [[wiki-links]] from other notes
Wiki/Locations/princetown-prison.md: orphan - no inbound [[wiki-links]] from other notes
Wiki/Locations/yew-alley.md: orphan - no inbound [[wiki-links]] from other notes
OK - 23 concept(s) valid
Manifest updated - 0 new, 0 changed, 15 total
OK - 15 source(s) checked

It’s a tool the AI is required to run, and it’s what actually catches drift. The orphan warnings above are a good example: three pages nothing links to yet, flagged automatically rather than discovered three months later when someone notices the page never comes up in a query.

The other piece I incorporated is OKF, the Open Knowledge Format that Google’s Data Cloud team published as a formal spec for exactly this kind of frontmatter: an attempt to take the informal LLM-wiki pattern and give it one agreed-upon shape any producer or agent can speak, so two teams can hand each other a wiki and trust the same tooling reads it.

Every page carries the same required fields (type, title, last-updated date) and the same recommended ones (a one-line description, the sources it draws from). Page types come from a small, deliberately restricted vocabulary. For the Hound wiki that’s character, location, artifact, event, concept: five types, chosen because the source material is a novel. A different corpus would earn a different set. The rule for adding a new type is strict: one page that doesn’t fit gets forced into the closest existing type; only after three pages don’t fit does a new type get added.

That discipline is what lets build and lint validate and index a page they’ve never seen before, without special-casing anything. Because the schema itself knows nothing about Sherlock Holmes, or about novels at all, that same code, and in principle any other agent or tool that speaks OKF, can validate a completely different project’s wiki without changing a line. A schema anyone can adopt is a wiki anyone else’s tooling can read.

That portability matters beyond one novel, because most of what an organization knows is scattered the same way: a metadata catalog, a wiki nobody fully trusts, a docstring three repos deep, a senior engineer’s head. A model can’t reason well over knowledge it has to guess the shape of. One consistent, machine-checkable shape everywhere is what turns “ask the one person who remembers” into something an agent, or a new hire, can actually look up.

Where this wins, and where it doesn’t

The honest answer to “wiki or RAG” is that they’re solving different problems, and the corpus-size framing people usually reach for is really a proxy for the real question: does the value come from finding one thing fast, or from connecting several things over time?

Reach for RAG when you have a huge, constantly changing corpus and need a citation to the exact chunk: customer support, legal search, enterprise fact lookup. Reach for a wiki when you have a bounded, curated set of sources (a few hundred, not a few million) and the valuable answers require connecting five of them, not looking up one: a research project, a book you’re studying, a course, your own notes over a year. And the two aren’t mutually exclusive: a wiki with hundreds of pages can bolt on a lookup tool for the long tail; it doesn’t have to be either/or.

There’s a cost-model way to see the same split. RAG pays a small, flat cost on every single query and never carries anything over to the next one, which is exactly right when the corpus keeps changing underneath you and yesterday’s synthesis might already be stale. A wiki pays a real cost up front, doing the reconciliation once at ingest, so every later query gets to skip straight to the already-corrected answer. That’s a good trade if you’ll ask the corpus the same kind of question a hundred times over months. It’s a bad one if you’ll only ever ask it once.

The trade-offs I didn’t expect going in

A few things became clear only after building it, and they’re worth saying plainly rather than glossing over:

• • It doesn’t scale past a point without a rethink. The index that lets the AI find the right pages has to fit in a context window. Comfortably, that’s on the order of a couple hundred pages before you need a retrieval layer anyway; at which point you’re building a hybrid, not a pure wiki.

• • It assumes one writer at a time. There’s no access control and no transaction model. Two people or two agents editing concurrently is a conflict, not a feature, and the design doesn’t pretend otherwise.

• • A wrong inference can get treated as settled. The Barrymore example worked because chapter 13 corrected the record. If a source is ambiguous and the AI guesses wrong, and lint doesn’t catch it, that guess sits in the wiki looking exactly as authoritative as everything else, until someone or something flags it. The immutable raw-source layer is the safety net; it’s not a substitute for actually checking.

• • The wiki is only as good as the model running it. All of the judgment calls (what’s worth a new page, when a contradiction is real versus just imprecise phrasing) sit with the AI. A weaker model propagates weaker judgment straight into the knowledge base.

None of that is a reason not to build one. It’s a reason to be honest about what it’s for: a curated, bounded, actively-maintained knowledge base, not a replacement for retrieval at scale.

Where I’d take this next

The natural next step is a team version: fed by Slack threads, meeting notes, or ticket exports instead of a novel, with the same ingest/query/lint discipline behind it. The other direction worth exploring is the hybrid: once a wiki’s index outgrows a context window, bolt a lightweight search layer on top of the curated base rather than replacing it. That gets you the compounding synthesis for the material you’ve actually curated, and retrieval for everything else.

For now, what I have is a much clearer sense of where this pattern is worth the maintenance and where it isn’t, the Sherlock Holmes wiki was just how I got there.