用对象存储部署 AI 友好的个人知识库

# LLM Wiki

A pattern for building personal knowledge bases using LLMs.

This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.

## The core idea

Most people's experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM is rediscovering knowledge from scratch on every question. There's no accumulation. Ask a subtle question that requires synthesizing five documents, and the LLM has to find and piece together the relevant fragments every time. Nothing is built up. NotebookLM, ChatGPT file uploads, and most RAG systems work this way.

The idea here is different. Instead of just retrieving from raw documents at query time, the LLM **incrementally builds and maintains a persistent wiki** — a structured, interlinked collection of markdown files that sits between you and the raw sources. When you add a new source, the LLM doesn't just index it for later retrieval. It reads it, extracts the key information, and integrates it into the existing wiki — updating entity pages, revising topic summaries, noting where new data contradicts old claims, strengthening or challenging the evolving synthesis. The knowledge is compiled once and then *kept current*, not re-derived on every query.

This is the key difference: **the wiki is a persistent, compounding artifact.** The cross-references are already there. The contradictions have already been flagged. The synthesis already reflects everything you've read. The wiki keeps getting richer with every source you add and every question you ask.

You never (or rarely) write the wiki yourself — the LLM writes and maintains all of it. You're in charge of sourcing, exploration, and asking the right questions. The LLM does all the grunt work — the summarizing, cross-referencing, filing, and bookkeeping that makes a knowledge base actually useful over time. In practice, I have the LLM agent open on one side and Obsidian open on the other. The LLM makes edits based on our conversation, and I browse the results in real time — following links, checking the graph view, reading the updated pages. Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase.

This can apply to a lot of different contexts. A few examples:

- **Personal**: tracking your own goals, health, psychology, self-improvement — filing journal entries, articles, podcast notes, and building up a structured picture of yourself over time.
- **Research**: going deep on a topic over weeks or months — reading papers, articles, reports, and incrementally building a comprehensive wiki with an evolving thesis.
- **Reading a book**: filing each chapter as you go, building out pages for characters, themes, plot threads, and how they connect. By the end you have a rich companion wiki. Think of fan wikis like [Tolkien Gateway](https://tolkiengateway.net/wiki/Main_Page) — thousands of interlinked pages covering characters, places, events, languages, built by a community of volunteers over years. You could build something like that personally as you read, with the LLM doing all the cross-referencing and maintenance.
- **Business/team**: an internal wiki maintained by LLMs, fed by Slack threads, meeting transcripts, project documents, customer calls. Possibly with humans in the loop reviewing updates. The wiki stays current because the LLM does the maintenance that no one on the team wants to do.
- **Competitive analysis, due diligence, trip planning, course notes, hobby deep-dives** — anything where you're accumulating knowledge over time and want it organized rather than scattered.

## Architecture

There are three layers:

**Raw sources** — your curated collection of source documents. Articles, papers, images, data files. These are immutable — the LLM reads from them but never modifies them. This is your source of truth.

**The wiki** — a directory of LLM-generated markdown files. Summaries, entity pages, concept pages, comparisons, an overview, a synthesis. The LLM owns this layer entirely. It creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. You read it; the LLM writes it.

**The schema** — a document (e.g. CLAUDE.md for Claude Code or AGENTS.md for Codex) that tells the LLM how the wiki is structured, what the conventions are, and what workflows to follow when ingesting sources, answering questions, or maintaining the wiki. This is the key configuration file — it's what makes the LLM a disciplined wiki maintainer rather than a generic chatbot. You and the LLM co-evolve this over time as you figure out what works for your domain.

## Operations

**Ingest.** You drop a new source into the raw collection and tell the LLM to process it. An example flow: the LLM reads the source, discusses key takeaways with you, writes a summary page in the wiki, updates the index, updates relevant entity and concept pages across the wiki, and appends an entry to the log. A single source might touch 10-15 wiki pages. Personally I prefer to ingest sources one at a time and stay involved — I read the summaries, check the updates, and guide the LLM on what to emphasize. But you could also batch-ingest many sources at once with less supervision. It's up to you to develop the workflow that fits your style and document it in the schema for future sessions.

**Query.** You ask questions against the wiki. The LLM searches for relevant pages, reads them, and synthesizes an answer with citations. Answers can take different forms depending on the question — a markdown page, a comparison table, a slide deck (Marp), a chart (matplotlib), a canvas. The important insight: **good answers can be filed back into the wiki as new pages.** A comparison you asked for, an analysis, a connection you discovered — these are valuable and shouldn't disappear into chat history. This way your explorations compound in the knowledge base just like ingested sources do.

**Lint.** Periodically, ask the LLM to health-check the wiki. Look for: contradictions between pages, stale claims that newer sources have superseded, orphan pages with no inbound links, important concepts mentioned but lacking their own page, missing cross-references, data gaps that could be filled with a web search. The LLM is good at suggesting new questions to investigate and new sources to look for. This keeps the wiki healthy as it grows.

## Indexing and logging

Two special files help the LLM (and you) navigate the wiki as it grows. They serve different purposes:

**index.md** is content-oriented. It's a catalog of everything in the wiki — each page listed with a link, a one-line summary, and optionally metadata like date or source count. Organized by category (entities, concepts, sources, etc.). The LLM updates it on every ingest. When answering a query, the LLM reads the index first to find relevant pages, then drills into them. This works surprisingly well at moderate scale (~100 sources, ~hundreds of pages) and avoids the need for embedding-based RAG infrastructure.

**log.md** is chronological. It's an append-only record of what happened and when — ingests, queries, lint passes. A useful tip: if each entry starts with a consistent prefix (e.g. `## [2026-04-02] ingest | Article Title`), the log becomes parseable with simple unix tools — `grep "^## \[" log.md | tail -5` gives you the last 5 entries. The log gives you a timeline of the wiki's evolution and helps the LLM understand what's been done recently.

## Optional: CLI tools

At some point you may want to build small tools that help the LLM operate on the wiki more efficiently. A search engine over the wiki pages is the most obvious one — at small scale the index file is enough, but as the wiki grows you want proper search. [qmd](https://github.com/tobi/qmd) is a good option: it's a local search engine for markdown files with hybrid BM25/vector search and LLM re-ranking, all on-device. It has both a CLI (so the LLM can shell out to it) and an MCP server (so the LLM can use it as a native tool). You could also build something simpler yourself — the LLM can help you vibe-code a naive search script as the need arises.

## Tips and tricks

- **Obsidian Web Clipper** is a browser extension that converts web articles to markdown. Very useful for quickly getting sources into your raw collection.
- **Download images locally.** In Obsidian Settings → Files and links, set "Attachment folder path" to a fixed directory (e.g. `raw/assets/`). Then in Settings → Hotkeys, search for "Download" to find "Download attachments for current file" and bind it to a hotkey (e.g. Ctrl+Shift+D). After clipping an article, hit the hotkey and all images get downloaded to local disk. This is optional but useful — it lets the LLM view and reference images directly instead of relying on URLs that may break. Note that LLMs can't natively read markdown with inline images in one pass — the workaround is to have the LLM read the text first, then view some or all of the referenced images separately to gain additional context. It's a bit clunky but works well enough.
- **Obsidian's graph view** is the best way to see the shape of your wiki — what's connected to what, which pages are hubs, which are orphans.
- **Marp** is a markdown-based slide deck format. Obsidian has a plugin for it. Useful for generating presentations directly from wiki content.
- **Dataview** is an Obsidian plugin that runs queries over page frontmatter. If your LLM adds YAML frontmatter to wiki pages (tags, dates, source counts), Dataview can generate dynamic tables and lists.
- The wiki is just a git repo of markdown files. You get version history, branching, and collaboration for free.

## Why this works

The tedious part of maintaining a knowledge base is not the reading or the thinking — it's the bookkeeping. Updating cross-references, keeping summaries current, noting when new data contradicts old claims, maintaining consistency across dozens of pages. Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don't get bored, don't forget to update a cross-reference, and can touch 15 files in one pass. The wiki stays maintained because the cost of maintenance is near zero.

The human's job is to curate sources, direct the analysis, ask good questions, and think about what it all means. The LLM's job is everything else.

The idea is related in spirit to Vannevar Bush's Memex (1945) — a personal, curated knowledge store with associative trails between documents. Bush's vision was closer to this than to what the web became: private, actively curated, with the connections between documents as valuable as the documents themselves. The part he couldn't solve was who does the maintenance. The LLM handles that.


## Note

This document is intentionally abstract. It describes the idea, not a specific implementation. The exact directory structure, the schema conventions, the page formats, the tooling — all of that will depend on your domain, your preferences, and your LLM of choice. Everything mentioned above is optional and modular — pick what's useful, ignore what isn't. For example: your sources might be text-only, so you don't need image handling at all. Your wiki might be small enough that the index file is all you need, no search engine required. You might not care about slide decks and just want markdown pages. You might want a completely different set of output formats. The right way to use this is to share it with your LLM agent and work together to instantiate a version that fits your needs. The document's only job is to communicate the pattern. Your LLM can figure out the rest.

# The /llms.txt file
Jeremy Howard
2024-09-03

## Background

Large language models increasingly rely on website information, but face
a critical limitation: context windows are too small to handle most
websites in their entirety. Converting complex HTML pages with
navigation, ads, and JavaScript into LLM-friendly plain text is both
difficult and imprecise.

While websites serve both human readers and LLMs, the latter benefit
from more concise, expert-level information gathered in a single,
accessible location. This is particularly important for use cases like
development environments, where LLMs need quick access to programming
documentation and APIs.

## Proposal

<figure>
<img src="logo.png" class="lightbox floatr" width="150"
alt="llms.txt logo" />
<figcaption aria-hidden="true">llms.txt logo</figcaption>
</figure>

We propose adding a `/llms.txt` markdown file to websites to provide
LLM-friendly content. This file offers brief background information,
guidance, and links to detailed markdown files.

llms.txt markdown is human and LLM readable, but is also in a precise
format allowing fixed processing methods (i.e. classical programming
techniques such as parsers and regex).

We furthermore propose that pages on websites that have information that
might be useful for LLMs to read provide a clean markdown version of
those pages at the same URL as the original page, but with `.md`
appended. (URLs without file names should append `index.html.md`
instead.)

The [FastHTML project](https://fastht.ml) follows these two proposals
for its documentation. For instance, here is the [FastHTML docs
llms.txt](https://www.fastht.ml/docs/llms.txt). And here is an example
of a [regular HTML docs
page](https://www.fastht.ml/docs/tutorials/by_example.html), along with
exact same URL but with [a .md
extension](https://www.fastht.ml/docs/tutorials/by_example.html.md).

This proposal does not include any particular recommendation for how to
process the llms.txt file, since it will depend on the application. For
example, the FastHTML project opted to automatically expand the llms.txt
to two markdown files with the contents of the linked URLs, using an
XML-based structure suitable for use in LLMs such as Claude. The two
files are: [llms-ctx.txt](https://fastht.ml/docs/llms-ctx.txt), which
does not include the optional URLs, and
[llms-ctx-full.txt](https://fastht.ml/docs/llms-ctx-full.txt), which
does include them. They are created using the
[`llms_txt2ctx`](https://llmstxt.org/intro.html#cli) command line
application, and the FastHTML documentation includes information for
users about how to use them.

The versatility of llms.txt files means they can serve many purposes -
from helping developers find their way around software documentation, to
giving businesses a way to outline their structure, or even breaking
down complex legislation for stakeholders. They’re just as useful for
personal websites where they can help answer questions about someone’s
CV, for e-commerce sites to explain products and policies, or for
schools and universities to provide quick access to their course
information and resources.

Note that all [nbdev](https://nbdev.fast.ai/) projects now create .md
versions of all pages by default. All Answer.AI and fast.ai software
projects using nbdev have had their docs regenerated with this feature.
For an example, see the [markdown
version](https://fastcore.fast.ai/docments.html.md) of [fastcore’s
docments module](https://fastcore.fast.ai/docments.html).

## Format

At the moment the most widely and easily understood format for language
models is Markdown. Simply showing where key Markdown files can be found
is a great first step. Providing some basic structure helps a language
model to find where the information it needs can come from.

The `llms.txt` file is unusual in that it uses Markdown to structure the
information rather than a classic structured format such as XML. The
reason for this is that we expect many of these files to be read by
language models and agents. Having said that, the information in
llms.txt follows a specific format and can be read using standard
programmatic-based tools.

The llms.txt file spec is for files located in the root path `/llms.txt`
of a website (or, optionally, in a subpath). A file following the spec
contains the following sections as markdown, in the specific order:

- An optional byte-order mark (BOM)
- An H1 with the name of the project or site. This is the only required
  section
- A blockquote with a short summary of the project, containing key
  information necessary for understanding the rest of the file
- Zero or more markdown sections (e.g. paragraphs, lists, etc) of any
  type except headings, containing more detailed information about the
  project and how to interpret the provided files
- Zero or more markdown sections delimited by H2 headers, containing
  “file lists” of URLs where further detail is available
  - Each “file list” is a markdown list, containing a required markdown
    hyperlink `[name](url)`, then optionally a `:` and notes about the
    file.

Here is a mock example:

``` markdown
# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)
```

Note that the “Optional” section has a special meaning—if it’s included,
the URLs provided there can be skipped if a shorter context is needed.
Use it for secondary information which can often be skipped.

## Existing standards

llms.txt is designed to coexist with current web standards. While
sitemaps list all pages for search engines, `llms.txt` offers a curated
overview for LLMs. It can complement robots.txt by providing context for
allowed content. The file can also reference structured data markup used
on the site, helping LLMs understand how to interpret this information
in context.

The approach of standardising on a path for the file follows the
approach of `/robots.txt` and `/sitemap.xml`. robots.txt and `llms.txt`
have different purposes—robots.txt is generally used to let automated
tools know what access to a site is considered acceptable, such as for
search indexing bots. On the other hand, `llms.txt` information will
often be used on demand when a user explicitly requests information
about a topic, such as when including a coding library’s documentation
in a project, or when asking a chat bot with search functionality for
information. Our expectation is that `llms.txt` will mainly be useful
for *inference*, i.e. at the time a user is seeking assistance, as
opposed to for *training*. However, perhaps if `llms.txt` usage becomes
widespread, future training runs could take advantage of the information
in `llms.txt` files too.

sitemap.xml is a list of all the indexable human-readable information
available on a site. This isn’t a substitute for `llms.txt` since it:

- Often won’t have the LLM-readable versions of pages listed
- Doesn’t include URLs to external sites, even though they might be
  helpful to understand the information
- Will generally cover documents that in aggregate will be too large to
  fit in an LLM context window, and will include a lot of information
  that isn’t necessary to understand the site.

## Example

Here’s an example of `llms.txt`, in this case a cut down version of the
file used for the FastHTML project (see also the [full
version](https://www.fastht.ml/docs/llms.txt)):

``` markdown
# FastHTML

> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's `FT` "FastTags" into a library for creating server-rendered hypermedia applications.

Important notes:

- Although parts of its API are inspired by FastAPI, it is *not* compatible with FastAPI syntax and is not targeted at creating API services
- FastHTML is compatible with JS-native web components and any vanilla JS library, but not with React, Vue, or Svelte.

## Docs

- [FastHTML quick start](https://fastht.ml/docs/tutorials/quickstart_for_web_devs.html.md): A brief overview of many FastHTML features
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): Brief description of all HTMX attributes, CSS classes, headers, events, extensions, js lib methods, and config options

## Examples

- [Todo list application](https://github.com/AnswerDotAI/fasthtml/blob/main/examples/adv_app.py): Detailed walk-thru of a complete CRUD app in FastHTML showing idiomatic use of FastHTML and HTMX patterns.

## Optional

- [Starlette full documentation](https://gist.githubusercontent.com/jph00/809e4a4808d4510be0e3dc9565e9cbd3/raw/9b717589ca44cedc8aaf00b2b8cacef922964c0f/starlette-sml.md): A subset of the Starlette documentation useful for FastHTML development. 
```

To create effective `llms.txt` files, consider these guidelines:

- Use concise, clear language.
- When linking to resources, include brief, informative descriptions.
- Avoid ambiguous terms or unexplained jargon.
- Run a tool that expands your `llms.txt` file into an LLM context file
  and test a number of language models to see if they can answer
  questions about your content.

## Directories

Here are a few directories that list the `llms.txt` files available on
the web:

- [llmstxt.site](https://llmstxt.site/)
- [directory.llmstxt.cloud](https://directory.llmstxt.cloud/)

## Integrations

Various tools and plugins are available to help integrate the llms.txt
specification into your workflow:

- [`llms_txt2ctx`](https://llmstxt.org/intro.html#cli) - CLI and Python
  module for parsing llms.txt files and generating LLM context
- [JavaScript Implementation](./llmstxt-js.html) - Sample JavaScript
  implementation
- [`vitepress-plugin-llms`](https://github.com/okineadev/vitepress-plugin-llms) -
  VitePress plugin that automatically generates LLM-friendly
  documentation for the website following the llms.txt specification
- [`docusaurus-plugin-llms`](https://github.com/rachfop/docusaurus-plugin-llms) -
  Docusaurus plugin for generating LLM-friendly documentation following
  the llmtxt.org standard
- [Drupal LLM Support](https://www.drupal.org/project/llm_support) - A
  Drupal Recipe providing full support for the llms.txt proposal on any
  Drupal 10.3+ site
- [`llms-txt-php`](https://github.com/raphaelstolt/llms-txt-php) - A
  library for writing and reading llms.txt Markdown files
- [`VS Code PagePilot Extension`](https://dmux.github.io/pagepilot) -
  PagePilot is a VS Code Chat participant that automatically loads
  external context (documentation, APIs, README files) to provide
  enhanced responses.

## Next steps

The `llms.txt` specification is open for community input. A [GitHub
repository](https://github.com/AnswerDotAI/llms-txt) hosts [this
informal
overview](https://github.com/AnswerDotAI/llms-txt/blob/main/nbs/index.qmd),
allowing for version control and public discussion. A [community discord
channel](https://discord.gg/aJPygMvPEN) is available for sharing
implementation experiences and discussing best practices.