Written by The problem: pasting documentation page by page
This started on a client job: a legacy vendor API whose reference docs WebFetch couldn’t handle, and that no hosted docs catalogue (like Context7) had indexed. The workflow degenerated into hunting the right page and pasting it into the context window, one question at a time. A token-hungry developer-experience nightmare. So we built a small Model Context Protocol server over the docs. Scrape once to markdown, index with plain lexical search, serve a search_docs tool locally. No retrieval-augmented generation (RAG), no embeddings, no API keys.
On this page
MCP is the open standard Anthropic shipped in late 2024. Claude Code, the Codex CLI, Cursor, Continue, and Zed all speak it, so a server you write once shows up as native tools in whichever assistant you pick up next, and only the snippet it asked for comes back, only when it asks. The shift is from “give the LLM a context bundle and hope” to “give it a search call and let it pull what it needs.”
The build below is the same pattern pointed at docs you can check for yourself: the PostgreSQL and PostGIS manuals, all 1,832 pages.
Four files make a docs MCP server
scrape_docs.py: walks the docs, saves each page as markdown. The only source-aware part.search_index.py: TF-IDF (term frequency-inverse document frequency), fuzzy, and exact ranking over the markdown.server.py: the MCP server, exposing asearch_docstool andpgdocs://resources.setup.sh: scrapes, indexes, prints the oneclaude mcp addline.
The boundary between the scraper and everything downstream is the whole trick. The scraper knows the upstream’s HTML; downstream is just “a directory of markdown.” Draw that line and the engine is source-agnostic. Repoint one list of URLs and it moves to a new manual untouched.
Wiring it in is one call, claude mcp add pgdocs -- python /path/to/server.py, and search_docs appears alongside Read, Edit, WebFetch. The same binary works in Codex, Cursor, or anything else that speaks MCP.
Why TF-IDF, not embeddings
The temptation is a vector database and an embedding model. We didn’t reach for either, because what you search technical docs for is almost always identifiers (function names, types, configuration parameters (GUCs), error codes), and identifiers need lexical match. ST_GeomFromText and ST_GeomFromGeoJSON are near-identical in meaning and very different lexically, and the tool has to keep them apart. Embeddings earn their keep in a retrieval-augmented generation pipeline; here they’d add a model dependency for a workload lexical ranking already fits.
So the index runs three rankers per query and takes the max:
- exact substring, boosted; wins for
ST_Intersects - fuzzy on titles; handles
ST_Intersect→ST_Intersects - TF-IDF cosine; wins for “spatial buffer with negative distance”
Cold build over 1,832 files is a few seconds; every query after is sub-50 ms from a pickled index on a laptop. Searching over the manuals is a different job from full-text search inside Postgres; if you want the latter, see our PostgreSQL full-text search in-depth guide.
For search_docs("window function frame clause", strategy="semantic") the server returns tutorial-window (0.52), functions-window (0.44), then sql-expressions (0.33): concept, then reference, then grammar, with a clear gap to the noise floor. That gap is the signal the model uses to pick a page.
The three things the scraper has to handle
This is where all the source-specific work, and the iterations, actually live.
- Breadth-first search (BFS), not single-level. Scraping the index and fetching what it links to misses ~1,000 leaves; the index is the root of a tree, not a sitemap. Follow links recursively, restricted by a same-domain regex, deduped via a
seenset, capped so a bad regex can’t crawl the internet. The corpus went from 121 URLs to 1,832. - One pass, not two. A discover-then-scrape split is clean but fetches every page twice. Fetch once, extract, queue further links from the same response. Runtime halved.
- Resumable. ~1,800 pages at a polite 0.3 s each is ~9 minutes; skip files already on disk, so a dropped connection costs one page, not the run, and re-runs after upstream updates are incremental.
Files land namespaced (pg__select.md, postgis__st_intersects.md); one index, filtered by prefix. Two manuals, zero schema churn. (A bonus tool, get_function_signature, regexes the funcname ( arg type ) → return line out of any page: clean overloads instead of the model reading 4 KB of HTML.)
When it’s worth it
The test is simple: do you open the same docs site more than twice a week and paste from it into an LLM? If yes, build it. If not, WebFetch is fine.
Two things don’t matter: volatility (re-scraping is one incremental command) and size in the small-to-medium range (a 5,000-term TF-IDF vocabulary handles 1,832 pages; we haven’t tested past that). Three things do:
- Structure. Static server-rendered HTML scrapes cleanly; a JS-rendered site needs a headless browser or a fallback, the kind of crawl in getting data AI-ready with crawl4ai, so check before you start.
- Versioning. The scraper points at one version. Straddling majors in production means a per-version MCP or a
versionargument. Decide up front. - Freshness. The MCP serves the last scrape. A critical upstream fix won’t show until you re-run. Schedule it, or accept the lag.
Why not Context7? If your target is a popular open-source library, Context7 already serves its docs over MCP and is the shorter path: add it and go. It is also a hosted service, so every query is a network call against their index, and the catalogue is theirs, not yours. The pattern here is for what a hosted catalogue doesn’t cover (a vendor’s REST API reference, internal docs, a pinned PostgreSQL major) and for teams that want the answers local, with no account, no rate limit, and nothing leaving the machine.
Clone the docs MCP
Porting it from the original vendor API to the Postgres manuals changed the scraper’s SOURCES entry and one DOM selector; the engine, server, and setup script moved untouched. A working afternoon.
The shape of “give Claude a docs tool” should be this small: four files, ~400 lines, at github.com/FloreData/pgdocs-mcp. If your team keeps paying for a manual in context tokens, point SOURCES at your docs and stop. If you’d rather have it built and maintained for you, that’s the kind of work we do.
