On this page
AI Strategy7 min · 22 Oct 2026
Give your apps and agents memory: RAG, knowledge graphs & document parsing
Memory isn't one thing — RAG, agent memory, and code context each need a different retrieval shape. Here's the decision guide and tools for each.
Geddy
Senior Web Engineer / Lead
0:00
AI-generated content. This article was generated by AI from material I collected and has not been fully reviewed by me. It may contain errors — verify anything you rely on. More in the disclaimer.
TL;DR30 sec
- "Memory" isn't one problem — document Q&A, repo awareness, and cross-session user recall each need a different retrieval shape, and using the wrong one produces confident garbage
- Naive top-k vector search is a demo; production RAG retrieves wide with hybrid search then reranks narrow with a cross-encoder, which is the cheapest quality win available
- For long structured documents, vectorless tree-reasoning retrieval crushes embeddings — PageIndex hits 98.7% on FinanceBench versus 31% for GPT-4o search
- Codebases want a symbol/import/call graph served over MCP plus grep, never embedded code chunks
- If your corpus fits in ~100K tokens and rarely changes, skip retrieval entirely and cache it into the context window
- Fine-tuning changes behaviour and format, not facts — anything that changes weekly belongs in retrieval, and parsing quality caps everything downstream Pick the retrieval shape that matches the job, then build a golden eval set before you tune a single parameter.
"Memory" is one word doing several different jobs.
Answering questions over a pile of documents is one job. Keeping a coding agent aware of a whole repo is another. Remembering a user across months of conversations is a third. Each one has its own best retrieval shape, and picking the wrong one is how you end up with a vector database that returns plausible-sounding garbage.
This is the comparison I wish I'd had: the approaches, the tools from my saved-screenshot archive that are actually worth using, and three stacks you can build today — one in a Next.js app, one for Claude Code, and one over the screenshot archive itself.
For harness design, context engineering and framework basics, see the companion piece, LLMs, agents & harnesses. This one stays on retrieval and memory.
1. TL;DR
- Classic vector RAG is still the default, but a naive top-k setup is a demo, not a product. Production RAG retrieves wide (top-50, hybrid) then reranks narrow with a cross-encoder.
- For long structured documents — filings, contracts, manuals — vectorless tree-reasoning retrieval can beat embeddings outright. PageIndex claims 98.7% on FinanceBench against 31% for GPT-4o search.
- Parsing sets the ceiling on everything downstream. Use a layout-aware parser that keeps page and bounding-box provenance, because that's what makes citations possible at all.
- For codebases, a knowledge graph of symbols, imports and calls served over MCP beats embedding code chunks.
- Agent memory is its own product category now. Capture what the agent did, then resurface only what's relevant — agentmemory reports ~1,900 tokens versus 22K+ for a bloated
CLAUDE.md. - Static corpora reused on every query don't need retrieval at all. Cache them or put them straight in the context window.
- Evaluate before you tune. Build a small golden Q&A set and track retrieval recall and answer faithfulness from day one.
2. The approaches in one screen
| Approach | How it works (1 line) | Best for | Trade-offs |
|---|---|---|---|
| Classic vector RAG | Chunk, embed, store in a vector DB, retrieve top-k by similarity, generate | Large, changing corpora; FAQ-style Q&A | Finds text that sounds relevant rather than text that is relevant; quality depends on chunking and reranking |
| Hybrid + rerank (two-stage) | Dense and keyword (BM25) search pull ~50 candidates, then a cross-encoder picks the best few | Any serious RAG system | One extra model call per query adds latency and cost |
| Vectorless / reasoning-based | Build a hierarchical ToC tree of each document; the LLM walks the tree from root to the relevant leaf | Long, structured professional docs where traceability matters | More LLM calls per query; weaker on huge flat corpora of short docs |
| CAG (cache-augmented) | Preload static knowledge into the KV cache once and reuse it across queries | Stable policies and docs hit by every query | Only works for data that fits the window and rarely changes; pair with RAG for dynamic data |
| Long-context stuffing | Put the whole corpus in the prompt, usually with prompt caching | Corpora under ~100–150K tokens; synthesis across everything | Cost scales with corpus size on every uncached call; recall degrades in the middle of very long contexts |
| Knowledge graph / GraphRAG | Extract entities and relations into a graph, query by traversal ("what depends on X?") | Codebases, multi-hop questions, "how is A connected to B" | Building the graph is costly; extracted edges can be wrong (tag inferred vs extracted) |
| Temporal knowledge graph | Every fact has start and end dates plus a source sentence, and nothing is overwritten | Enterprise "what was true on date X", audit trails | Heavier schema; overkill for plain Q&A |
| Agent memory store | Capture interactions and tool calls, distill them into facts, surface the relevant ones at session start | Coding agents, personal assistants, long-running threads | Stale or wrong memories persist, so it needs expiry and editing |
| Fine-tuning | Bake behaviour or domain style into weights (SFT, DPO, LoRA) | Format, tone and task skill; narrowing the gap with cheaper open models | Poor for facts that change; catastrophic forgetting; impossible on closed models |
| MEMo (memory as a model) | A small 14B "Memory" model internalizes the corpus; a frozen large model asks it sub-questions | Research direction; closed-model setups needing robustness to noise | Early paper, no off-the-shelf tooling; you still have to train a model |
3. Decision guide
- Corpus fits in ~100K tokens and changes rarely → skip retrieval entirely. Stuff it into the context with prompt caching. That's CAG in practice.
- Many short, independent docs (tickets, notes, FAQs) → hybrid search plus a reranker, stored in pgvector.
- A few long, structured docs (PDF reports, contracts, manuals) → a vectorless tree index in the PageIndex style, or section-aware chunks carrying page refs.
- Multi-hop or relationship questions ("what calls this?", "who owned X?") → a knowledge graph. Make it temporal if history matters.
- Code → a symbol/call graph over MCP, plus grep. Not embedded code chunks.
- The agent must remember across sessions → a memory store plus a short, hand-curated
CLAUDE.md. - You need a behaviour or format change, not new facts → fine-tune. Facts that change weekly belong in retrieval.
- Mixed static and dynamic data → cache the static part, retrieve the dynamic part.
4. Building blocks & tools
Document parsing, OCR and ingestion
| Tool | What | Use it for |
|---|---|---|
| OpenDataLoader PDF | Local PDF → Markdown/JSON/HTML, 100+ pages/sec on CPU, OCR for 80+ languages, bounding boxes and page numbers per element, prompt-injection filtering | Citation-grade ingestion. Hybrid mode sends hard pages to an AI backend, raising table accuracy from 0.49 to 0.93 |
| LiteParse v2 (LlamaIndex) | Rust PDF parser that handles a 457-page, 100MB document in 0.777 seconds; bindings for Python, Node, browser and edge; 50+ document types | Fast local parsing inside a Node/Next.js pipeline with no model dependency |
| Docling / LlamaParse / MegaParser / Extract Thinker | Document-to-structured-text extractors | Alternatives to benchmark on your own documents |
| Crawl4AI / Firecrawl / ScrapeGraphAI | Web crawlers that output LLM-ready Markdown | Ingesting docs sites and web knowledge |
| RAGFlow | Open-source end-to-end RAG engine with a UI | Standing up "chat with company docs" without writing the pipeline |
Embeddings, vector stores, retrieval and compression
| Tool | What | Use it for |
|---|---|---|
| pgvector | Vector type with HNSW/IVFFlat indexes inside Postgres | Keeping vectors next to app data, with SQL filters, joins and row-level security. The default for a Next.js app |
| Chroma / Qdrant / Milvus / Pinecone / Weaviate / OpenSearch | Dedicated vector databases | When you outgrow Postgres or need managed scale |
| Nomic / SBERT / Voyage AI / Cohere / OpenAI / Google embeddings | Text embedding models, open and hosted | Choose by eval on your data. Open models allow air-gapped setups |
| BGE-reranker / Cohere Rerank | Cross-encoder rerankers | Stage two of two-stage retrieval, and the biggest quality gain for the least effort |
| PageIndex | Vectorless, tree-based reasoning retrieval; 98.7% on FinanceBench; 30,000+ GitHub stars | Long professional PDFs needing exact page and section references |
| TurboVec | Rust vector index on Google's TurboQuant; squeezes a 31GB float32 index into 4GB (16x), no training, online ingest; beats FAISS by 12–20% on ARM | Large self-hosted indexes on limited RAM. Drop-in for LangChain, LlamaIndex, Haystack and Agno |
| Headroom | Compresses tool outputs, logs and RAG chunks before they reach the model (60–95% fewer tokens); reversible; local; library, proxy or MCP server | Cutting the cost of large retrieved context |
| pxpipe | Lossy context-compression proxy (npx pxpipe-proxy); ~70% savings depending on workload; tuned for Fable 5 (Opus misreads ~7%) |
Code-heavy context only, and only after you measure the accuracy |
Geddy
Senior Web Engineer / Lead
Engineering leadership • AI innovation • Product thinking. 20+ years of web engineering, from independent contractor to engineering leader. Passionate about developer experience and product engineering.
Follow on LinkedInKeep reading