The chatbot doesn’t know your docs

New to embeddings or to calling a model? Open a quick recap. Each takes about two minutes.

Quick recapEmbeddings· 2 min

An embedding turns a text into a list of numbers, so texts that mean similar things get similar numbers. “Can I send it back?” and “returns policy” land close together, even with no word in common. Cosine similarity compares two lists: 1 means the same direction, 0 means nothing in common.

Quick recapCalling a model· 2 min

You send a prompt (text) to a chat model’s API and get text back. Models read tokens, pieces of words: about 4 tokens for every 3 English words. You pay for every token you send and get back, and a model can only read so many at once (its context window).

See the problem

Your shop’s support chatbot answers questions about returns, shipping and payments. At launch, the help center had 11 short pages, so the bot pasted all of them into every prompt. It worked.

A year later the help center has 2,000 pages. That’s about a million tokens for every question. Even where a model can take that much, you pay for all of it every time. Answers take seconds longer, and the bot starts mixing up pages.

“Build a chatbot that answers from our docs” is one of the most common AI engineering interview questions. The answer interviewers want is RAG, and what goes wrong with it.

Define the task

Input: a customer’s question, and the shop’s help docs. In this lesson, 11 short docs, so you can see every score. The same steps work for 2,000 pages.

Output: a short answer taken from the docs, with a citation like [1] that points to the doc it came from. If the docs don’t cover the question, the bot says “The docs don’t say.” instead of guessing.

The goal: the right doc reaches the model for almost every question, the prompt stays small at any number of docs, and you can measure both.

Pause and think

⏸ Pause and think

Before reading on: if you can only send the model 3 docs per question, how would you pick them? What could make you pick the wrong 3?

Give it two minutes with pen and paper before you open a hint.

Hint 1 of 3

You can’t read 2,000 pages per question, but you can prepare something about each page once, ahead of time.

Hint 2 of 3

Embeddings put similar meanings close together. Which docs are closest to the question?

Hint 3 of 3

What does a code like E-42 mean to an embedding? Is it a meaning or an exact word?

Understand the idea

Follow one question through basic RAG. Our toy embedding has 8 named meanings, so you can read it; a real one has hundreds or thousands of numbers that mean nothing to a human. Then type your own question.

One question through basic RAG

⤢ Full view

1. Question
2. Embed
3. Retrieve
4. Prompt
5. Answer

QuestionCan I return something I bought on sale?

Step 1 of 5. A customer asks the shop’s chatbot: “Can I return something I bought on sale?” Before answering, it looks up the help docs.

◀ BackPlayNext ▶Reset Ask me first Show as text

Try your own input

Your questionShow it

★Key insight

RAG is search first, then generation. The model only sees what search found. If search finds the wrong doc, the best model in the world gives a confident, wrong answer. Most RAG work is making search better and measuring it.

Climb the ladder

Three versions of the bot. Each one shows the idea, the code, its cost per question and where it breaks. The code runs the toy pipeline in toy_rag.py, with simple fakes for the parts that need a model, so it runs without an API key. llm is any function that sends a prompt to a chat model.

Okay

Paste every doc into the prompt

The idea: Put the whole help center in the prompt and let the model find the answer.

For 11 short docs this is fine, and it’s a good first answer in an interview: it works, and it shows you know the limit.

"""Okay: paste the whole handbook into every prompt. `llm` is any function that
sends a prompt to a chat model and returns its reply."""

from toy_rag import DOCS


def ask(question: str, llm) -> str:
    handbook = "\n".join(f"- {d['title']}: {d['text']}" for d in DOCS)
    return llm(f"Answer from this handbook.\n{handbook}\nQuestion: {question}")

Cost per question

every doc: about 280 tokens for our 11

Weak spot

grows with every page you add

Why

Simple and often right while the docs are small. The model sees everything, so search can't miss.

Where it breaks: 2,000 pages is about a million tokens per question. Cost and waiting time grow with every page. And models use a line in the middle of a long prompt less well than one near the start or end. So the one right sentence gets lost.

Better

Basic RAG: the 3 closest docs

The idea: Embed every doc once and store the vectors. For each question, embed it, find the 3 closest docs, and send only those.

The waste in Okay: it sends every doc to answer a question that only one doc answers.

Long docs are split into chunks of a few hundred tokens first, so each vector is about one thing (practice problem 1). The prompt numbers the chunks and tells the model to answer only from them and to cite them.

Highlighted lines are new in this rung.

"""Better: basic RAG. Embed every doc once and store the vectors. For each
question, send the model only the 3 closest docs."""

from toy_rag import build_prompt, vector_search


def retrieve(question: str, k: int = 3) -> list[str]:
    """The k docs closest in meaning. A vector database does this over millions."""
    return [doc_id for doc_id, _ in vector_search(question, k)]


def ask(question: str, llm) -> str:
    return llm(build_prompt(question, retrieve(question)))

Cost per question

3 docs: about 120 tokenswas every doc

Weak spot

misses exact words: codes, names, ids

Why

The prompt stays the same size at 11 docs or 2,000. The vectors are made once; a vector database finds the closest ones among millions in milliseconds.

What changed from the previous rung

"""Okay: paste the whole handbook into every prompt. `llm` is any function that
sends a prompt to a chat model and returns its reply."""
"""Better: basic RAG. Embed every doc once and store the vectors. For each
question, send the model only the 3 closest docs."""

from toy_rag import DOCS
from toy_rag import build_prompt, vector_search


def retrieve(question: str, k: int = 3) -> list[str]:
    """The k docs closest in meaning. A vector database does this over millions."""
    return [doc_id for doc_id, _ in vector_search(question, k)]


def ask(question: str, llm) -> str:
    handbook = "\n".join(f"- {d['title']}: {d['text']}" for d in DOCS)
    return llm(f"Answer from this handbook.\n{handbook}\nQuestion: {question}")
    return llm(build_prompt(question, retrieve(question)))

Where it breaks: ask “My payment failed with error E-42. What does it mean?” Our toy embedding sees the payment meaning, and both payment docs score high. Payment methods comes first, so the bot answers about cards. The doc with “E-42” in it is second. Vectors keep what words mean, not rare exact words. Real embeddings do better than ours, but codes, product names and ids are still their weak spot.

Best

Hybrid search, a reranker, citations and an eval set

The idea: Search by meaning and by exact words, and merge the two lists by rank. Rerank the best few with a stronger model, and answer only from those, with citations. Measure it all with an eval set.

The waste in Better: it trusts one kind of search, and it never checks its picks.

• Keyword search (BM25) scores a doc by the question’s words in it. Rare words count the most, so “e-42” lifts the checkout errors doc to first place.
• Reciprocal rank fusion gives each doc 1 / (60 + its rank) in each list and adds them up. A BM25 score of 5.6 and a cosine of 0.96 use different scales, but ranks can always be added.
• A reranker (a cross-encoder model) reads the question and one candidate together, so it scores better than comparing two vectors. It’s too slow to run on every doc, so it runs on the top few.
• Citations and “The docs don’t say.” The prompt says to answer only from the numbered docs. When search finds nothing, the code answers without calling the model at all.
• Also at this scale: rewrite vague questions before searching (“it broke” becomes “item arrived damaged”), filter by metadata (product, language, who may see it; practice problem 4), and cache answers to common questions.

Highlighted lines are new in this rung.

"""Best: search two ways, merge, rerank, and answer only from what was found,
with citations. eval_set.py measures it."""

from toy_rag import bm25_search, build_prompt, rerank, rrf, vector_search


def retrieve(question: str, k: int = 3, candidates: int = 5) -> list[str]:
    """candidates is 5 for our 11 docs; on a real corpus, 50 or so."""
    by_meaning = [d for d, _ in vector_search(question, candidates)]
    by_words = [d for d, _ in bm25_search(question, candidates)]
    fused = [d for d, _ in rrf([by_meaning, by_words])][:candidates]
    return [d for d, _ in rerank(question, fused, k)]  # a cross-encoder, in real life


def ask(question: str, llm) -> str:
    ids = retrieve(question)
    if not ids:
        return "The docs don't say."  # no model call, no made-up answer
    return llm(build_prompt(question, ids))  # the prompt asks for [1]-style citations

Cost per question

3 docs, plus a rerank of 5was 3 docs

Weak spot

more parts to run and measure

Why

Keyword search (BM25) catches exact words; vectors catch meaning. Reciprocal rank fusion merges them by rank, and the reranker reads each candidate with the question to put the best first.

What changed from the previous rung

"""Better: basic RAG. Embed every doc once and store the vectors. For each
question, send the model only the 3 closest docs."""
"""Best: search two ways, merge, rerank, and answer only from what was found,
with citations. eval_set.py measures it."""

from toy_rag import build_prompt, vector_search
from toy_rag import bm25_search, build_prompt, rerank, rrf, vector_search


def retrieve(question: str, k: int = 3) -> list[str]:
    """The k docs closest in meaning. A vector database does this over millions."""
    return [doc_id for doc_id, _ in vector_search(question, k)]
def retrieve(question: str, k: int = 3, candidates: int = 5) -> list[str]:
    """candidates is 5 for our 11 docs; on a real corpus, 50 or so."""
    by_meaning = [d for d, _ in vector_search(question, candidates)]
    by_words = [d for d, _ in bm25_search(question, candidates)]
    fused = [d for d, _ in rrf([by_meaning, by_words])][:candidates]
    return [d for d, _ in rerank(question, fused, k)]  # a cross-encoder, in real life


def ask(question: str, llm) -> str:
    return llm(build_prompt(question, retrieve(question)))
    ids = retrieve(question)
    if not ids:
        return "The docs don't say."  # no model call, no made-up answer
    return llm(build_prompt(question, ids))  # the prompt asks for [1]-style citations

When it isn’t best: every part adds time and something to watch. With a few dozen pages, Okay or Better is enough. Add each part when the eval set shows a miss it would fix.

Watch the Best rung find the error code

The same question that tricked basic RAG. Watch both searches, the tie after fusing, and the reranker breaking it.

One question through hybrid RAG

⤢ Full view

1. Question
2. Search
3. Fuse
4. Rerank
5. Prompt
6. Answer

QuestionMy payment failed with error E-42. What does it mean?

Step 1 of 6. A customer asks: “My payment failed with error E-42. What does it mean?” This time the bot searches two ways, by meaning and by exact words, then checks its picks. Question: Which doc will keyword search put first?

What happens next? Which doc will keyword search put first?

Checkout errorsPayment methodsReturns policyJust show me

◀ BackPlayNext ▶Reset Ask me first Show as text

Try your own input

Your questionShow it

Measure it: an eval set

“It seems better” isn’t a measurement. An eval set is a list of real questions, each with the doc that answers it. Run it after every change to chunking, search or prompts.

eval_set.py

"""An eval set: questions with the doc that answers each one. Run it after
every change to chunking, search or prompts, and keep the numbers."""

EVAL_SET = [
    ("Can I return something I bought on sale?", "sale"),
    ("My payment failed with error E-42. What does it mean?", "error-codes"),
    ("Can I get cash for a gift card?", "gift-cards"),
    ("How long does express shipping take?", "shipping-times"),
    ("Is shipping free?", "shipping-costs"),
    ("My RX-200 headphones broke after 18 months. Are they covered?", "warranty"),
    ("The mug arrived broken. What do I do?", "damaged"),
    ("Can I pay cash on delivery?", "payment-methods"),
    ("I forgot my password.", "password"),
    ("Can I cancel my order?", "order-changes"),
    ("How many days do I have to return a jacket?", "returns"),
]


def evaluate(retrieve, eval_set=EVAL_SET) -> dict[str, int]:
    """first: the right doc came first. top_3: it was in the top 3 (recall@3)."""
    first = top_3 = 0
    for question, right in eval_set:
        ids = retrieve(question)[:3]
        first += ids[:1] == [right]
        top_3 += right in ids
    return {"first": first, "top_3": top_3, "questions": len(eval_set)}

On our 11 questions:

Rung	Right doc first	Right doc in the top 3 (recall@3)
Better (vectors only)	5 of 11	9 of 11
Best (hybrid and rerank)	10 of 11	11 of 11

One of Better’s two top-3 misses comes from our toy: “cancel” isn’t one of its 8 meanings, so “Can I cancel my order?” finds nothing by meaning. A real embedding would find the doc about changing an order.

The one Best still gets wrong is “The mug arrived broken.” The doc says “damaged”, not “broken”, so the warranty doc comes first. The eval set found it; rewriting the question or adding “broken” to the doc would fix it. Real eval sets also check the answer: does every sentence come from a cited doc (faithfulness)?

Write it yourself

The heart of vector search: score every vector against the question with cosine similarity, and keep the best k. A vector database does the same, with an index so it doesn’t have to score every one.

Find the k closest vectors

Ctrl + Enter runs

import math def top_k(query, vectors, k): # query: a list of numbers. vectors: lists of numbers, each as long as query. # Return the positions of the k vectors most similar to query # (cosine similarity), best first. On a tie, the earlier position first. # Leave out vectors that score 0 or less, so [] means nothing matched. return []

Run testsVisualizeReset codeLoads about 6 MB the first time. Best on Wi-Fi.

1. ○Not run yet: top_k([1, 0], [[1, 0], [0, 1], [0.6, 0.8]], 2) → [0, 2]
2. ○Not run yet: A tie goes to the earlier position → [2, 0]
3. ○Not run yet: Extra test
4. ○Not run yet: Extra test
5. ○Not run yet: Extra test
6. ○Not run yet: Extra test

Extra tests check tricky cases. You see their input only if they fail.

Try your own input

Arguments, separated by commas as in Python

top_k()Run onceVisualize

Compare

Rung	Docs in each prompt	Our 11 docs	2,000 pages	Good	Not so good	Use it when
Okay	all of them	~280 tokens	~1,000,000 tokens	No search to get wrong	Cost grows with every page; lost lines	A few short pages
Better	the closest 3	~120 tokens	~1,500 tokens	Small prompt at any size	Misses codes and names; never checks	A first version, to measure
Best	the best 3 of 5	~120 tokens	~1,500 tokens + rerank	Finds exact words, checks its picks	More parts to run, tune and pay for	Real users, with an eval set

How much does each rung send? Each bar is one rung at the same number of docs, counted in docs per prompt. Pick a size, or run the race.

Docs sent to the model, at n docs

1. OkayPaste every doc · n≈ 2,000 docs

2. BetterBasic RAG, top 3 · 3≈ 3 docs

3. BestHybrid, rerank, top 3 · 3≈ 3 docs

At n = 2,000, Okay does about 670 times more work than Better.

Every bar does the same number of steps per second, so the race shows which rung finishes first. Turn on Log scale to compare the short bars.

Common mistakes and edge cases

!The model can only be as right as the search

When the bot is wrong, look at what search found before you change the prompt or the model. Log the retrieved doc ids with every answer, so you can see it.

• Chunks that are too big or too small. A whole page per vector mixes too many meanings; one sentence per vector loses the context. A few hundred tokens with some overlap is a common start. Then let the eval set decide.
• Embed questions and docs with the same model. Vectors from two models can’t be compared. Change the model, and you re-embed every doc.
• Stale docs. When a page changes, re-embed it. Keep the page id and its updated date with each chunk.
• “The docs don’t say” needs a rule. Without one, the model fills gaps from what it learned in training. Tell it to answer only from the docs, and skip the model call when nothing matched.
• Prompt injection in docs. A doc can contain “Ignore your instructions…”. Treat retrieved text as data, and never give the bot powers a doc could misuse.
• Who may see what. Search must only return docs the user may read. Filter before or during search, not in the prompt (practice problem 4).

Say it in the interview

Questions to ask first

• How many docs, how long, and how often do they change?
• What questions do users ask? Do they include codes, product names or ids?
• Can every user see every doc?
• What does a wrong answer cost: a confused customer, or a legal problem?
• How fast must it answer, and what’s the budget per question?

A 60-second answer

“If the docs are small, I’d paste them all into the prompt: simple, and nothing to retrieve wrong. At 2,000 pages that’s about a million tokens per question, so I’d use RAG. I’d chunk the docs into a few hundred tokens each, embed them once, and store the vectors. For each question, embed it, take the closest chunks and send them with an instruction to answer only from them and cite them. Vectors miss exact words like error codes, so I’d add BM25 keyword search. I’d merge the two lists with reciprocal rank fusion, then rerank the top 20 to 50 with a cross-encoder and keep 3 to 5. Before any of that ships, I’d build an eval set of real questions with the right doc for each. I’d track recall@k and answer faithfulness after every change.”

Likely follow-ups

• How do you choose the chunk size? Start with a few hundred tokens and some overlap, and split on headings where you can. Then compare recall@k on the eval set for two or three sizes.
• Why not fine-tune the model on the docs instead? Fine-tuning teaches style and format well, but it’s a poor way to add facts that change. RAG updates when a doc changes, and it can cite the source.
• How do you know an answer came from the docs? Ask for citations, then check them: each sentence should be supported by the cited chunk. Sample answers, and use an LLM judge carefully, checked against human labels.
• What if search finds nothing useful? Use a score threshold, answer “The docs don’t say,” and log the question, so the docs team can fill the gap.

Red flags interviewers notice

• Blaming the model before looking at what search returned.
• No way to measure: “I’d try it and see if it looks good.”
• Vector search only, with no plan for codes, names or ids.
• Putting access rules in the prompt instead of filtering the search.

Practice

Four more problems, from easy to hard. Each has a hint or two, then Okay, Better and Best answers. Try each one before you open the answers.

1. Split long docs into chunks (easy)

The returns page is 3,000 words. As one vector, it matches every question a little and none well. Split it so each piece can be found on its own, without cutting the key sentence in half.

Hint 1 of 2

What happens to a sentence that sits right on the line between two chunks?

Hint 2 of 2

What if each chunk starts a little before the last one ended?

Show the solutions

Okay

One chunk per page

The idea: Embed each whole page as one vector.

chunks = [page.text for page in pages]

Chunks

1 per page

Weak spot

a long page's vector is about everything

Why

Simple, and fine for short pages. A 3,000-word page mixes many topics into one vector, and sending it fills the prompt.

Better

Fixed-size chunks

The idea: Cut every page into pieces of 200 words.

chunks = [words[i : i + 200] for i in range(0, len(words), 200)]

Chunks

about words ÷ 200was 1 per page

Weak spot

a sentence on an edge is cut in two

Why

Each chunk is about one thing. But the answer may start at the end of one chunk and finish in the next.

Best

Overlap, headings and the page title

The idea: Start each chunk a little before the last one ended, and cut at headings or paragraphs where you can. Keep the page title and id with each chunk. The code shows the overlap.

"""Split a document into chunks of `size` words, each starting `size - overlap`
words after the last, so a sentence cut at one edge is whole in the next chunk."""


def chunk(words: list[str], size: int, overlap: int) -> list[list[str]]:
    if not 0 <= overlap < size:
        raise ValueError("overlap must be at least 0 and smaller than size")
    step = size - overlap
    chunks = []
    for start in range(0, len(words), step):
        chunks.append(words[start : start + size])
        if start + size >= len(words):
            break
    return chunks

Chunks

about words ÷ (size − overlap)

Weak spot

a bit more to store and embed

Why

With 200 words and an overlap of 40, a sentence cut at one edge is whole in the next chunk. The title gives a chunk context, and the id lets you cite and update it.

2. Merge two result lists (medium)

Vector search returns cosine scores from 0 to 1. Keyword search returns BM25 scores like 5.6 or 1.8. You want one ranked list. How do you merge them?

Hint 1 of 2

Can you add 0.96 and 5.6 and get something meaningful?

Hint 2 of 2

Scores differ, but every list has a first place, a second place…

Show the solutions

Okay

Add the raw scores

The idea: Each doc gets its cosine plus its BM25 score.

score[doc] = cosine.get(doc, 0) + bm25.get(doc, 0)

Works on

no real pair of searches

Weak spot

BM25's bigger numbers always win

Why

BM25 scores can be 5 or 10; cosines are under 1. The sum is just keyword search with a little noise.

Better

Scale both to 0–1, then add

The idea: Divide each list's scores by its best score, then add them, maybe with weights.

score[doc] = cosine.get(doc, 0) / max_cos + bm25.get(doc, 0) / max_bm25

Works on

lists with similar spreadswas none

Weak spot

one outlier squashes everything else

Why

Fairer than raw scores. But one very high BM25 score pushes every other keyword match close to 0, and the weights need tuning.

Best

Reciprocal rank fusion

The idea: Ignore the scores. Each list gives a doc 1 / (60 + its rank), and you add those up.

"""Reciprocal rank fusion: merge rankings from different searches (say,
keyword and vector) without comparing their scores, which use different scales."""


def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Each ranking gives an id 1 / (k + rank), rank counted from 1.
    Ids best first; ties keep the order in which ids first appear."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (k + rank)
    return sorted(scores, key=lambda doc_id: -scores[doc_id])

Works on

any lists, any scales

Weak spot

drops how far ahead a doc was

Why

Ranks are on the same scale in every list. The 60 keeps one first place from hiding the rest. A doc both lists rank high comes first.

3. The bot answers confidently and wrongly (medium)

A customer asks “Do you ship to Canada?”. The docs don’t say. The bot answers “Yes, shipping to Canada takes 5 to 7 days.” What’s going on, and how do you stop it?

Type the question into either player above: our toy bot makes the same mistake. Search finds docs that say “shipping” or “ship”, and the fake model copies a sentence from one of them.

Hint 1 of 2

What did search return for this question? What did the model do with it?

Hint 2 of 2

What should the bot say when the docs don’t cover a question, and who decides that: the model or your code?

Show the solutions

Okay

Tell the model not to make things up

The idea: Add 'Don't make things up' to the prompt.

prompt = "Don't make things up.\n" + docs + question

Stops

some made-up answers

Weak spot

the model still fills gaps

Why

Search returned docs that mention shipping, which are close but don't answer. The model stretched them into an answer.

Better

Answer only from the docs, with citations

The idea: Number the docs, tell the model to answer only from them and cite each claim, and to reply 'The docs don't say.' otherwise. Skip the model when no doc scores above a threshold.

if not ids:
  return "The docs don't say."
return llm(build_prompt(question, ids))  # answer only from [1], [2], [3]

Stops

most made-up answerswas some

Weak spot

unmeasured; the threshold is a guess

Why

The model has a clear way out, and code, not the model, handles 'nothing found'. Citations let users and you check the answer.

Best

Measure faithfulness, and fix the docs

The idea: Add questions the docs don't answer to the eval set. Check that its doc supports every cited sentence. Log every 'The docs don't say.' for the docs team.

NOT_IN_DOCS = ["Do you ship to Canada?", "Can I pay in bitcoin?"]
for question in NOT_IN_DOCS:  # run with the eval set, after every change
  assert "The docs don't say" in ask(question, llm)

Stops

what you measure, and keeps it stopped

Weak spot

an eval set to keep up to date

Why

The eval set catches made-up answers before users do, and the log shows which docs are missing.

4. Only show docs the user may see (hard)

The same bot now serves customers and staff. Staff docs include refund limits and internal phone numbers. A customer asks “What’s the most you can refund without a manager?” How do you keep staff docs out of customers’ answers?

Hint 1 of 3

Is the prompt a safe place for a security rule?

Hint 2 of 3

Search returns the top 3. What if all 3 are staff docs and you drop them afterwards?

Hint 3 of 3

Can the search itself skip docs the user can’t see?

Show the solutions

Okay

Ask the model to keep secrets

The idea: Put staff docs in the same index and tell the model not to share them with customers.

prompt = "Never share staff-only docs with customers.\n" + docs + question

Safe against

nothing

Weak spot

the secret is already in the prompt

Why

A prompt is not a security boundary. A clever question, or a model mistake, and the staff doc's text is in the answer.

Better

Filter the results after search

The idea: Tag each chunk with who may read it. After search, drop chunks the user can't see.

ids = [d for d in retrieve(question, k=20) if user.group in ACCESS[d]][:3]

Safe against

leakswas nothing

Weak spot

the top 20 may be all staff docs

Why

Staff docs never reach a customer's prompt. But if the best 20 matches are all staff-only, the customer gets nothing, though a public doc would have answered.

Best

Filter inside the search, and test it

The idea: Store who may read each chunk as metadata and filter during the search, so the top k is always from allowed docs. For strict separation, like different companies, use a separate index per tenant. Test with a user from each group.

ids = vector_db.search(embed(question), k=3, filter={"groups": user.groups})

Safe against

leaks, and empty answers

Weak spot

the access tags must stay correct

Why

Every result is one the user may read, and there are always k of them if k exist. The same filter belongs on keyword search too.

Check yourself

A few quick questions. Ones you miss are remembered, so you can find your weak spots later.

1. Multiple choiceThe help docs grew from 11 pages to 2,000. Why not paste them all into every prompt?

   Models can’t read docsYou pay for every token on every question, and the one right line gets lost in a long promptPasting docs is against the API rulesThe answer would have no citations

   Check answer
2. Predict the outputCosine similarity with the question. Which doc does top-1 vector search return?

   question = [0.7, 0.7, 0.0]
   docs = {
       "returns":  [1.0, 0.0, 0.0],
       "sale":     [0.6, 0.8, 0.0],
       "shipping": [0.0, 0.0, 1.0],
   }

   returnssaleshippingnone of them

   Check answer
3. Spot the bugCustomers ask "What does error E-42 mean?" and the bot answers about payment methods. The search code:

   def retrieve(question, k=3):
       q = embed(question)
       return nearest(q, doc_vectors, k)

   k is too smallVectors keep meaning, not rare exact words like codes; add keyword search and mergeThe question needs to be lowercaseCosine similarity can’t compare short texts

   Check answer
4. Fill in the answerReciprocal rank fusion gives a doc 1 / (60 + its ____) in each list.

   Your answerCheck answer
5. Pick the patternYou changed the chunk size. How do you know the bot got better, not worse?

   Ask it three questions and see if the answers look rightRun the eval set before and after: is the right doc first, and in the top 3?Check that the prompt got shorterAsk the model to grade its own answers

   Check answer

Flashcards

Add these to your review deck. Each card comes back a little later every time you get it right.

• Card 1What is RAG?Show answer
• Card 2What is an embedding?Show answer
• Card 3Why add keyword search to vector search?Show answer
• Card 4What does a reranker do?Show answer
• Card 5How do you stop the bot making things up?Show answer

Add 5 cards to my review deck

Cheat sheet

• RAG: search first, then the model answers from what search found, with citations.
• Paste everything only while the docs are small. Cost and lost lines grow with every page.
• Chunk long docs: a few hundred tokens, some overlap, keep the title and id.
• Embed docs once, with the same model as the questions. Cosine similarity finds the closest.
• Vectors miss exact words (codes, names, ids). Add BM25 and merge with reciprocal rank fusion: 1 / (60 + rank).
• Rerank the top 20 to 50 with a cross-encoder; keep 3 to 5.
• “The docs don’t say.” is decided by code when nothing matches, and asked of the model when the docs fall short.
• An eval set of real questions and their right docs: track recall@k and faithfulness after every change.

Go deeper

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, the 2020 paper that named RAG.
• Lost in the Middle: why a line in the middle of a long prompt is used less well.
• Okapi BM25, the keyword-search score, with its formula.
• The lesson’s code: content-code/ai-engineering/docs-chatbot/ in this site’s repository, with the toy pipeline, the three rungs, the eval set and their tests.

Can you do these now?

If you’re not sure of any of these, scroll back to that part before moving on.

• Explain why pasting every doc into the prompt stops working, in tokens and in answer quality
• Walk through basic RAG out loud, from embedding the question to an answer with a citation
• Say why vector search misses exact words like error codes, and fix it with keyword search and rank fusion
• Measure a RAG system with an eval set, so changes are judged by numbers

✓Mark as completeI already know this

Status: Not started