Advanced: Retrieval and Model Serving in Depth

Why it matters#

Chapter 9 built the tutor's retrieval-augmented generation (RAG) pipeline, and chapter 16 made it hold up under load. Both treated retrieval and the model as parts you choose and tune. At a larger scale, with more tenants, more kinds of documents, and stricter data rules, two things that were background become central:

• Retrieval quality decides answer quality. A model can only be faithful to evidence it receives. When retrieval misses, a better model produces a more fluent wrong answer or a correct refusal; neither helps the student.
• Where and how the model runs becomes an architecture decision. Data rules, cost at different volumes, latency, and who operates the hardware all depend on it. Getting it wrong is expensive in both directions: self-hosting too early buys idle GPUs; refusing to self-host when a contract requires it loses the customer.

This module goes underneath both. You'll see how search ranks evidence and how to measure it properly, what changes when many tenants share an index, how model serving actually works, and how to decide among hosted, managed, and self-hosted models with numbers instead of preferences.

Learning objectives#

After this module you will be able to:

1. Explain lexical, semantic, and hybrid search, and when reranking helps.
2. Build a relevance set and compute recall, mean reciprocal rank, and nDCG, then run retrieval experiments one change at a time.
3. Design retrieval for many tenants: filtering with approximate indexes, deletion, and index rebuilds.
4. Explain prefill and decode, the KV cache, batching, quantization, and speculative decoding well enough to read a serving benchmark.
5. Compare hosted, managed, and self-hosted models on quality, latency, cost, and data rules, using your own evaluation set.
6. Produce a retrieval evaluation report, a serving benchmark, and a model hosting decision record.

23.1 How retrieval finds evidence#

Lexical search and BM25#

Lexical search matches the words in the question with the words in the documents. The standard way to rank results is BM25, a formula from decades of information retrieval research (Robertson & Zaragoza, 2009). In plain terms, it gives a passage a higher score when:

• it contains the question's words, with diminishing returns for repeating them (the tenth “density” adds less than the second);
• those words are rare across the whole collection (matching “CHEM 1210” is more telling than matching “the”);
• the passage isn't padded out (a short passage containing the words beats a long one that mentions them in passing).

Lexical search is fast, needs no model, and is excellent at exact terms: course codes, names, formulas, error messages. It fails on paraphrase. A student asking “why does ice float?” won't match notes that say “solid water is less dense than liquid water”.

Semantic search#

Semantic search converts each passage and each question into an embedding, a list of numbers that places text with similar meaning close together (chapter 7). Search becomes “find the passages whose embeddings are nearest the question's”. It handles paraphrase well and struggles with exact identifiers, which carry little meaning to an embedding model.

Comparing a question against every passage gets slow as the collection grows, so most systems use an approximate nearest neighbor index, which finds very close matches quickly without checking everything. A common type, HNSW, builds a layered graph of passages linked to their neighbors and walks it toward the question. “Approximate” is literal: it can occasionally miss a true nearest neighbor, and you trade speed against that miss rate with the index's settings.

Hybrid search and reciprocal rank fusion#

Because the two kinds of search fail differently, combining them usually beats either alone. The difficulty is that their scores aren't comparable: a BM25 score of 14.2 and a similarity of 0.81 aren't on the same scale.

Reciprocal rank fusion (RRF) avoids comparing scores at all. Each passage gets, from each list it appears in, a contribution of 1 ÷ (k + its rank in that list), and the contributions are added. Cormack, Clarke, and Büttcher introduced it in 2009, fixed k at 60 after finding the exact value wasn't critical, and found that this simple merge beat both the individual rankings and more elaborate fusion methods in their experiments (Cormack et al., 2009).

A small example. A passage ranked 1st by keyword search and 4th by semantic search scores 1/61 + 1/64 ≈ 0.0320. One ranked 2nd by semantic search and absent from the keyword list scores 1/62 ≈ 0.0161. Being found by both methods matters more than being first in one, which is usually what you want.

Chapter 7 already used hybrid search for the Learning Coach. What changes at this level is knowing why it works, so you can tell whether a new corpus (Northfield's slide decks) needs more weight on one side, and testing that instead of guessing.

Reranking#

Both kinds of search score the question and each passage separately, then compare. A reranker reads the question and one passage together and judges how well that passage answers that question. It's more accurate and much slower, so it's used as a second stage: search retrieves a few dozen candidates, the reranker reorders them, and only the best go to the model.

Nogueira and Cho showed the pattern with an early language model used as a reranker over candidates from BM25: it took first place on a public passage-ranking benchmark, improving the previous best result by 27% on its main measure (Nogueira & Cho, 2019). Rerankers have improved a great deal since. The architecture question is unchanged: does the quality gain on your questions justify the added latency and cost on every request? Chapter 16 measured exactly that trade.

flowchart LR
    Q["Question +<br/>student and campus"] --> F["Permission filter:<br/>only passages this student may see"]
    F --> L["Lexical search<br/>(BM25), top 30"]
    F --> S["Semantic search<br/>(embeddings), top 30"]
    L --> RRF["Reciprocal rank fusion"]
    S --> RRF
    RRF --> RR["Reranker:<br/>reorder top 30"]
    RR --> CA["Context assembly:<br/>top 8, deduplicated,<br/>strongest first and last"]
    CA --> M["Model"]

23.2 Measuring retrieval#

Building a relevance set#

You can't improve retrieval you don't measure, and you can't measure it without knowing the right answers. A relevance set is a list of realistic questions, each paired with the passages a careful person judged relevant.

Build it from real use where you're allowed to. The Learning Coach's consented weekly sample (ADR-014, chapter 17) supplied real questions with identifying details removed. Teaching assistants at each campus labeled which passages in the course materials answered each one, using a three-level scale: directly answers, partly helps, not relevant. Aim for at least a hundred questions per distinct kind of corpus, including questions the materials can't answer, so you can also check that retrieval returns nothing convincing for them. Northfield's set started with 140 questions across nine courses.

Keep the set versioned, and don't tune on all of it. Hold back a portion that you look at only when you think you're done, or you'll slowly fit your changes to the questions rather than to students.

Recall, MRR, and nDCG#

Three measures cover most needs. Suppose one question has two relevant passages, A and B, and retrieval returns this top five: C, A, D, E, B.

• Recall at k asks what share of the relevant passages appear in the top k. At k = 3, A is there and B isn't: recall@3 = 0.5. At k = 5, both are: recall@5 = 1.0. Choose k to match how many passages you actually send the model (8 for the tutor).
• Mean reciprocal rank (MRR) looks at the position of the first relevant passage. Here that's position 2, so the reciprocal rank is 1/2. Averaged over all questions, it tells you how high the first useful result tends to appear.
• nDCG (normalized discounted cumulative gain) uses the graded labels and rewards putting the most relevant passages first, discounting relevance found lower in the list and comparing with the best possible ordering (Järvelin & Kekäläinen, 2002). It's the measure to use when “directly answers” and “partly helps” should count differently.

For the tutor, recall@8 is the headline: if the evidence isn't in the eight passages, the model can't use it. nDCG explains whether reranking is earning its cost by putting the best passages near the top.

One change at a time#

Retrieval has many settings: chunk size and overlap, what text is added to each chunk before indexing, how many candidates each search returns, the fusion weights, whether to rerank. Change one, rerun the set, record the result, and keep the change only if it helps on the held-out portion too. An experiment log is a table with one row per run: date, the single change, recall@8, nDCG@8, p95 retrieval latency, and a note.

Agents are very good at running these experiments and very bad at resisting the temptation to change three things at once and report the best number. Ask for one change per run and the full table, including the runs that got worse.

Position in the context#

Finding the evidence isn't the last step. Liu and colleagues found that language models used information in a long context unevenly: performance was highest when the relevant information was at the very beginning or end of the input and dropped significantly when it sat in the middle (Liu et al., 2024). Models have improved since that study, so treat it as a reason to test rather than a fixed rule. The tutor's context assembly places the strongest passage first and the second strongest last, and sends eight passages rather than twenty. Sending more raised recall slightly in Lena's tests, and lowered faithfulness more.

23.3 Retrieval in a multi-tenant system#

Filtering with approximate indexes#

Every tutor search must return only passages the student may see: her own notes, and course materials for classes she's enrolled in at her campus (ADR-007, ADR-012). In a database that's a WHERE clause. With an approximate index, it hides a trap.

pgvector, the PostgreSQL extension the Coach uses for embeddings, documents it plainly: with approximate indexes, filtering is applied after the index is scanned, so a filtered query can return fewer results than asked for. Since version 0.8.0 it can scan more of the index until enough results are found, up to a configurable limit (pgvector, README). Without that, a student whose course materials are a small slice of a large shared index may get three results instead of thirty, and the tutor quietly answers from too little evidence. Nothing errors. Recall just falls for small courses.

The options, roughly in order of effort:

• Turn on iterative scanning and set its limit, then measure recall for the smallest courses specifically.
• Add an ordinary index on the filter column, so the database can choose exact search over a small filtered set.
• Partition the index by campus, so each search runs over one campus's data. This also helps when a campus is large.
• For a dedicated tenant like Northfield (chapter 22), the problem mostly disappears: its index holds only its own data.

The test that catches the trap is a per-segment recall report: recall@8 for the smallest tenth of courses as well as the average.

Freshness and deletion#

An index is a copy of the source documents, and copies go stale. Two directions matter:

• Freshness: a new or edited document should be searchable soon. The ingestion queue (chapter 16) handles this, with a target time from upload to searchable.
• Deletion: a deleted document must stop being searchable, and its embeddings and chunks must be deleted, not just hidden. The Learning Coach promised deletion within seven days (NFR-6). Every place a chunk lives (the main index, any cache of search results, any backup used to rebuild the index) has to honor that promise.

Deletion is the harder one, because forgetting a copy produces no error. The failure clinic in this module is about exactly that.

Changing the embedding model#

Embeddings from different models aren't comparable. Switching embedding models means re-embedding every passage, and until that finishes, old and new vectors can't be searched together.

A safe pattern: build a second, versioned index alongside the first; write new and changed passages to both; backfill the new index from the source of truth; compare the two on the relevance set; switch reads with a flag; keep the old index briefly for rollback; then delete it. Treat it like a database migration (chapter 18), with the same expand-and-contract shape, because that's what it is.

flowchart TD
    A["Create index v2<br/>(new embedding model)"] --> B["Every write goes to both:<br/>new, edited, and deleted passages"]
    B --> C["Backfill v2 from the<br/>live source tables"]
    C --> D{"Checks pass?<br/>relevance set, latency,<br/>v2 IDs ⊆ source IDs,<br/>deletion test on v2"}
    D -- "No" --> C
    D -- "Yes" --> E["Flag: reads switch to v2"]
    E --> F["Watch quality and errors;<br/>v1 kept for rollback"]
    F --> G["Stop writing v1, delete it,<br/>and confirm it's gone"]

23.4 How model serving works#

Prefill and decode#

When a model answers, it works in two phases. In prefill, it reads the whole prompt (instructions, passages, question) at once. In decode, it generates the answer one token at a time, each new token depending on all the ones before it. Leviathan and colleagues summarize the consequence: generating K tokens takes K serial runs of the model (Leviathan et al., 2023).

The two phases show up as two latency numbers:

• Time to first token: mostly prefill. Long prompts, such as eight passages of course notes, make it longer.
• Tokens per second during decode: how fast the answer streams after it starts. Long answers make the total longer.

When someone quotes one “latency” number for a model, ask which one it is and at what prompt and answer length.

The KV cache#

To avoid recomputing everything for each new token, the model keeps intermediate results for every token so far in memory, called the KV cache (key-value cache). It grows with every token of prompt and answer, for every request being served at once. On a GPU, memory, not arithmetic, is often what limits how many requests fit at the same time.

Kwon and colleagues found that earlier serving systems wasted much of that memory, reserving space for each request's maximum possible length and fragmenting what was left. Their PagedAttention technique stores the cache in small blocks, like an operating system's virtual memory pages, and their vLLM serving engine built on it achieves near-zero waste, which lets far more requests share a GPU (Kwon et al., 2023).

Batching#

A GPU is efficient when it does many things at once. Batching runs several requests through the model together. The simple version waits for a batch of requests, runs them all to completion, and only then starts the next batch, so a short answer waits for the longest one in its batch and new requests wait for the whole batch.

Yu and colleagues' Orca system introduced scheduling at the level of each generation step instead: after every step, finished requests leave and waiting requests can join (Yu et al., 2022). Often called continuous batching, the idea is now standard in serving engines, and vLLM's authors describe it as complementary to PagedAttention (Kwon et al., 2023). For you, the practical consequence is that throughput rises with load up to a point, while each request's latency rises too. A serving benchmark must report both, at several levels of concurrency.

Quantization and speculative decoding#

Two more techniques appear in every serving discussion:

• Quantization stores the model's numbers with fewer bits, for example 4 instead of 16, so the model needs less memory and runs faster on the same hardware. Done carelessly it loses quality. Methods such as AWQ protect the small fraction of weights that matter most and reduce that loss considerably (Lin et al., 2024). Whether a quantized model is good enough is, again, a question for your evaluation set.
• Speculative decoding uses a small, fast model to guess several tokens ahead and the large model to check those guesses in one pass. Leviathan and colleagues showed it can speed up generation without changing the outputs (Leviathan et al., 2023). It helps decode speed most when the small model guesses well.

You don't need to implement any of these. You need to know they exist, what they trade, and that a benchmark run without them may understate what self-hosting can do, while a benchmark run with all of them on short prompts may overstate it.

23.5 Hosted, managed, or self-hosted#

Three ways to run a model#

Which column satisfies a data rule like Northfield's is a legal and contractual question, not an engineering one. Your job is to describe the data path exactly (what is sent, where it's processed, whether it's stored or used for training, under which agreement) and let the privacy office decide. Say so in the decision record.

The utilization arithmetic#

A rented GPU costs the same every hour. Per-token pricing spreads idle time across a provider's many customers; self-hosting puts all of your idle time on your own bill. The comparison is simple to write:

• Self-hosted cost per answer ≈ (hourly hardware cost × hours in the month + operating time) ÷ answers in the month.
• Provider cost per answer ≈ average tokens per answer × price per token.

The Learning Coach's traffic is spiky: quiet at night and in summer, intense before exams. Self-hosting has to be sized for the peak and paid for at the average. Chapter 16 showed the whole AI bill at 7,000 students was a few hundred dollars a month. A GPU server large enough for a strong open-weight model at exam-week peaks costs, as an illustrative assumption, several thousand dollars a month before anyone's time. At Northfield's size (about 4,000 students), average utilization would be in the single digits.

The arithmetic flips when traffic is high and steady, when the provider bill is many times the hardware cost, or when a rule leaves no other option.

A model gateway#

As the number of models grows (one for most campuses, perhaps another for Northfield, a smaller one for summaries), put a thin model gateway between the application and the models. It's a small internal service, or a module, that:

• routes each request to a model by tenant and task, from configuration;
• applies timeouts, retries, and fallbacks consistently (chapter 24 goes deeper);
• meters tokens and cost per tenant (chapter 22's cost visibility);
• logs which model and prompt version produced each answer (chapter 17).

The application asks for “a tutor answer for this campus” and never names a model. Swapping one becomes a configuration change behind a flag, evaluated before it's turned on.

Evaluating a model swap#

The golden set and graders from chapters 9, 13, and 17 decide whether a model is good enough. For a swap, add three things: run the candidate on the same retrieved contexts as the current model, so retrieval differences don't muddy the comparison; report the measures per campus and per course type, since a model can be fine on prose and poor on formulas; and include latency at realistic prompt lengths and concurrency, not a vendor's figure.

Architect's decision: how should Northfield's tutor run?#

AI collaboration recipe#

Agents are useful for the tedious parts of retrieval and serving work: running experiments, generating benchmark scripts, summarizing results. Keep the judgments (what counts as relevant, what the data rule means) with people.

1. Draft relevance labels, don't decide them.

For each question in this file, list the ten passages from the attached course materials most likely to answer it, with a one-line reason for each. Don't assign relevance grades. Teaching assistants will grade them.

2. One experiment per run.

Run the retrieval evaluation with exactly one change from the baseline configuration: [the change]. Report recall@8, nDCG@8, MRR, and p95 retrieval latency, overall and for the smallest tenth of courses, on the tuning portion only. Append a row to the experiment log. Don't change anything else, even if you see an improvement.

3. Write the benchmark, state its conditions.

Write a load-test script for the serving endpoint that replays prompts sampled from this file of real prompt lengths (not synthetic short prompts), at concurrency 1, 4, 16, and 32. Report time to first token and tokens per second at the 50th and 95th percentiles, throughput, and errors. Print the model, quantization, serving settings, and hardware at the top of the report.

4. Describe the data path exactly.

From the attached architecture and configuration, describe every place a student's note text goes during a tutor request: each service, what's sent, where it's processed, whether it's stored and for how long, and which agreement covers it. Mark anything you can't confirm from the documents as UNKNOWN. Make no legal conclusions.

Boundaries to state every time:

• “Use only the documents and data provided; mark gaps.”
• “Never send real student text to any service not already in the approved data path.”
• “Report every run, including worse ones.”
• “No conclusions about compliance; list questions for the privacy office.”

Worked example: fixing Northfield's retrieval, then choosing its model#

Week 1: Find where retrieval fails. Lena split Northfield's 140 questions into 100 for tuning and 40 held back. Recall@8 on the tuning set was 0.71. She asked an agent to group the misses by cause, then checked its grouping by hand. Three causes covered most of them:

• Slide decks were split into chunks by a fixed number of tokens, so one slide's title landed in one chunk and its bullets in the next.
• Questions using course codes and formula names (“NaCl lab 4”) were found by keyword search but pushed down by semantic results in the fusion, because the keyword list for those questions was short.
• For courses with few documents, the filtered approximate search returned fewer than eight results.

Week 2: One change at a time. The experiment log, on the tuning set:

Run 5 made things slightly worse and was dropped. On the 40 held-back questions, runs 1 to 4 together scored 0.87, and the smallest tenth of courses reached 0.79. All numbers here are from this illustrative scenario; your own will differ.

The changes went to every campus behind a flag, not just Northfield. At the main university, recall@8 rose from 0.86 to 0.90 with no change to the model. Golden-set faithfulness rose by three points. The reranker added about 170 ms to retrieval at the 95th percentile, well inside the tutor's latency target.

Week 3: Benchmark the models. With retrieval fixed, Sam ran the golden set on the same retrieved contexts for three candidates: the current provider model, the best model in the managed catalog, and an open-weight model served with vLLM in a staging account, 4-bit quantized to fit one GPU.

Sam's first benchmark of the self-hosted model, run by an agent with a default script, reported a time to first token of 0.3 seconds. It had used ten-word prompts. With real tutor prompts of around 4,000 tokens, prefill dominated and the number rose to 2.2 seconds. That's why the recipe insists on real prompt lengths.

Week 4: Data path and decision. An agent drafted the data path description from the architecture; Maya corrected two places where it had guessed at retention, and sent it to Ms. Lindqvist with the three options. Northfield's privacy office reviewed its own cloud contract and confirmed in writing that the managed service in its account and region met its rule. The team recorded ADR-018, configured the gateway to route Northfield's tutor requests to the managed model, and kept the self-hosted setup in a staging account as a tested fallback, to be rerun each term.

The retrieval work took three weeks and improved every campus. The model decision took one, and the difference between the top two models was smaller than the retrieval gain.

Verification: is retrieval finding the evidence, and is the model using it?#

Failure clinic: the index that remembered#

Symptom. In February, a student at the main university emailed support: the tutor had cited a note she'd deleted in December, quoting a paragraph she'd written about a family situation. She was upset, and she was right to be.

Root cause. In January, the team had switched to a better embedding model, following the pattern in 23.3: a second index, dual writes for new and changed passages, and a backfill. The backfill read from the nightly database export, which was faster than reading the live tables. The export used was from early December. Passages deleted after that export were in it. Dual writes copied new uploads and edits to the new index, but deletions weren't part of the dual-write path. When reads switched to the new index, 212 deleted chunks came back.

Every test passed. The relevance set didn't include deleted documents. The deletion test ran against the old index, which was still the configured one when the test suite ran.

Fix.

• Remove the 212 chunks immediately and confirm with an exact-text search across every index and cache. Tell the affected students what happened, as the privacy office advised.
• Backfill only from the live source of truth, never from exports; if an export is unavoidable, replay the deletion log from the export's time before switching.
• Make deletions part of the dual-write path, like any other write.
• Run the deletion test against every index that exists, not just the active one, and add a pre-switch check: the set of chunk IDs in the new index must be a subset of the IDs in the source tables.
• Add “every copy of personal data” to the migration checklist from chapter 18, with the search index, caches, exports, and backups listed by name.

Lesson. An index is a copy of personal data. Every copy must honor deletion, including the ones you make for a week.

Architect's checklist#

• ☐ Retrieval and generation are measured separately, with a relevance set per kind of corpus and a held-out portion.
• ☐ Recall is reported for small segments, not just the average.
• ☐ Retrieval experiments change one thing at a time, and the log keeps every run.
• ☐ Filtered approximate searches are configured to return enough results, and that's tested.
• ☐ Deletion reaches every index, cache, and export, and a test proves it.
• ☐ Embedding model changes follow an expand-and-contract migration with a pre-switch check.
• ☐ Serving benchmarks use real prompt lengths and report time to first token, decode speed, and throughput at several concurrency levels.
• ☐ Hosting choices include the utilization arithmetic, with assumptions labeled.
• ☐ The data path is written down exactly, and the people who own the data rule have confirmed it in writing.
• ☐ Models are reached through a gateway that routes, meters, and logs by tenant.
• ☐ A model swap is evaluated on the same retrieved contexts, per tenant and per content type.

Exercises#

Beginner. The Meal Planner searches a recipe collection. Write ten questions a user might ask, five with exact terms (“gluten-free banana bread”) and five as paraphrases (“something sweet I can bake without wheat”). For each, predict whether keyword search, semantic search, or both would find good recipes, and explain why in a sentence. Then compute recall@3 and reciprocal rank by hand for two made-up result lists.

Intermediate. The Trip Planner answers questions from a library of travel guides. Design a relevance set: how you'd collect questions, who would label them, the grading scale, and how many you need. Then plan three retrieval experiments, each changing exactly one setting, with the measure you'd use to decide and the result that would make you keep the change. Ask an AI assistant to critique the plan, and note which suggestions you accept.

Advanced. The Shop Help Chat serves forty shops from one shared index, and one large shop wants its own model for brand voice. Design the retrieval filtering so no shop ever sees another's documents, with a test for small shops getting too few results. Then write a hosting comparison for the large shop's model (provider API, managed service, self-hosted), including the utilization arithmetic with labeled assumptions, and a decision record with a revisit trigger.

Lab, artifact, and review#

Lab. Measure and improve your Learning Coach's retrieval. Build a relevance set of at least 60 questions with graded labels, hold back a fifth, and record a baseline. Run at least four single-change experiments, keeping the full log. Then benchmark two model options on the same retrieved contexts, using real prompt lengths at two concurrency levels.

Deliverable. Add to your portfolio:

• the relevance set's description and a sample, with the labeling guide;
• the experiment log, including runs that made things worse, and held-out results for the final configuration;
• a serving benchmark report that states its conditions (model, settings, hardware, prompt lengths, concurrency);
• a hosting decision record with the data path written out and the utilization arithmetic.

Review. Give your benchmark report to someone else and ask them to name every condition under which its numbers would be wrong. Add a sentence to the report for each one they find.

Reflection#

• In your system, how would you know if retrieval were quietly returning less evidence for some users than others?
• Where does a copy of your users' data exist that your deletion process doesn't know about?
• Which number in a vendor's model announcement would you most want to rerun on your own data, and why?
• What would have to be true for you to run your own model, and how close is that today?
• When did a change to the evidence matter more than a change to the model in something you've built?

The next module turns from where answers come from to what happens when the things they depend on fail. With five campuses, three model routes, and a reranker in every request, the Learning Coach now had more ways to break, and the first exam week of the spring term was going to find them.

References#

• Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. SIGIR 2009.
• Järvelin, K., & Kekäläinen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4), 422–446.
• Kwon, W., Li, Z., Zhuang, S., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
• Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023.
• Lin, J., Tang, J., Tang, H., et al. (2024). AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. MLSys 2024.
• Liu, N. F., Lin, K., Hewitt, J., et al. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12.
• Nogueira, R., & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085.
• pgvector. pgvector README. GitHub.
• Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333–389.
• Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.

Links checked October 2026.