From RAG Prototype to Deployment Decision: A CFPB Case Study從 RAG 原型到部署決策：CFPB 案例研究

Couple years ago, if you ask an AI chatbot a question, you may encounter the situation like below:

User: "What is our Q3 revenue?"
LLM: "I don't have access to that information." ← or hallucinates

Although large language models (LLMs) are powerful reasoning engines, they suffer from three fundamental limitations:

• their knowledge is frozen at a past training cutoff,
• they lack access to private enterprise data, and
• they hallucinate plausible-sounding falsehoods when they encounter a knowledge gap.

Retrieval-augmented generation (RAG) directly addresses these flaws. The core insight of RAG is that instead of baking all knowledge into static, expensive model weights, the system should retrieve current, relevant facts at request time and inject them directly into the model's context. A query that once yielded a hallucination now produces a grounded, verifiable answer:

User query → retrieve Q3 financial report chunks → inject into prompt
LLM: "According to the Q3 report, revenue was $42M, up 18% YoY."

However, missing knowledge is only one reason an application might fall short. Ambiguous instructions might just need a clearer prompt, while inconsistent model behavior might require fine-tuning. Choosing among prompting, fine-tuning, and RAG must therefore begin with the business problem, not the architecture.

This article breaks down that decision from first principles and tests it through a case study using the US Consumer Financial Protection Bureau (CFPB) complaint database. The proposed application classifies and summarizes consumer complaints while providing supporting evidence. The goal is not simply to prove that a RAG pipeline can be built, but to determine whether it improves complaint triage enough to justify its cost and risk. The article is structured as:

• Part 1: Prompting, Fine-Tuning, and RAG
• Part 2: How a RAG System Works
• Part 3: RAG Evaluating the RAG Application
• Part 4: CFPB Complaint Data Case Study

Part 1: Prompting, Fine-Tuning, and RAG

All three approaches can improve an LLM application, but they manipulate completely different levers:

• Prompting changes the instruction. It clarifies the task, supplies a small amount of context (e.g. few-shot learning), and defines the expected response.
• Fine-tuning changes model behavior. It updates underlying parameters so that the model reproduces patterns, tones, or structures demonstrated in training examples.
• RAG changes the evidence available at inference time. It retrieves relevant records from an external collection and supplies them to the model for the current request.

This distinction is far more useful than treating them as three competing features.

A useful decision sequence is therefore:

Can a clearer prompt solve the problem?
    ├─ yes → keep the prompt and measure it
    └─ no
        ↓
Is the missing capability knowledge or evidence?
    ├─ yes → test retrieval (RAG)
    └─ no
        ↓
Is the required behavior stable, repeated, and represented by good examples?
    ├─ yes → evaluate fine-tuning

    └─ no  → revisit the task, workflow, or model choice

This is not a forced choice. A production system can use a carefully designed prompt, retrieve current evidence, and run a fine-tuned model that has learned a stable output contract. The components remain valuable for different reasons. The companion article on LLM fine-tuning examines the training side in detail.

Part 2: How a RAG System Works

A production RAG application is not a single script; it is a system with two distinct data paths. The offline pipeline prepares the searchable knowledge base, while the online pipeline handles the live inference request.

The Offline Pipeline: Building the Knowledge Base

1. Loading and Chucking: Define the unit of meaning

The offline path prepares the data. This pipeline must be deterministic: given a source snapshot, it should produce the exact same searchable records every time.

A useful record contains:

• stable source and document identifiers;
• the text presented to retrieval;
• metadata used for filtering or access control;
• source timestamps;
• a content hash for deduplication;
• parser, chunker, and embedding-model versions.

Chunking is arguably the most impactful representation decision in a RAG system. If a chunk is too large (like embedding an entire 200-page PDF), the resulting vector averages out too many concepts, and the relevant paragraph gets buried in noise. If a chunk is too small (like a single sentence), it loses surrounding meaning—"The rate was 5%" is useless if the retriever stripped away what the rate actually applies to.

To balance precision and context, engineering teams typically evaluate four chunking strategies:

• Fixed-Size Chunking (The Baseline): Splitting text by a strict token count with a sliding overlap so that sentences spanning a boundary are not lost. Given the precision-versus-context tradeoff above, the sweet spot is typically 256–512 tokens with a 10–20% overlap, avoiding the pitfalls of extreme chunk sizes.

Document: [----512 tokens----][----512 tokens----][----512 tokens----]
                        [64 tokens]          [64 tokens]
                          overlap              overlap

• Document-Aware Chunking: Respecting the natural hierarchy of the file. This means chunking PDFs by section headers, code by functions or classes, HTML by <section> tags, and keeping tabular data structurally intact. Arbitrary token splits can slice a table mid-row, completely breaking the data's relationship.

• Semantic Chunking: Splitting text dynamically at the exact point where the embedding similarity between sentences drops. While more computationally expensive during ingestion, this ensures chunks align with actual topic boundaries rather than arbitrary token counts.

• Parent-Child Chunking (Small-to-Big): A pattern that decouples the retrieval unit from the generation unit. The system embeds small "child" chunks (e.g., 128 tokens) for hyper-precise vector matching, but when a match is found, it returns the larger "parent" chunk (e.g., 1024 tokens) to the LLM to provide rich context.

Indexing:
Parent chunk (1024 tokens) → store as context
↓ split into
Child chunks (128 tokens each) → embed and index for retrieval

Query time:
1. Retrieve top-k child chunks (precise embedding match)
2. Look up their parent chunks
3. Send parent chunks (full context) to LLM

2. Embedding and Vector Store: Build a Reproducible Corpus

Once chunks are defined, they are passed through an embedding model and loaded into a vector store.

Query: "What is machine learning?" → [0.23, -0.41, ..., 0.12]

"Explain ML to me"      → [0.25, -0.39, ..., 0.11] ← similar vector
"Paris is in France"。   → [-0.91, 0.34, ..., 0.55]  ← different vector

The most critical rule of RAG is that the exact same embedding model must be used at both indexing and query time. Embedding spaces are model-specific; you cannot compare an OpenAI vector to a Cohere vector. Therefore, changing your model later requires rebuilding the entire index.

A vector store is a specialized database designed to house these high-dimensional arrays alongside their original text chunks and metadata. Unlike traditional relational databases that retrieve records via exact keyword matches (SQL), vector stores retrieve information by mathematical proximity—finding the stored vectors that are closest in distance to the user's query in semantic space.

Calculating exact cosine similarity across millions of 1536-dimensional vectors per query is too slow for production. Vector databases solve this using Approximate Nearest Neighbor (ANN) algorithms—most commonly HNSW (Hierarchical Navigable Small World).

Storing the vector and the text is not enough. A searchable index is a liability if you cannot debug it. Every record requires metadata (source IDs, timestamps, content hashes, and chunker versions) that makes targeted deletion and incident response possible. Furthermore, the embedding contract is strict: if you upgrade your embedding model, you must rebuild the entire index, because vectors from different models cannot be compared.

The Online Pipeline: Serving the Request

A RAG response is only as good as the evidence retrieved, the context assembled, and the model's ability to use that context. Each layer requires its own test.

While the offline pipeline is measured in indexing throughput and reproducibility, the online pipeline is measured in end-to-end latency (typically 100–500 ms), retrieval precision, and factual grounding.

3. Query Embedding & Retrieval

First-stage retrieval must evaluate millions of candidate passages within tens of milliseconds. Production architectures achieve this by pairing lexical matching with vector approximations, ultimately combining them into a hybrid system.

Sparse Retrieval: BM25 (Lexical Exact Matching)

Sparse retrieval ranks documents using pure lexical evidence. It is exceptionally strong when exact language matters, such as searching for a specific product name, an error code, or a distinctive phrase.

The most common algorithm, BM25, evaluates term frequency and rarity. In simple terms:

• The score increases when the query term appears frequently in a document (high term frequency) or when the term is rare across the entire corpus (high inverse document frequency).
• The score decreases when the document is excessively long, or if the query term is ubiquitous across all documents (e.g., "the", "is", "and").

Dense Retrieval: Embeddings & Approximate Nearest Neighbors (ANN)

Dense retrieval translates text into continuous geometric vectors. This is achieved using a bi-encoder embedding method, which encodes the query and the candidate document independently into a shared high-dimensional vector space. Because the document vectors are precomputed offline and saved in the vector store, only the user's query needs to be embedded at runtime.

Once the query is embedded, the system must search the vector store. Computing exact cosine similarity across millions of high-dimensional vectors at request time is computationally prohibitive ($O(N \cdot d)$ per query). Production vector databases solve this using Approximate Nearest Neighbor (ANN) indexing algorithms like HNSW (Hierarchical Navigable Small World):

• Build Phase: Constructs a multi-layer graph where upper layers contain sparse, long-range connections (coarse search) and bottom layers contain dense, localized clusters (fine search).
• Search Phase: Traverses the graph from the top down to rapidly home in on candidate neighborhoods, trading a tiny fraction of recall for logarithmic search speed ($O(\log N)$).

Dense vs. Sparse: The Fundamental Tradeoff

Neither method dominates every query. Hybrid retrieval executes both BM25 and dense retrieval in parallel, leveraging the exact-match precision of sparse search and the semantic reach of dense search, and then mathematically merges their ranked lists.

4. Re-Ranking & Fusion

Information retrieval in modern RAG systems hinges on a fundamental engineering tradeoff between computational efficiency and semantic expressiveness. Managing this tradeoff requires pairing two distinct neural architectures into a two-stage funnel:

A bi-encoder searches an entire corpus efficiently because document vectors are precomputed offline. However, it evaluates queries and documents in isolation.

BI-ENCODER (First-Stage Retrieval)

Online Query      → [ Encoder ] → Vector (q) ──┐
                                           ├→ Cosine Similarity → Rank
Offline Document  → [ Encoder ] → Vector (d) ──┘

A cross-encoder processes the query and document jointly through full self-attention layers. This allows every query token to interact directly with every document token.

CROSS-ENCODER (Second-Stage Re-ranking)

[ Query + Document ] → [ Joint Encoder with Cross-Attention ] → Relevance Score

Running a cross-encoder over 1,000,000 documents is computationally impossible (1M forward passes per query $\approx 16\text{ minutes}$), but running it over the top 50 candidates takes $\approx 50\text{--}100\text{ ms}$.

Fusing Hybrid Results

When combining sparse (BM25) and dense (vector) retrieval, systems employ one of two strategies to merge their candidate lists:

1. Reciprocal Rank Fusion (RRF): RRF merges ranked lists without requiring score normalization. It uses relative rank positions rather than raw scores:

Where $R$ is the set of retrievers and $k$ is a smoothing constant (typically $k=60$) that dampens the disproportionate advantage of top-ranked items. RRF rewards documents that perform consistently well across both retrieval paradigms.

Example ($k=60$):

• Document A: Rank 1 in BM25, Rank 5 in Dense $\rightarrow \frac{1}{61} + \frac{1}{65} = 0.0164 + 0.0154 = \mathbf{0.0318}$
• Document B: Rank 3 in BM25, Rank 1 in Dense $\rightarrow \frac{1}{63} + \frac{1}{61} = 0.0159 + 0.0164 = \mathbf{0.0323}$

Document B wins because it maintained a high rank across both retrievers.

2. Linear Score Fusion: Alternatively, raw scores can be normalized to $[0, 1]$ via min-max scaling and combined using a tuned weight factor $\alpha$:

Note: Linear fusion requires careful calibration because BM25 scores are unbounded, while cosine similarities are bounded between -1 and 1.

from sentence_transformers import CrossEncoder
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
# Score each candidate against the query
pairs = [(query, doc) for doc in candidates]

import cohere
co = cohere.Client("api-key")

results = co.rerank(
    query=query,
    documents=candidates,
    model="rerank-english-v3.0",
    top_n=5
)

Prompt: "On a scale of 1-10, how relevant is this passage to the query?
Query: {query}
Passage: {passage}
Score:"

Run for each candidate, sort by score.
Only feasible for small candidate sets (<20) due to cost.

5. Context Assembly & Generation

Retrieving the right passages is only half the battle; those records must be assembled into the model's input window and governed by strict behavioral constraints.

The Information Budget

Context assembly is fundamentally an information-budget problem. The system must decide:

• How many records to include within token limits.
• Which metadata fields the generator is allowed to inspect.
• How passages are formatted, separated, and deduplicated.
• Which text represents trusted instructions versus untrusted user data.

More context is not necessarily safer. Extraneous noise distracts the model, redundant passages inflate confidence artificially, and crucial facts can easily be buried.

Lost in the Middle

LLMs attend better to information at the beginning and end of the context window. Information in the middle is often missed (Liu et al., 2024):

PROMPT ATTENTION CURVE ("Lost in the Middle")
[ System Instructions & Schema ]     ← High Attention / Primacy Effect
[ Chunk #1 (Highest Relevance) ]     ← Strong Attention
[ Chunk #2 (Moderate Relevance) ]    ← Weakest Attention ("Lost in the Middle")
[ Chunk #3 (Moderate Relevance) ]    ← Weakest Attention ("Lost in the Middle")
[ Chunk #4 (High Relevance) ]        ← Strong Attention
[ User Query & Citation Rule ]       ← High Attention / Recency Effect

To counter the "Lost in the Middle" phenomenon:

• Truncate aggressively: Keep only the top 3–5 candidate chunks rather than filling the entire context window.
• Re-order strategically: Place the best evidence at the very beginning and very end of the context block.

Grounding with Citations

Formatting retrieved chunks with clear identifiers allows the model to attribute claims directly to source passages.

Generation Contracts

An instruction like "use only the supplied sources" is a soft suggestion, not a strict safeguard. Production generation requires mechanical controls:

• Typed Output Schema: Enforce structured outputs (e.g., JSON Schema or Pydantic) to prevent downstream parsing errors.
• Deterministic Decoding: Set the temperature to 0.0 for reproducible inference and reliable evaluation.
• Data Separation: Maintain a strict boundary between evidence and references.
  • Evidence: Verifiable, exact quotes extracted directly from the user's current submission.
  • Retrieved References: Historical records or precedents retrieved to explain classification categories.
• Validation & Fallback: Programmatically validate citations after generation. If retrieved evidence scores fall below a minimum threshold, route the request to a human-in-the-loop triage queue or an abstention path rather than risking a hallucinated answer.

6. Advanced RAG Patterns: Dynamic and Agentic Workflows

The linear pipeline—embed, retrieve, insert, generate—is sufficient for basic document lookup. However, enterprise systems often require more dynamic intervention to handle ambiguous user behavior and complex analytical tasks. When standard RAG fails, engineering teams introduce advanced routing, compression, and agentic patterns.

• Query Routing & Expansion (Pre-Retrieval): Users rarely write perfectly optimized semantic queries. Rather than passing a raw, ambiguous prompt (e.g., "error connecting") directly to the vector database, an LLM intercepts it first. an LLM first intercepts it.

  • Routing classifies the intent and directs it to the appropriate system (e.g., routing a statistical query to a SQL database versus routing a policy question to the vector index).
  • Expansion rewrites the query into an optimized format ("database connection error troubleshooting steps") or generates multiple variations to maximize recall. A specialized variant called HyDE (Hypothetical Document Embeddings) prompts the LLM to write a fake, hypothetical answer and embeds that text, which often matches the structural style of the target documents better than the original question.

• Contextual Compression (Post-Retrieval): A standard 512-token chunk might contain only one sentence relevant to the user's prompt. Passing the entire chunk wastes the model's context budget. Contextual compression uses a fast, lightweight model to filter out irrelevant sentences from the retrieved chunks before they reach the final generator, maximizing the information density of the prompt.

• Agentic and Self-RAG (Execution): Linear pipelines fail on complex comparative questions (e.g., "Compare the revenue growth of our top 3 products in Q3 vs Q2").

  • Agentic RAG turns the LLM into an orchestrator that executes multiple iterative searches—first retrieving Q3 data, then Q2 data, then cross-referencing—before generating a final response.
  • Self-RAG gives the model the autonomy to decide if it even needs to retrieve information, evaluate whether the fetched chunks are actually relevant, and iteratively regenerate its answer if it detects that its claims lack grounding in the provided text.

Part 3: RAG Evaluating the RAG Application

An end-to-end score can tell us that a system became worse. It rarely tells us why. RAG evaluation should preserve the boundaries between retrieval, generation, operations, maintenance, and business outcomes.

1. Retrieval Evaluation: Did the system surface useful evidence?

A retrieval test set consists of historical queries mapped to human-verified relevance judgments. Depending on the task, relevance may be binary (relevant/not relevant), graded, or tied to a known source document. Common metrics include:

• Hit Rate@k: A binary measure of whether at least one relevant document appears in the top $k$ results.
• Recall@k: Of all the relevant documents that exist in the corpus, what fraction successfully appeared in the top $k$ results?
• Precision@k: Of the top $k$ documents retrieved, what fraction is actually relevant?
• Mean Reciprocal Rank (MRR): Evaluates how early the first relevant result appears in the ranked list.
• nDCG (Normalized Discounted Cumulative Gain): Measures overall ranking quality when relevance has multiple weighted grades (e.g., "perfect match" vs. "partial match").

Retrieval should also be sliced by rare labels, complaint length, time period, and lexical difficulty. An average dominated by common checking-account cases can conceal a retriever that fails on small categories.

2. Generation Evaluation: Did the model use the context correctly?

Retrieving the right document means nothing if the LLM hallucinates or ignores it. Generation evaluation has several independent dimensions:

• Task correctness: Are the predicted product and issue labels accurate? (Best measured using exact match and Macro-F1, which gives equal weight to all labels, preventing common categories from overwhelming the score).
• Schema validity: Can the JSON/structured response be parsed and consumed safely by downstream APIs?
• Faithfulness: Are all generated claims explicitly supported by the retrieved context?
• Evidence validity: Are the evidence quotations exact, verbatim spans of the submitted complaint? (This can be evaluated via deterministic string-matching).
• Summary coverage: Does the generated summary preserve the material facts of the complaint without inventing new details?
• Abstention quality: Does the system gracefully defer to a human when the retrieved evidence is insufficient?

Product and issue prediction will use exact match plus macro-F1. Macro-F1 gives each label equal weight and therefore prevents common categories from overwhelming the score. Evidence validity can begin with deterministic checks: every quote must occur in the input. Human review or a separately evaluated judge is still needed to determine whether the quote truly supports the claim.

Frameworks like RAGAS formalize dimensions such as faithfulness and context relevance using "LLM-as-a-judge" techniques. However, reference-free automated judgments are directional estimates, not ground truth. They are highly useful for rapid iteration only when calibrated against human review Es et al., 2024.

3. Operational Evaluation: Can the system meet its service objective?

Algorithmic quality is necessary but insufficient. A production service must be evaluated on its infrastructure realities:

• Latency: p50 and p95 response times separated strictly by stage (retrieval, reranking, and generation).
• Throughput: Requests per second under expected concurrent loads.
• Cost: Token counts (input, context budget, and output) translated into cost per accepted response.
• Reliability: Timeout rates, schema parsing failures, and dependency-error rates.

Stage-level measurement matters. If generation consumes 95% of response time, optimizing vector search will not materially improve the user experience.

4. Maintenance Evaluation: Can quality survive change?

A RAG system is a living database. The maintenance scorecard must monitor the health of the underlying data pipeline:

• Data Health: Ingestion delays, failed parsing records, duplicate rates, and source freshness.
• Semantic Drift: Shifts in user query distributions, introduction of new taxonomy values, or retirement of old product labels.
• Operational Agility: The exact time and compute cost required to rebuild, validate, and roll back an index.

Index versions should be tested with the same rigor as model weights. A candidate index must pass the golden retrieval set before promotion, and the previous version must remain instantly recoverable.

5. Business Evaluation: Did the intervention improve the workflow?

Ultimately, the decisive measures live entirely outside the model's architecture. Success is defined by:

• Efficiency: Average analyst handling time and backlog age.
• Quality: Inter-reviewer agreement and the severity/expected cost of incorrect automated routing.
• Adoption: Analyst acceptance rates, correction rates, and rework frequency.
• ROI: The fully loaded cost per completed complaint triage.

The evaluation ladder should progress sequentially: from offline static tests, to shadow operation (running silently alongside humans), to a limited analyst-assistance pilot. Full automation is a separate business decision with a significantly higher evidence threshold.

Part 4: CFPB Complaint Data Case Study

1. Business problem

The CFPB Consumer Complaint Database publishes information about complaints concerning consumer financial products and services. When a consumer opts to publish their narrative, the CFPB takes steps to remove personal information before publication. The Bureau also warns that the database is not a statistical sample and should not be treated as representative of every consumer's experience. The publication process is described in the CFPB's guidance on how complaint data is shared.

The proposed application addresses a narrow workflow:

Input:  an unstructured consumer complaint narrative

Output: suggested product
        suggested issue
        concise summary
        exact supporting quotations
        retrieved precedent complaints
        confidence and abstention status

The application could support several internal users. Quality and operations analysts could use it to classify, summarize, and review individual complaints; risk teams could use aggregated patterns to identify emerging control issues; and communications or public-relations teams could monitor recurring themes that may require a response. The system remains a decision-support tool: it does not determine whether a complaint is valid, decide consumer relief, rank companies, or provide legal advice. For individual cases, the original narrative and supporting evidence remain visible for human review.

The current-workflow hypothesis

Manual triage creates systemic inefficiency. Without an assistance tool, an analyst must read the narrative, navigate a product-and-issue taxonomy, write a summary, and preserve enough rationale for review. Long or ambiguous complaints take more time. Similar cases can be classified differently, creating rerouting and downstream rework.

The RAG value hypothesis

The RAG value hypothesis is that dynamically retrieved, historically labelled complaints provide more useful precedents than a fixed few-shot prompt. If the system also extracts traceable evidence, an analyst can review the suggestion instead of starting from a blank form.

That hypothesis implies measurable outcomes:

• Lower handling time without a higher correction rate;
• Greater agreement on product and issue labels;
• Fewer rerouted complaints;
• Useful summaries and evidence that analysts accept;
• Acceptable latency and cost at the expected volume.

It also establishes a strong baseline requirement. Classification is a supervised-learning problem with existing labels, so RAG must be compared with a simpler classifier rather than only with a weaker prompt. If a classifier routes complaints more accurately and cheaply, retrieval must justify itself through evidence, precedent discovery, adaptability, or an improved workflow.

2. Data Analysis and Offline Pipeline

Our dataset contains 98,185 complaint records received between January 2025 and June 2026. However, just 36.7% (36,063 records) contain a published consumer narrative. The service can learn and be evaluated only on that narrative-bearing subset.

This non-random missingness creates substantial selection bias. Narratives are published only when consumers consent, and recent complaints can appear before the de-identification "scrubbing" process is complete. A model evaluated strictly on published narratives is not validated for the entire intake stream entering the CFPB workflow.

Deduplication Prevents Memory-Leakage

Standard text normalization (whitespace, Unicode, case folding) reveals 1,068 repeated narrative rows within the published subset. If a record enters the retrieval index (train) and its duplicate enters the test set, nearest-neighbor retrieval performance is artificially inflated: the system retrieves the ground-truth answer from effectively identical text rather than generalizing from similar examples.

import hashlib

def narrative_hash(text: str) -> str:
    """Computes SHA-256 hash of normalized text for exact deduplication."""
    canonical = normalize_narrative(text).casefold()
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

After removing duplicates, the dataset is split chronologically. This is stricter than a random shuffle split and simulates the real-world operational requirement: can an index of past complaints help classify new complaints arriving tomorrow?

Imbalanced Label Distribution (The Long Tail)

Product labels are concentrated: the three largest categories account for 74.5% of the training index. The largest product category, "Checking or savings accounts," contributes 36.0% of the training index. Of the 77 unique issue labels observed in training, 37 have fewer than 50 examples, and 19 have fewer than ten.

The evaluation period (Test set) included five complaints whose product–issue pairs never appeared in the historical index; four of their issue labels were entirely new to the taxonomy. These "cold start" examples demonstrate that continuous index updates are not merely a freshness optimization: they are how the application acquires new operational categories.

This severe imbalance has three evaluation and design consequences:

• Macro-F1 is critical. Unlike overall accuracy, Macro-F1 gives rare labels equal weight, preventing dominant categories from masking failure on niche labels.
• Abstention is required. The model must be capable of recognizing when its taxonomy is incomplete or lacks historical precedents for a specific complaint, deferring to a human.
• Index refresh is dynamic. Index updates are functional requirements for acquiring new categories, not just optimization steps.

Labels are Measurable Targets, Not Unquestionable Truth

Product and issue fields make this dataset valuable for evaluation, but they should be understood as operational, non-objective labels. A single narrative may detail multiple events, fit several plausible issue codes, or omit critical details available elsewhere in the investigative process. The evaluation benchmark measures agreement with the published label, not a final, objective determination of the consumer's experience.

Because the ground truth is imperfect, the proposed human-in-the-loop workflow keeps the original narrative visible, emphasizes traceable evidence quotations, and presents the classification as a suggestion rather than a final verdict. Human review is a permanent requirement, not a temporary inconvenience.

Indexing Strategy: Parent-Child Representation

Each complaint remains the definitive parent record, holding the authoritative metadata (product, issue, complaint date, and complaint ID). In the LangChain implementation, this becomes a parent Document whose metadata carries these operational and audit trail fields.

Because some narratives exceed standard embedding context windows, the dense retrieval pipeline employs a "small-to-big" design using Parent/Child relationships:

1. Child Splitting: Long narratives are segmented into overlapping child chunks of at most 1,200 words (with a 100-word overlap).
2. Passage Indexing: The 28,174 training complaints were processed into 28,444 child passages. Only 215 complaints required splitting into more than one passage. Chroma vector stores index these high-density child passages, maximizing semantic matching precision without naive truncation.
3. Parent Retrieval: At retrieval time, the system performs a search across child passages, and the results are programmatically deduplicated back to their parent complaint IDs before the results fusion and reranking stages.

In contrast, BM25 indexes the complete parent narrative text, leveraging its strength with specific keywords across long contexts.

This data architecture enables a relevant passage inside a 5,000-word complaint to match a user query directly (maximizing retrieval precision) while keeping the labels, citations, and complete audit trail anchored strictly at the complaint level for generation and human review.

3. Building the application

The proposed implementation is intentionally lightweight, reproducible, and inspectable. It utilizes:

• LangChain as the core orchestration framework for standardizing Document, Retriever, Prompt, and structured-generation interfaces;
• Chroma as the persistent vector store for dense retrieval;
• rank-bm25 for sparse, lexical retrieval;
• Ollama to serve local quantization of embedding and generator models.

By utilizing local hosting via Ollama, all raw narratives, dense vector embeddings, and generated triage outputs remain entirely on the physical machine running the experiment, eliminating regulatory concerns regarding public cloud API usage.

                          OFFLINE PIPELINE
CFPB CSV → clean → hash/deduplicate → chronological split
                                       ↓
                              parent Documents
                                ├─ full narrative → BM25 retriever
                                └─ child windows → Ollama embeddings → Chroma


                          ONLINE PIPELINE
Narrative → BM25 parent top-50 ───────────────┐
Narrative → Chroma child search → parents   ──────┴─ weighted RRF
                                                        ↓
                                               top-5 precedents
                                                        ↓
                                      typed generation + validation
                                                        ↓
                                           response or safe fallback

Hybrid Retrieval, Unified Identity

The sparse retrieval path subclasses LangChain's BaseRetriever and executes rank-bm25 directly over the complete parent narratives. Its custom tokenizer removes common function words (stop words), standalone numbers, and standard CFPB redaction placeholders (e.g. XXXX) before scoring. Filtering this noise ensures the lexical retriever focuses purely on unique, discriminative terms.

The dense path utilizes LangChain's OllamaEmbeddings integration, running the locally available nomic-embed-text model. It indexes the 28,444 child embeddings and their parent metadata in a persistent Chroma collection. At query time, Chroma returns semantic child passage candidates. The application programmatically aggregates these results up to their definitive parent IDs, ensuring that only unique parent complaints populate the generator’s context window, even if a single, long complaint produces multiple high-scoring child fragments.

Both sparse and dense searches return 50 initial candidates. The ranked lists are combined using weighted Reciprocal Rank Fusion (RRF):

This heavy bias toward dense retrieval (0.9 weight) is not a theoretical assumption; it was selected through rigorous testing against the chronological Development split. Naive 50:50 hybrid fusion diluted stronger dense results in early smoke tests. The Development split was used for this final architectural optimization, keeping the Test set completely frozen.

A cross-encoder reranker is excluded. Reranking remains a candidate future experiment; in this design, every component must demonstrate enough incremental accuracy to justify its latency and maintainability costs. It is not considered a mandatory box in a production RAG diagram.

Generator as Judge: Validation and Fallbacks

The hybrid retrieval process supplies dynamic, labelled few-shot examples to the generator. Each precedent carries its historical, human-verified product and issue label.

A deterministic diagnostic baseline is calculated by performing a rank-weighted vote over the labels of the top-five retrieved precedents. This isolates the distinct accuracy contribution of retrieval itself and provides a robust, safe fallback when the LLM’s generative output fails validation.

The generative path utilizes a LangChain ChatPromptTemplate to keep system policy (grounding instructions), the user's submitted narrative, and the historical examples distinctly segmented. The prompt, populated with the five retrieved precedents, is sent to a local llama3.1 model through ChatOllama. Structured output is bound via with_structured_output() to a Pydantic schema.

The generation model is required to:

1. Extract: A concise summary and one-to-two supporting evidence quotes.
2. Analyze: Either choose the most supported product-issue pair from the provided dynamic precedents or abstain if grounding is insufficient.

The strict response contract is defined as follows:

{
  "product": "Checking or savings account",
  "issue": "Closing an account",
  "summary": "The consumer says the bank closed an account without notice.",
  "evidence": [
    {
      "quote": "The bank closed my account without warning.",
      "reason": "Directly supports the account-closure classification."
    }
  ],
  "retrieved_references": [
    {
      "complaint_id": "...",
      "product": "Checking or savings account",
      "issue": "Closing an account",
      "score": 0.0159
    }
  ],
  "confidence": 0.72,
  "abstained": false
}

The generation pipeline enforces immediate programmatic validation upon response. The generated product and issue must form a valid, observed operational pair. Critically, every evidence quote must pass a strict substring check against the original submitted narrative. The retrieved precedent text is treated strictly as untrusted context examples, never as trusted input data or instructions.

If structured generation fails Pydantic parsing, uses an invalid label, or fabrication fails the substring evidence check, the specific reason is recorded, and the system gracefully falls back to the deterministic retrieval-only diagnostic prediction.

Observability and Service Boundary

The application is structured as a FastAPI service exposing three endpoints:

• GET /healthz verifies process responsiveness.
• GET /readyz confirms a retrieval index is loaded and reports its version.
• POST /v1/analyze executes the hybrid retrieval and validated structured generation pipeline.

All analytic responses include exhaustive diagnostic metadata: embedding and generator model details, index version, total latency, and a granular breakdown of retrieval and generation latency. For example:

{
  "trace": {
    "index_version": "20260825T195722Z",
    "embedding_model": "nomic-embed-text:latest",
    "generation_model": "llama3.1:latest",
    "retrieval_method": "langchain_chroma_bm25_weighted_rrf",
    "retrieval_ms": 684.3,
    "generation_ms": 2511.7,
    "total_ms": 3197.4,
    "fallback_reason": "generation did not return a valid exact evidence span"
  }
}

For a successfully validated generation, fallback_reason is null. Index updates are managed via an atomic promotion pattern: construction writes a complete staging directory with Chroma data and metadata, promotes it only upon success, and refuses to overwrite an active index version.

While this case study is designed with "production-shaped" rigor, it is not a production deployment. Real-world activation would require authentication, RBAC, encrypted storage, concurrency load testing, data retention policies, active monitoring, and an approved human-in-the-loop review workflow before processing sensitive complaint data.

4. Evaluation results

The initial evaluation isolates retrieval quality and label prediction accuracy across the entire 2,855-record Test set. All retrieval methods utilize the same frozen corpus, test records, top-five weighted vote logic, and product–issue hierarchical constraints. Quality metrics cover the complete frozen Test set.

Latency analysis is provided as a separate local-machine diagnostic. It is measured on the first 50 test complaints using the actual application pipeline—including query embedding time, parent aggregation, and RRF calculation. This diagnostic measures the current in-process implementation bottlenecks rather than serving as a general benchmark for the underlying Chroma vector store.

Dense retrieval performs strongly against the standard supervised baseline, outperforming it on product accuracy and both Macro-F1 measures. However, the simpler Naive Bayes classifier still achieves the highest issue accuracy (45.4%) while predicting labels in approximately 0.45 ms per record. For a production operations team prioritizing only overall routing accuracy, Naive Bayes remains a compelling, highly efficient candidate.

The most encouraging signal for RAG is the improvement in the long tail. Chroma dense retrieval increases Issue Macro-F1 from 0.187 to 0.207. This lift confirms the core hypothesis: dynamically selected precedents provide high-quality context for rare and niche operational categories that fixed prompts struggle to represent. In this experiment, hybrid fusion marginally raised overall issue accuracy over dense retrieval but lowered its Macro-F1, indicating it is not universally superior.

Absolute scores remain modest. Neither the dense (0.207) nor hybrid (0.202) Issue Macro-F1 is sufficient for autonomous routing; both require a robust human-in-the-loop review workflow.

Retrieval Recall Defines the Generation Ceiling

The retrieval evaluation uses historical operational labels as weak labels to determine relevance. "Relevant" here means sharing the same product and issue; it is a repeatable benchmark, not a final semantic judgment.

The hybrid retrieval process successfully places a complaint sharing the published label in the first position (Hit@1) 40.2% of the time, and within the top five (Hit@5) 74.4% of the time.

This defines the hard performance ceiling for the generative model. Because the generator is constrained to choose only from the retrieved product–issue pairs, it is physically impossible for it to return the published operational label in the remaining 25.6% of test cases. This "Selection Constraint" is a critical architectural trade-off: forcing the model to select from dynamic examples eliminates taxonomy invention (hallucinated categories), but it transforms retrieval recall from an optimization step into a hard dependency.

Although hybrid retrieval provides the highest quality results, its median search time (p50) jumps significantly from 59 ms to 706 ms. This is a consequence of the current local implementation bottleneck: the in-process BM25 scorer scans all parent documents sequentially. Performance at the tail (p95: 1,870 ms) also varies directly with complaint narrative length. This bottleneck is not an inherent flaw in sparse search. A production deployment test should replace the in-process scorer with a scalable inverted index before judging hybrid latency. In the context of this local-machine evaluation code, dense-only retrieval remains the more credible default choice.

Structured Generation is an Operational Integration Test

The LangChain generative pipeline was exercised via an end-to-end integration test using a live complaint. The user stated that a bank closed a checking account without warning or notice. Retrieval successfully supplied relevant checking-account and account-closing precedents.

ChatOllama returned a structured object that satisfied the JSON schema requirements. However, the generated output failed the extensive post-generation validation checks: the identified evidence quote was not an exact substring of the original narrative. The validator successfully rejected the output, recorded the hallucination failure reason, and automatically reverted to returning the deterministic retrieval baseline with extractive evidence instead.

This integration test demonstrates an essential distinction in production RAG design: typed generation validates the shape of a response; it does not validate the truth of its content. The additional substring and label consistency checks described in Section 3 successfully caught a critical failure that schema validation alone would have masked.

A planned 11-case qualitative comparison was aborted because a local Ollama service call failed to terminate. This is framed as a critical operational test failure. This case study makes no claim about generation accuracy, summary quality, or generation latency. The evaluation runner is now identified as requiring standard operational resilience patterns, which must be implemented before rerunning the comprehensive generation comparison:

• Per-request timeouts;
• Active query cancellation;
• Incremental result persistence (streaming to disk);
• A defined idempotent retry policy.

Finally, human evaluation remains non-negotiable. Programmatic substring validation proves only that a quote was not fabricated; it does not prove that the quotation semantic supports the selected issue or that the summary provides actionable decision support for an analyst.

5. Should this system be deployed?

Not as an autonomous complaint classifier. The current evidence does support a controlled analyst-assistance experiment, but not a production rollout.

The engineering and operational case against immediate automation is concrete:

• Insufficient Accuracy: Overall issue accuracy remains below 50% across all tested systems.
• Weak Tail Performance: The best Issue Macro-F1 is only 0.207, exposing unacceptable performance on rare compliance categories.
• The Retrieval Ceiling: The correct product–issue pair is entirely absent from the top-five retrieved context window for 25.6% of test complaints.
• Taxonomy Drift: Entirely new taxonomy pairs appeared during the test period, proving the system encounters out-of-distribution data.
• Unproven Generation: Generation quality has not yet completed a reliable multi-case evaluation, and the runner lacks proven timeout and recovery paths.
• Unmeasured ROI: Crucial business metrics—summary usefulness, evidence support, handling-time reduction, and correction costs—have not yet been measured with actual analysts.
• Selection Bias: The modelling population strictly excludes complaints without published narratives, meaning the system is untested against the true intake stream.

Deploying despite these gaps would dangerously confuse an inspectable prototype with a validated, bank-grade operating process.

6. Strategic Recommendations and Next Steps

Architectural Pivot: A Composite Pattern

The experiment suggests a more pragmatic architecture than simply "forcing RAG everywhere." A supervised classifier is exceptionally cheap and achieves the highest overall issue accuracy. Dense retrieval improves product accuracy and surfaces excellent precedents. Structured generation excels at producing readable summaries and extracting verbatim evidence.

The next iteration should abandon the pure RAG routing paradigm and test a composite pipeline that assigns each component the specific job it demonstrated best:

1. Supervised Classifier: Calculates broad product and issue probabilities at high speed.
2. Dense Retrieval: Surfaces historical precedents, optionally filtered or boosted by the classifier's probabilities.
3. LLM Generation: Constrained strictly to generating a summary and extracting exact evidence, not predicting an unconstrained operational label.
4. Human Analyst: Reviews the composite dossier to accept, correct, or reject the suggested triage.

The sparse (BM25) branch should remain only if analysts ultimately find its specific precedents materially more useful than dense results. In the current local implementation, its marginal MRR gain penalizes median retrieval latency by roughly 647 ms. An optimized inverted index could change that comparison, so the branch must be optimized and remeasured before final promotion or deprecation.

The Path Forward: Shadow Piloting and Business Validation

A business case must be measured, not manufactured. The next phase must run silently (a shadow pilot) beside the existing manual workflow to establish baseline handling times, label agreement rates, rerouting frequencies, and exact correction costs. Following shadow validation, a limited visible pilot will randomize eligible complaints between the legacy interface and the AI-assisted interface, strictly preserving final human control.

Full promotion requires hard evidence across four operational gates:

1. Quality: Acceptable error severity by product and issue, reliably calibrated abstention rates, and zero unacceptable regressions on tail categories.
2. Human utility: Analysts actively accept or only lightly edit generated summaries and evidence, resulting in a measurable drop in handling time and downstream rerouting.
3. Operations: Latency, throughput, failure recovery, index refresh pipelines, access controls, and audit logging meet all agreed-upon Service Level Objectives (SLOs).
4. Economics: Measured operational savings conclusively exceed the combined costs of compute, infrastructure, human review, maintenance, and expected errors.

The final financial viability will be judged against a strict annual value model:

The inputs for this equation must be derived exclusively from the pilot. Inserting guessed analyst salaries, hypothesized volumes, or assumed time savings would fabricate a business case rather than validate an investment.