RAG & Embeddings

What is RAG, and What Problem Does It Solve?

An LLM's knowledge is frozen at its training cutoff. It knows nothing about your product catalogue, your order history, your internal documentation, or anything that changed after training. The naive fix is to include your data in the prompt — but a context window has a token limit, and sending an entire product catalogue on every request is expensive, slow, and often impossible.

Retrieval-Augmented Generation (RAG) solves this by storing your data as vector embeddings in a database and retrieving only the relevant chunks at query time. The model gets exactly the context it needs — not everything, just the right thing.

/*
 * BEFORE — stuffing the entire product catalogue into the prompt.
 * Breaks at scale: 50,000 products × 200 tokens each = 10 million tokens.
 * gpt-4o has a 128k token context. This does not fit, costs a fortune,
 * and degrades answer quality as the context grows.
 */
String allProducts = productRepository.findAll().stream()
    .map(Product::toString)
    .collect(Collectors.joining("\n"));

String answer = chatClient.prompt()
    .user("Here are all our products:\n" + allProducts    // token explosion
        + "\n\nCustomer asks: " + customerQuestion)
    .call()
    .content();

/*
 * AFTER — RAG. Retrieve the 5 most relevant product chunks, send only those.
 * Works at 50,000 products, at 5 million products. Cost and latency are
 * proportional to the retrieved context size, not the catalogue size.
 */
String answer = chatClient.prompt()
    .advisors(QuestionAnswerAdvisor.builder(vectorStore)
        .searchRequest(SearchRequest.builder()
            .topK(5)
            .similarityThreshold(0.6)
            .build())
        .build())
    .user(customerQuestion)
    .call()
    .content();
// Spring AI retrieves the 5 most semantically similar product chunks,
// injects them into the prompt automatically, then calls the model.

Embeddings — Text as Vectors

An embedding is a dense numerical vector (typically 768–3072 numbers) that represents the semantic meaning of a piece of text. Two texts that mean the same thing in different words have vectors that are close together in this high-dimensional space. Two texts with unrelated meanings are far apart.

This is the mechanism that makes semantic search possible: you embed the user's question and retrieve the stored texts whose embeddings are closest to it — not by keyword match, but by meaning.

// EmbeddingModel is auto-configured from your Spring AI starter.
// You rarely call it directly — VectorStore uses it internally.
@Service
public class ProductIngestionService {

    private final VectorStore vectorStore;

    public ProductIngestionService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    // Call this once at startup, or whenever products change.
    // VectorStore embeds each document automatically using the configured EmbeddingModel.
    public void ingestProducts(List<Product> products) {
        List<Document> documents = products.stream()
            .map(p -> {
                // The text you embed determines what queries will match.
                // Include all fields a customer might search for.
                String content = String.format(
                    "Product: %s. Category: %s. Description: %s. Price: %s.",
                    p.name(), p.category(), p.description(), p.price());

                return new Document(content, Map.of(
                    "productId",  p.id(),
                    "category",   p.category(),
                    "price",      p.price().toString()
                ));
            })
            .toList();

        vectorStore.add(documents);
        // Each Document is sent to the EmbeddingModel, which returns a vector.
        // The vector + original text + metadata are stored in the vector store.
    }
}

Vector Stores — Where Embeddings Live

Spring AI's VectorStore interface abstracts over different vector databases. You write code against the interface; the implementation is a dependency choice.

PgVector — production setup

<!-- pom.xml -->
<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>

# application.properties
spring.ai.vectorstore.pgvector.index-type=HNSW
spring.ai.vectorstore.pgvector.distance-type=COSINE_DISTANCE
spring.ai.vectorstore.pgvector.dimensions=1536        # must match the embedding model's output size
spring.ai.vectorstore.pgvector.initialize-schema=true # creates the vector_store table on startup

// VectorStore is auto-configured — inject it directly.
@Service
public class ProductSearchService {

    private final VectorStore vectorStore;

    public ProductSearchService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    public List<Document> findSimilarProducts(String query) {
        return vectorStore.similaritySearch(
            SearchRequest.builder()
                .query(query)
                .topK(5)
                .similarityThreshold(0.6)  // 0–1; below this, the result is noise
                .filterExpression("category == 'footwear'")  // metadata filter
                .build()
        );
    }
}

The Full RAG Pipeline

RAG has two phases that run at different times:

/*
 *  INGESTION PHASE (run once, or on data change)
 *
 *  Raw data → chunk → embed → store
 *
 *  Product descriptions are split into chunks (if long), each chunk is sent
 *  to the embedding model, the resulting vector is stored in pgvector
 *  alongside the original text and metadata.
 *
 *  RETRIEVAL PHASE (run on every user query)
 *
 *  User query → embed query → find nearest neighbours → inject into prompt → generate
 *
 *  The user's question is embedded with the SAME model used at ingestion.
 *  The nearest stored vectors are retrieved. Their text is injected into
 *  the prompt context. The model answers using that context.
 */

QuestionAnswerAdvisor — the wiring in one line

@Service
public class ProductAssistantService {

    private final ChatClient chatClient;

    public ProductAssistantService(ChatClient.Builder builder,
                                        VectorStore vectorStore) {
        this.chatClient = builder
            .defaultSystem("""
                You are a product assistant for an e-commerce platform.
                Answer questions ONLY using the product information provided in the context.
                If the context does not contain relevant information, say so clearly —
                do not invent product details.
                """)
            .defaultAdvisors(
                QuestionAnswerAdvisor.builder(vectorStore)
                    .searchRequest(SearchRequest.builder()
                        .topK(5)
                        .similarityThreshold(0.6)
                        .build())
                    .build()
            )
            .build();
    }

    public String answer(String customerQuestion) {
        return chatClient.prompt()
            .user(customerQuestion)
            .call()
            .content();
        // What happens inside .call():
        // 1. QuestionAnswerAdvisor embeds the question
        // 2. Retrieves the 5 most similar product documents from pgvector
        // 3. Injects them into the prompt context
        // 4. Calls the model with the augmented prompt
        // 5. Returns the model's answer
    }
}

Chunking — when documents are long

// A product description of 50 words fits in one document.
// An order history PDF of 20 pages does not — it needs chunking.
// Spring AI provides TokenTextSplitter for this.

@Component
public class DocumentIngestionService {

    private final VectorStore vectorStore;
    private final TokenTextSplitter splitter = new TokenTextSplitter();

    public DocumentIngestionService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    public void ingestPolicyDocument(String documentText, String documentId) {
        Document doc = new Document(documentText,
            Map.of("source", documentId, "type", "policy"));

        List<Document> chunks = splitter.apply(List.of(doc));
        // Each chunk is ~512 tokens with ~100-token overlap between chunks.
        // Overlap ensures context around a concept is not split across boundaries.
        vectorStore.add(chunks);
    }
}

Tuning and Common Pitfalls

Best Practices

✅ Do use the same embedding model for ingestion and retrieval. Different models produce incompatible vector spaces — retrievals will be meaningless.

✅ Do include a similarityThreshold. Without it, the vector store returns the top-K documents regardless of how dissimilar they are — including completely unrelated content that poisons the prompt.

✅ Do store metadata on every document (source ID, category, timestamp) and filter on it. Metadata filters are evaluated before vector similarity — they are free, and they dramatically improve retrieval precision.

✅ Do instruct the model explicitly to answer only from the provided context and to say so when the context does not contain the answer. Without this instruction, the model will blend retrieved context with its training knowledge — producing confident answers that mix real data with hallucinated data.

✅ Do measure retrieval quality separately from generation quality. Log which documents were retrieved for each query. If retrieval is bad, better prompts do not help — fix the embedding and chunking strategy first.

❌ Don't embed entire documents without chunking when documents exceed ~500 tokens. A single embedding for a long document averages out the meaning of the whole — it retrieves the document when any part is relevant, not when the most relevant part is closest to the query.

❌ Don't skip re-ingestion when source data changes. Stale embeddings produce stale answers. Build ingestion into your data pipeline, not as a one-off startup job.

❌ Don't use RAG as a substitute for a proper search index. RAG retrieves by semantic similarity. Exact-match requirements (order ID lookup, SKU search) are better served by a structured query. RAG and traditional search are complementary, not alternatives.

Interview Questions