Integrating LLMs from Java

What is LLM Integration, and What Problem Does It Solve?

A large language model (LLM) is a neural network trained on text that can generate, summarise, classify, and extract structured data from natural language. From a Java developer's perspective, an LLM is a remote service you call over HTTP — you send a prompt, you receive a completion. The integration problem is not the HTTP call itself; it is everything around it: prompt construction, response parsing, error handling, token limits, provider lock-in, and making the output usable by the rest of your typed, compiled codebase.

Before dedicated libraries existed, integrating an LLM from Java meant writing a raw HTTP client, manually building the provider's JSON payload, parsing the response, and handling every edge case yourself. Every provider (OpenAI, Anthropic, Google, Mistral) has a different request schema, different error codes, and different streaming formats. Switching providers meant rewriting the integration from scratch.

/*
 * BEFORE — raw HttpClient against the OpenAI API.
 * Works, but: provider-specific JSON, manual parsing, no type safety on the
 * response, no streaming, no structured output, no retry, and switching to
 * Anthropic means rewriting everything below.
 */
HttpClient http = HttpClient.newHttpClient();

String requestBody = """
    {
      "model": "gpt-4o",
      "messages": [
        {"role": "system", "content": "You are a product recommender for an e-commerce store."},
        {"role": "user",   "content": "Suggest 3 products related to order #4821"}
      ],
      "temperature": 0.3
    }
    """;

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://api.openai.com/v1/chat/completions"))
    .header("Authorization", "Bearer " + apiKey)
    .header("Content-Type", "application/json")
    .POST(HttpRequest.BodyPublishers.ofString(requestBody))
    .build();

HttpResponse<String> response = http.send(request, HttpResponse.BodyHandlers.ofString());
// Now parse the JSON, extract choices[0].message.content, handle errors,
// handle rate limits, handle token overflow... all manually.

/*
 * AFTER — Spring AI ChatClient.
 * Provider-agnostic, typed, streamable. Switching from OpenAI to Anthropic is
 * a dependency swap + one property change — zero code changes.
 */
@Service
public class ProductRecommendationService {

    private final ChatClient chatClient;

    public ProductRecommendationService(ChatClient.Builder builder) {
        this.chatClient = builder
            .defaultSystem("You are a product recommender for an e-commerce store.")
            .build();
    }

    public String recommend(Order order) {
        return chatClient.prompt()
            .user("Suggest 3 products related to order #" + order.id())
            .call()
            .content();                // returns the model's text response as a String
    }
}

Project Setup

<!-- pom.xml — Spring AI 2.0 with the OpenAI starter -->
<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>2.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
        <!-- version managed by the BOM -->
    </dependency>
</dependencies>

# application.properties
spring.ai.openai.api-key=${OPENAI_API_KEY}           # NEVER hardcode — use env var or secret manager
spring.ai.openai.chat.options.model=gpt-4o
spring.ai.openai.chat.options.temperature=0.3         # lower = more deterministic

ChatClient — The Core API

ChatClient is a fluent facade over the underlying ChatModel. It handles prompt assembly, system messages, call execution, and response extraction. If you have used RestClient or WebClient, the pattern is familiar.

@Service
public class OrderSummaryService {

    private final ChatClient chatClient;

    // ChatClient.Builder is auto-configured by Spring AI — inject it,
    // configure defaults, and build. One ChatClient instance per use case
    // is a clean pattern (each with its own system prompt).
    public OrderSummaryService(ChatClient.Builder builder) {
        this.chatClient = builder
            .defaultSystem("""
                You are an order assistant for an e-commerce platform.
                Summarise the order clearly and concisely for the customer.
                Always include the order total and estimated delivery date.
                """)
            .build();
    }

    // Simple text response
    public String summarise(Order order) {
        return chatClient.prompt()
            .user(u -> u.text("Summarise this order: {order}")
                         .param("order", order.toString()))
            .call()
            .content();
    }
}

Prompt templates with parameters

// Parameters are injected into the prompt template at runtime.
// This is safer and cleaner than string concatenation.
String summary = chatClient.prompt()
    .user(u -> u.text("""
        Customer {name} placed order #{orderId} on {date}.
        Items: {items}
        Generate a confirmation message.
        """)
        .param("name", customer.name())
        .param("orderId", order.id())
        .param("date", order.createdAt().toString())
        .param("items", order.itemsSummary()))
    .call()
    .content();

System prompt vs user prompt

Structured Output — From Text to Records

An LLM returns text. Your Java code needs objects. Structured output bridges that gap: you tell the model to conform to a schema, and Spring AI maps the response directly to a Java record. No manual JSON parsing.

// Define the shape you want as a record
public record ProductRecommendation(
    String productName,
    String reason,
    BigDecimal estimatedPrice
) {}

// Ask the model to return that shape — .entity() handles the conversion
List<ProductRecommendation> recommendations = chatClient.prompt()
    .user("Suggest 3 products related to a customer who bought running shoes")
    .call()
    .entity(new ParameterizedTypeReference<List<ProductRecommendation>>() {});

// Result: a typed List<ProductRecommendation> — no String parsing, no regex,
// no "hope the model returned valid JSON". The records are usable immediately
// by the rest of your service layer.

// Single entity — even simpler
public record OrderClassification(
    String category,           // "return", "complaint", "question", "praise"
    String urgency,            // "low", "medium", "high"
    String suggestedAction
) {}

OrderClassification classification = chatClient.prompt()
    .user("Classify this customer message: \"" + message + "\"")
    .call()
    .entity(OrderClassification.class);

Streaming Responses

A non-streaming call blocks until the entire response is generated — which can take seconds for long completions. Streaming returns tokens as they are produced, enabling real-time display in a UI and reducing perceived latency.

// Streaming — returns a Flux<String> of token chunks
@GetMapping(value = "/api/orders/{id}/summary/stream",
            produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> streamSummary(@PathVariable Long id) {
    Order order = orderService.findById(id);

    return chatClient.prompt()
        .user("Summarise order #" + order.id() + " for the customer")
        .stream()
        .content();        // Flux<String> — each element is a token chunk
}

Key Model Parameters

These parameters control the model's behaviour. Understanding them is the difference between an integration that works in a demo and one that works in production.

# Override per-call via runtime options (Spring AI 2.0)
String result = chatClient.prompt()
    .user("Classify this return request")
    .options(ChatOptions.builder()
        .temperature(0.0)           // deterministic for classification
        .maxTokens(100)             // short response expected
        .build())
    .call()
    .content();

Cost, Latency, and Production Concerns

An LLM call is an external HTTP request to a paid API. It has latency (hundreds of milliseconds to seconds), cost (per token), and failure modes (rate limits, timeouts, model errors) that are fundamentally different from a database query or a local method call. Treating it like an ordinary service call is the most common production mistake.

Best Practices

✅ Do use ChatClient.Builder injection and configure one ChatClient per use case — each with its own system prompt. This keeps the system prompt close to the service that owns the behaviour.

✅ Do pin the model version in production configuration. gpt-4o is an alias that can change without notice; gpt-4o-2024-11-20 is a contract.

✅ Do set max-tokens on every call. An unbounded response can generate thousands of tokens, burning budget and exceeding downstream limits.

✅ Do validate structured output before using it in business logic. The model is not a compiler — it can return nulls, unexpected enum values, or fields that violate your domain invariants.

✅ Do load system prompts from resource files rather than embedding them as Java string literals. Prompts change frequently and should be version-controlled alongside application configuration, not buried inside a service class.

❌ Don't hardcode API keys in source code, application.properties, or test fixtures. Use environment variables or a secret manager.

❌ Don't treat an LLM call like a local method — it has latency, cost, and failure modes. Always set a timeout and have a fallback path.

❌ Don't use the largest model for every task. Classification, extraction, and formatting tasks run faster and cheaper on smaller models. Reserve the largest models for complex reasoning or generation.

❌ Don't concatenate user input directly into prompts without sanitisation. Prompt injection — where user input hijacks the system prompt — is a real attack vector. See Testing AI Systems & Production Risks.

Interview Questions