Integrating LLMs with Ollama and Spring AI

Introduction

Not every AI use case has to talk to a cloud API. For prototyping, offline development, or handling sensitive data, running a Large Language Model (LLM) locally is often the better choice. Ollama makes this straightforward: it downloads, manages, and serves open-weight models such as Llama, Mistral, Gemma, or Qwen behind a simple HTTP API on your own machine.

Spring AI 2.0 ships with a dedicated Ollama starter that integrates this local endpoint into the familiar ChatClient abstraction. The result: the exact same programming model you would use for OpenAI or Anthropic, but pointing at localhost — no API keys, no per-token billing, no data leaving your machine or data center.

This article walks through a minimal but complete example: a Spring Boot application that sends a prompt to a locally running Ollama model and prints the answer. The full source code is available on GitHub.

What is Ollama?

Ollama is a lightweight runtime that wraps model weights, configuration, and a REST API into a single tool. After installation it exposes an HTTP server on port 11434 and lets you pull and run models with a single command:

1# Install (macOS example)
2brew install ollama
3
4# Start the background service
5ollama serve
6
7# Pull and run a model
8ollama pull qwen3.6
9ollama run qwen3.6

Once the model is pulled, it stays cached locally and is served through the same http://localhost:11434 endpoint that Spring AI will talk to.

Spring AI and Ollama

Spring AI talks to the local Ollama server through its native HTTP/REST API. The Spring Boot application only ever sees the standard ChatClient; the Ollama server sits behind that interface and manages the actual models it serves locally:

Because the connection is a plain HTTP endpoint, the same application can target any model Ollama manages — simply by changing the configured model name.

Project Setup

The example lives in its own Maven module, ollama-client, inside the multi-module spring-ai-examples project.

Parent POM (relevant excerpts):

1<parent>
2    <groupId>org.springframework.boot</groupId>
3    <artifactId>spring-boot-starter-parent</artifactId>
4    <version>4.1.0</version>
5</parent>
6
7<properties>
8    <java.version>21</java.version>
9    <spring-ai.version>2.0.0</spring-ai.version>
10</properties>
11
12<dependencyManagement>
13    <dependencies>
14        <dependency>
15            <groupId>org.springframework.ai</groupId>
16            <artifactId>spring-ai-bom</artifactId>
17            <version>${spring-ai.version}</version>
18            <type>pom</type>
19            <scope>import</scope>
20        </dependency>
21    </dependencies>
22</dependencyManagement>

The module itself needs a single starter:

1<dependency>
2    <groupId>org.springframework.ai</groupId>
3    <artifactId>spring-ai-starter-model-ollama</artifactId>
4</dependency>

This starter auto-configures everything needed to talk to Ollama: an OllamaApi client, an OllamaChatModel, and a ready-to-use ChatClient.Builder.

Configuration

All connection and model settings live in application.yml:

1# ollama-client/src/main/resources/application.yml
2spring:
3  application:
4    name: ollama-client
5  ai:
6    ollama:
7      init:
8        pull-model-strategy: never
9      base-url: http://localhost:11434
10      chat:
11        options:
12          model: qwen3.6
13
14logging:
15  level:
16    io.netty.resolver.dns: 'off'

A few points worth highlighting:

• base-url points at the local Ollama server. Change this if Ollama runs on another host or port.
• chat.options.model selects which pulled model to use for chat requests.
• pull-model-strategy: never tells Spring AI not to download the model on startup. This assumes you have already run ollama pull <model>. Set it to when_missing if you want Spring AI to fetch the model automatically the first time it is used.

Calling the Model

With the starter on the classpath, Spring AI injects a fully configured ChatClient.Builder. A simple CommandLineRunner builds a ChatClient from it and fires off a single prompt at startup:

1@Component
2public class OllamaCmdLineRunner implements CommandLineRunner {
3
4    private static final Logger LOGGER = LoggerFactory.getLogger(OllamaCmdLineRunner.class);
5
6    private final ChatClient client;
7
8    public OllamaCmdLineRunner(ChatClient.Builder builder) {
9        this.client = builder.build();
10    }
11
12    @Override
13    public void run(String... args) {
14        LOGGER.info("Calling Ollama ...");
15        String answer = client
16                .prompt("Where in the world is Carmen Sandiego?")
17                .call()
18                .content();
19        LOGGER.info(answer);
20    }
21}

That is the entire integration. Note that the code contains nothing Ollama-specific — it is the standard Spring AI ChatClient fluent API. Swapping the local model for a cloud provider later is a matter of changing the starter dependency and configuration, not the application code.

The application entry point is a plain Spring Boot app:

1@SpringBootApplication
2public class OllamaClientApplication {
3
4    public static void main(String[] args) {
5        SpringApplication.run(OllamaClientApplication.class, args);
6    }
7}

Running the Example

Make sure Ollama is running and the configured model has been pulled, then start the Spring Boot application. Homebrew's mvn may default to a newer JDK, so export a Java 21 JAVA_HOME first:

1export JAVA_HOME=/path/to/java-21
2
3# Ensure the model is available
4ollama pull qwen3.6
5
6# Run the client module
7mvn spring-boot:run -pl ollama-client

On startup the runner sends the prompt and logs the model's reply:

INFO  --- OllamaCmdLineRunner : Calling Ollama ...
INFO  --- OllamaCmdLineRunner : Carmen Sandiego is a fictional character ...

Next Steps

Want to go a little further? Try these small, self-contained exercises to get a feel for the ChatClient API and Ollama:

1. Pass the prompt as command-line arguments. Instead of the hard-coded question, build the prompt from the args of the CommandLineRunner — join the arguments with a single space and send the result to the model. Then run it, for example: mvn spring-boot:run -pl ollama-client -Dspring-boot.run.arguments="Where in the world is Carmen Sandiego?".

2. Try a different model. Pull another open-weight model (e.g. ollama pull llama3), change spring.ai.ollama.chat.options.model in application.yml, and compare the answers. The application code stays exactly the same.

3. Adjust the temperature of the LLM. Besides configuring it in application.yml, you can set options programmatically per request via OllamaChatOptions:

   1var response = client
   2        .prompt(prompt)
   3        .options(OllamaChatOptions.builder()
   4                .temperature(0.7))
   5        .call();

   Try low (e.g. 0.1) versus high (e.g. 1.0) values and observe how the model's responses become more deterministic or more creative. Per-request options override the defaults from application.yml.

Cloud or Local — Your Choice

The strength of Spring AI's abstraction is that the same ChatClient code works whether the model runs in the cloud or on your laptop. Ollama fits naturally into this picture and is a great fit when you want:

• Privacy — sensitive prompts never leave your infrastructure.
• Cost control — no per-token charges during development and testing.
• Offline capability — build and demo without an internet connection.
• Fast iteration — swap models with a single ollama pull and a config change.

Once your prototype is proven, moving to a hosted model (or offering both as a runtime option) requires only a dependency and configuration swap — the business logic stays untouched.

Further Reading

• Ollama documentation
• Spring AI reference documentation
• Spring AI Ollama chat integration