Ollama Quick Start - Llama 3.2

✅ Status

Ollama is now running in Docker on port 11434!

Current Status

# Check if Ollama is running
docker-compose ps ollama

# Check logs
docker-compose logs ollama

# Test API
curl http://localhost:11434/api/tags

Pull Llama 3.2 Model

The model download is in progress. To complete it:

# Option 1: Use the setup script (recommended)
./scripts/setup-ollama.sh

# Option 2: Manual pull
docker-compose exec ollama ollama pull llama3.2

Note: This will download ~2GB and may take 5-10 minutes depending on your internet speed.

Verify Model is Ready

# List available models
docker-compose exec ollama ollama list

# Test the model
docker-compose exec ollama ollama run llama3.2 "Hello, how are you?"

Configure AI Routing to Use Ollama

Add to your .env file:

AI_ROUTING_ENABLED=true
AI_ROUTING_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
AI_ROUTING_MODEL=llama3.2

Then restart the service:

docker-compose restart haya-routing

Test AI Routing with Ollama

curl -X POST http://localhost:3000/api/v1/route/ai \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "tenant-001",
    "app_id": "app-001",
    "roles": ["developer"],
    "query": "Show network device inventory"
  }'

Docker Service Details

Service Name: ollama
Image: ollama/ollama:latest
Port: 11434
Volume: ollama_data (persists models)
Mode: CPU (GPU support available if you have NVIDIA GPU)

Useful Commands

# Start Ollama
docker-compose up -d ollama

# Stop Ollama
docker-compose stop ollama

# View logs
docker-compose logs -f ollama

# Pull a different model
docker-compose exec ollama ollama pull llama3.2:3b  # Smaller, faster

# Remove a model
docker-compose exec ollama ollama rm llama3.2

# Check disk usage
docker system df

Next Steps

✅ Ollama is running
⏳ Wait for Llama 3.2 download to complete
Configure .env with Ollama settings
Restart haya-routing service
Test AI routing endpoint

Troubleshooting

Model download is slow?

Normal - Llama 3.2 is ~2GB
Check your internet connection
Monitor progress: docker-compose logs -f ollama

Container not starting?

Check logs: docker-compose logs ollama
Verify port 11434 is not in use: lsof -i :11434

Out of memory?

Use smaller model: llama3.2:1b or llama3.2:3b
Increase Docker memory limit

For detailed setup, see Ollama Docker Setup Guide