LLM Provider Configuration Guide

Overview

All LLM provider configuration is done through environment variables. The system reads these variables at startup and uses them to initialize the appropriate provider.

Environment Variables

Provider Selection

# Select which LLM provider to use
# Options: 'anthropic', 'openai', 'ollama'
# Default: 'anthropic'
AI_ROUTING_PROVIDER=anthropic

How it works:

• Read from process.env.AI_ROUTING_PROVIDER in llm-provider-factory.ts (line 22)
• Falls back to 'anthropic' if not set
• Used in ai-routing-orchestrator.service.ts (line 49)

Provider-Specific API Keys

Anthropic Claude

# Required for Anthropic provider
ANTHROPIC_API_KEY=sk-ant-...

How it works:

• Read from process.env.ANTHROPIC_API_KEY in anthropic-provider.ts (line 14)
• Used to initialize Anthropic SDK client
• Checked in isAvailable() method (line 27) - returns true if key exists

Code location:

// src/services/llm-providers/anthropic-provider.ts
constructor() {
  this.apiKey = process.env.ANTHROPIC_API_KEY;
  if (this.apiKey) {
    this.anthropic = new Anthropic({
      apiKey: this.apiKey,
    });
  }
}

OpenAI

# Required for OpenAI provider
OPENAI_API_KEY=sk-...

How it works:

• Read from process.env.OPENAI_API_KEY in openai-provider.ts (line 14)
• Used to initialize OpenAI SDK client
• Checked in isAvailable() method (line 27) - returns true if key exists and client is initialized

Code location:

// src/services/llm-providers/openai-provider.ts
constructor() {
  this.apiKey = process.env.OPENAI_API_KEY;
  if (this.apiKey) {
    this.client = new OpenAI({
      apiKey: this.apiKey,
    });
  }
}

Ollama (No API Key Needed)

# Ollama does NOT require an API key
# It runs locally and doesn't use authentication

How it works:

• No API key required (runs locally)
• Always considered "available" (line 21 in ollama-provider.ts)
• Connectivity is checked on first API call

Provider-Specific URLs

Anthropic & OpenAI

# No URL configuration needed
# Both use their official API endpoints:
# - Anthropic: https://api.anthropic.com (hardcoded in SDK)
# - OpenAI: https://api.openai.com (hardcoded in SDK)

How it works:

• SDKs handle the base URL internally
• No environment variable needed
• URLs are hardcoded in the respective SDKs

Ollama (Local)

# Optional: Customize Ollama base URL
# Default: http://localhost:11434
OLLAMA_BASE_URL=http://localhost:11434

How it works:

• Read from process.env.OLLAMA_BASE_URL in ollama-provider.ts (line 12)
• Defaults to 'http://localhost:11434' if not set
• Used to construct API endpoint: ${this.baseUrl}/api/generate

Code location:

// src/services/llm-providers/ollama-provider.ts
constructor() {
  this.baseUrl = process.env.OLLAMA_BASE_URL || 'http://localhost:11434';
}

// Later used in invoke():
const response = await fetch(`${this.baseUrl}/api/generate`, {
  // ...
});

Use cases for custom URL:

• Ollama running on different host: OLLAMA_BASE_URL=http://192.168.1.100:11434
• Ollama running on different port: OLLAMA_BASE_URL=http://localhost:8080
• Remote Ollama server: OLLAMA_BASE_URL=https://ollama.example.com

Model Selection

# Optional: Override default model for selected provider
# Defaults are provider-specific:
# - Anthropic: claude-3-5-sonnet-20241022
# - OpenAI: gpt-4-turbo-preview
# - Ollama: llama2
AI_ROUTING_MODEL=claude-3-5-sonnet-20241022

How it works:

• Read in llm-provider-factory.ts getDefaultModel() method (lines 95-108)
• Falls back to provider-specific default if not set
• Used in ai-routing-orchestrator.service.ts (line 50)

Code location:

// src/services/llm-provider-factory.ts
static getDefaultModel(provider?: string): string {
  const providerType = (provider || process.env.AI_ROUTING_PROVIDER || 'anthropic').toLowerCase();
  
  switch (providerType) {
    case 'openai':
      return process.env.AI_ROUTING_MODEL || 'gpt-4-turbo-preview';
    case 'anthropic':
      return process.env.AI_ROUTING_MODEL || 'claude-3-5-sonnet-20241022';
    case 'ollama':
      return process.env.AI_ROUTING_MODEL || 'llama2';
    default:
      return 'claude-3-5-sonnet-20241022';
  }
}

Timeout Configuration

# Optional: Request timeout in milliseconds
# Default: 10000 (10 seconds)
AI_ROUTING_TIMEOUT_MS=10000

How it works:

• Read in ai-routing-orchestrator.service.ts (line 51)
• Used for all providers
• Passed to provider's invoke() method as options.timeout

Code location:

// src/services/ai-routing-orchestrator.service.ts
this.timeout = parseInt(process.env.AI_ROUTING_TIMEOUT_MS || '10000');

// Later used in cotReasoning():
const options: LLMOptions = {
  model: this.model,
  maxTokens: 2000,
  temperature: 0,
  timeout: this.timeout,  // <-- Used here
};

Complete Configuration Examples

Example 1: Anthropic (Default)

# .env file
AI_ROUTING_ENABLED=true
AI_ROUTING_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-ant-api03-...
AI_ROUTING_MODEL=claude-3-5-sonnet-20241022
AI_ROUTING_TIMEOUT_MS=10000

What happens:

1. AI_ROUTING_PROVIDER=anthropic → Factory creates AnthropicProvider
2. ANTHROPIC_API_KEY=... → Provider reads key and initializes SDK
3. AI_ROUTING_MODEL=... → Used when calling LLM
4. No URL needed (SDK handles it)

Example 2: OpenAI

# .env file
AI_ROUTING_ENABLED=true
AI_ROUTING_PROVIDER=openai
OPENAI_API_KEY=sk-...
AI_ROUTING_MODEL=gpt-4-turbo-preview
AI_ROUTING_TIMEOUT_MS=10000

What happens:

1. AI_ROUTING_PROVIDER=openai → Factory creates OpenAIProvider
2. OPENAI_API_KEY=... → Provider reads key and initializes SDK
3. AI_ROUTING_MODEL=... → Used when calling LLM
4. No URL needed (SDK handles it)

Example 3: Ollama (Local)

# .env file
AI_ROUTING_ENABLED=true
AI_ROUTING_PROVIDER=ollama
OLLAMA_BASE_URL=http://localhost:11434
AI_ROUTING_MODEL=llama2
AI_ROUTING_TIMEOUT_MS=30000

What happens:

1. AI_ROUTING_PROVIDER=ollama → Factory creates OllamaProvider
2. OLLAMA_BASE_URL=... → Provider reads URL (defaults to localhost:11434)
3. AI_ROUTING_MODEL=... → Used when calling Ollama API
4. No API key needed (local, no authentication)

Example 4: Ollama (Remote)

# .env file
AI_ROUTING_ENABLED=true
AI_ROUTING_PROVIDER=ollama
OLLAMA_BASE_URL=http://192.168.1.100:11434
AI_ROUTING_MODEL=llama2
AI_ROUTING_TIMEOUT_MS=30000

What happens:

1. Same as Example 3, but connects to remote Ollama server
2. OLLAMA_BASE_URL points to remote host
3. API calls go to http://192.168.1.100:11434/api/generate

Configuration Flow

Startup Sequence

1. Service starts
   ↓
2. AIRoutingOrchestrator constructor called
   ↓
3. Reads AI_ROUTING_PROVIDER (default: 'anthropic')
   ↓
4. Calls LLMProviderFactory.getProvider(providerType)
   ↓
5. Factory checks provider type:
   - 'anthropic' → Creates AnthropicProvider
   - 'openai' → Creates OpenAIProvider
   - 'ollama' → Creates OllamaProvider
   ↓
6. Provider constructor reads its specific env vars:
   - AnthropicProvider → Reads ANTHROPIC_API_KEY
   - OpenAIProvider → Reads OPENAI_API_KEY
   - OllamaProvider → Reads OLLAMA_BASE_URL
   ↓
7. Provider checks isAvailable():
   - Anthropic: Checks if ANTHROPIC_API_KEY exists
   - OpenAI: Checks if OPENAI_API_KEY exists
   - Ollama: Always returns true
   ↓
8. If available, provider is cached and returned
   ↓
9. If not available, factory tries next available provider

Environment Variable Summary

*Required only if using that specific provider

Where Variables Are Read

1. Provider Selection

• File: src/services/llm-provider-factory.ts
• Line: 22
• Variable: process.env.AI_ROUTING_PROVIDER
• Default: 'anthropic'

2. Anthropic API Key

• File: src/services/llm-providers/anthropic-provider.ts
• Line: 14
• Variable: process.env.ANTHROPIC_API_KEY
• Required: Yes (for Anthropic provider)

3. OpenAI API Key

• File: src/services/llm-providers/openai-provider.ts
• Line: 14
• Variable: process.env.OPENAI_API_KEY
• Required: Yes (for OpenAI provider)

4. Ollama Base URL

• File: src/services/llm-providers/ollama-provider.ts
• Line: 12
• Variable: process.env.OLLAMA_BASE_URL
• Default: 'http://localhost:11434'

5. Model Selection

• File: src/services/llm-provider-factory.ts
• Line: 100, 102, 104
• Variable: process.env.AI_ROUTING_MODEL
• Default: Provider-specific

6. Timeout

• File: src/services/ai-routing-orchestrator.service.ts
• Line: 51
• Variable: process.env.AI_ROUTING_TIMEOUT_MS
• Default: '10000'

Automatic Provider Selection

If the requested provider is not available, the system automatically falls back:

// From llm-provider-factory.ts (lines 48-59)
if (!providerInstance.isAvailable()) {
  const availableProviders = this.getAvailableProviders();
  if (availableProviders.length === 0) {
    throw new Error('No LLM providers are configured');
  }
  // Automatically use first available provider
  return this.getProvider(availableProviders[0]);
}

Example:

• Requested: AI_ROUTING_PROVIDER=openai
• But: OPENAI_API_KEY not set
• Result: Falls back to Anthropic (if ANTHROPIC_API_KEY is set) or Ollama

Setting Environment Variables

Option 1: .env File (Recommended)

Create .env file in project root:

# .env
AI_ROUTING_ENABLED=true
AI_ROUTING_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-ant-...
AI_ROUTING_MODEL=claude-3-5-sonnet-20241022

Note: Make sure .env is in .gitignore to avoid committing secrets!

Option 2: System Environment Variables

# Linux/macOS
export AI_ROUTING_PROVIDER=openai
export OPENAI_API_KEY=sk-...
npm run dev

# Windows (PowerShell)
$env:AI_ROUTING_PROVIDER="openai"
$env:OPENAI_API_KEY="sk-..."
npm run dev

Option 3: Docker Compose

# docker-compose.yml
services:
  haya-routing:
    environment:
      - AI_ROUTING_ENABLED=true
      - AI_ROUTING_PROVIDER=anthropic
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - AI_ROUTING_MODEL=claude-3-5-sonnet-20241022

Option 4: Kubernetes Secrets

# kubernetes-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: llm-secrets
data:
  ANTHROPIC_API_KEY: <base64-encoded-key>

Validation and Error Messages

Missing API Key

Error: "ANTHROPIC_API_KEY is not configured" or "OPENAI_API_KEY is not configured"

Solution: Set the appropriate API key in environment variables

No Providers Available

Error: "No LLM providers are configured. Please set one of: ANTHROPIC_API_KEY, OPENAI_API_KEY, or configure Ollama"

Solution: Configure at least one provider

Ollama Connection Failed

Error: "Ollama API error (Connection refused)"

Solution:

1. Make sure Ollama is running: ollama serve
2. Check OLLAMA_BASE_URL is correct
3. Verify Ollama is accessible

Summary

✅ All configuration is via environment variables
✅ Provider selection: AI_ROUTING_PROVIDER
✅ API keys: ANTHROPIC_API_KEY or OPENAI_API_KEY
✅ URL: Only needed for Ollama (OLLAMA_BASE_URL)
✅ Model: AI_ROUTING_MODEL (optional, has defaults)
✅ Timeout: AI_ROUTING_TIMEOUT_MS (optional, default: 10000ms)

No code changes needed - just set environment variables and restart the service!