Test Script Guide

This guide explains how to use the test script to validate routing behavior and configuration changes.

Overview

The test script (src/scripts/run-tests.ts) executes a comprehensive test suite defined in test-queries.json to validate routing accuracy, access control, and error handling.

Running Tests

Basic Usage

npm test

This runs all tests from test-queries.json against the running service at http://localhost:3000.

Custom API URL

API_URL=http://localhost:3001 npm test

Test File Structure

Location

test-queries.json

Structure

{
  "test_suite": {
    "description": "Comprehensive test queries for all routing scenarios",
    "version": "1.0.0",
    "tenant_id": "tenant-001",
    "app_id": "app-001"
  },
  "test_categories": [
    {
      "category": "Document Search Service - User Accessible",
      "description": "Tests for collections accessible to 'user' role",
      "tests": [
        {
          "id": "test-001",
          "name": "Troubleshooting Guides - User Role",
          "query": "How do I troubleshoot network issues?",
          "roles": ["user"],
          "expected_service": "document-search-service",
          "expected_collection": "troubleshooting-guides",
          "expected_result": "success",
          "expected_confidence_min": 0.3
        }
      ]
    }
  ]
}

Test Case Fields

id: Unique test identifier (e.g., "test-001")
name: Human-readable test name
query: The query string to test
roles: Array of user roles for access control testing
expected_service: Expected service ID (optional, for success cases)
expected_collection: Expected collection ID (optional, for success cases)
expected_result: Expected outcome - "success", "validation_failed", or "no_match"
expected_error: Expected error code (optional, for error cases)
expected_confidence_min: Minimum confidence score (optional, for success cases)

Test Categories

The test suite includes:

Document Search Service - User Accessible: Tests for collections accessible to 'user' role
Document Search Service - Admin Only: Tests for collections requiring admin/it-support roles
SQL Database Service: Tests for SQL database queries
REST API Service: Tests for REST API endpoints
MCP Protocol Service: Tests for Model Context Protocol
Edge Cases and Error Scenarios: Tests for error handling and validation

Test Results

Output Format

🧪 Running Haya Routing Service Tests
=====================================

API URL: http://localhost:3000
Tenant ID: tenant-001
App ID: app-001

Running 25 tests...

Category: Document Search Service - User Accessible
✅ test-001: Troubleshooting Guides - User Role
❌ test-002: Troubleshooting Guides - IT Support Role
   Reason: Expected service "document-search-service" but got "none"
   Got: service="none", collection="troubleshooting-guides", confidence=0.804

=====================================
Test Summary
=====================================
Passed: 14
Failed: 11
Total: 25
Success Rate: 56.0%

Success Indicators

✅ Green checkmark: Test passed
❌ Red X: Test failed (shows reason and actual result)

Failure Reasons

Common failure reasons:

Wrong Service: Query matched different service than expected

Reason: Expected service "sql-service" but got "document-search-service"

Wrong Collection: Query matched different collection than expected

Reason: Expected collection "server-inventory-db" but got "hardware-software-inventory"

Access Denied: Role-based access control denied access

Reason: Expected success but got error: VALIDATION_FAILED

No Match: Query didn't match any service/collection

Reason: Expected success but got error: NO_RESOURCES

Low Confidence: Confidence score below minimum threshold
```
Reason: Confidence 0.25 below minimum 0.3
```

Adding New Tests

1. Add Test Case to test-queries.json

{
  "id": "test-026",
  "name": "New Feature Test",
  "query": "Your test query here",
  "roles": ["admin"],
  "expected_service": "sql-service",
  "expected_collection": "server-inventory-db",
  "expected_result": "success",
  "expected_confidence_min": 0.3
}

2. Run Tests

npm test

3. Verify Results

Check if the test passes and adjust expectations if needed.

Test Validation Logic

Success Cases

For expected_result: "success":

✅ Response must have success: true
✅ Response must have routes array with at least one route
✅ Service ID must match expected_service (if provided)
✅ Collection ID must match expected_collection (if provided)
✅ Confidence must be >= expected_confidence_min (if provided)

Validation Failed Cases

For expected_result: "validation_failed":

✅ Response must have error code VALIDATION_FAILED
✅ Error code must match expected_error (if provided)

No Match Cases

For expected_result: "no_match":

✅ Response must have error code NO_RESOURCES
✅ Error code must match expected_error (if provided)

Debugging Failed Tests

1. Check Service Response

curl -X POST "http://localhost:3000/api/v1/route" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Your test query",
    "tenant_id": "tenant-001",
    "app_id": "app-001",
    "roles": ["admin"]
  }' | jq '.'

2. Verify Hierarchy Data

Check if the expected service/collection exists:

curl "http://localhost:3000/api/v1/hierarchy/services?tenant_id=tenant-001&app_id=app-001" | jq '.'

3. Check Access Control

Verify roles are configured correctly:

curl "http://localhost:3000/api/v1/admin/hierarchy/tree?tenant_id=tenant-001&app_id=app-001" | jq '.data.services[].categories[].collections[] | {id, name, allowed_roles}'

4. Review Logs

Check service logs for routing decisions:

docker-compose logs haya-routing | grep -i "routing\|match\|boost\|penalty"

Common Test Scenarios

Testing Keyword Boosts

To test if keyword boosts are working:

Create test with query containing boost keywords
Verify it matches the correct service
Update boost values in hierarchy-data.json
Re-register and re-run test
Verify improved matching

Testing Access Control

To test role-based access:

Create test with restricted role
Set expected_result: "validation_failed"
Verify access is denied
Test with allowed role
Set expected_result: "success"
Verify access is granted

Testing Error Handling

To test error scenarios:

Create test with query outside domain
Set expected_result: "no_match"
Verify NO_RESOURCES error
Create test with invalid role
Set expected_result: "validation_failed"
Verify VALIDATION_FAILED error

Continuous Testing

After Configuration Changes

Always run tests after:

Updating hierarchy-data.json:
```
./fresh-register.sh
npm test
```

Updating config files:

npm run bootstrap-configs
docker-compose restart haya-routing
npm test

Updating keyword boosts:

# Edit hierarchy-data.json
./fresh-register.sh
npm test

CI/CD Integration

Add to your CI/CD pipeline:

- name: Run Tests
  run: |
    docker-compose up -d
    sleep 10
    ./fresh-register.sh
    npm test

Test Metrics

Success Rate

The test script reports overall success rate:

Success Rate: 56.0%

Target Metrics

Minimum: 80% pass rate for production
Ideal: 90%+ pass rate
Critical: All access control tests must pass

Improving Success Rate

Tune keyword boosts: Increase boost values for services that need stronger routing
Add penalty keywords: Prevent false matches with stronger penalties
Adjust confidence thresholds: Lower threshold if too many rejections
Review hierarchy structure: Ensure collections are under correct services
Check access control: Verify roles are configured correctly

Troubleshooting

Tests Failing After Changes

Verify changes were applied:
- Check hierarchy data was re-registered
- Check configs were bootstrapped
- Restart service if needed

Check for syntax errors:

python3 -m json.tool test-queries.json > /dev/null

Verify service is running:

curl http://localhost:3000/api/v1/health

All Tests Failing

Check service status: docker-compose ps
Check service logs: docker-compose logs haya-routing
Verify hierarchy is registered: Check /api/v1/hierarchy/services
Check Qdrant/Redis: Ensure databases are running

Intermittent Failures

Check confidence scores: May be near threshold
Review semantic matching: Embeddings may need re-indexing
Check cache: Clear cache if needed
Verify consistency: Ensure hierarchy data is consistent

Configuration Guide - How to update configuration
Routing Mechanics Guide - How routing works
API Documentation - API reference

Example: Testing Keyword Boost Changes

# 1. Edit hierarchy-data.json to increase SQL boost_value from 0.5 to 0.6
vim data/hierarchy-data.json

# 2. Validate JSON
python3 -m json.tool data/hierarchy-data.json > /dev/null

# 3. Re-register hierarchy
./fresh-register.sh

# 4. Run tests
npm test

# 5. Check if SQL tests (test-009 to test-013) now pass
# Look for: ✅ test-009: Server Inventory - Developer Role

Happy testing! 🧪