Monitoring & Observability System

Complete monitoring solution for the Node webhook system with Prometheus, Grafana, and Sentry.

📊 Features

• Real-time Metrics: Track webhooks, queues, workers, and Redis
• Error Tracking: Automatic error capture with Sentry
• Visualizations: Pre-built Grafana dashboards
• Health Checks: API endpoints for load balancers
• Alerts: Configurable alerts for critical issues
• Performance Monitoring: Latency and throughput tracking

🚀 Quick Start

1. Install Dependencies

npm install

2. Configure Sentry (Optional)

# Add to .env.local
SENTRY_DSN=https://your-key@sentry.io/project-id

3. Start Monitoring Stack

# Start Prometheus, Grafana, Redis Exporter
npm run monitoring:start

4. Start Workers

npm run workers

5. Test System

# Run monitoring tests
npm run monitoring:test

🔗 Access URLs

📈 Available Scripts

# Start monitoring stack
npm run monitoring:start

# Stop monitoring stack
npm run monitoring:stop

# View logs
npm run monitoring:logs

# Test monitoring system
npm run monitoring:test

📊 Metrics Collected

Webhook Metrics

• Total webhooks received/processed
• Processing success/failure rates
• Processing duration (p50, p95, p99)
• Enqueue latency
• Duplicate detection

Queue Metrics

• Queue size (waiting jobs)
• Active jobs
• Delayed jobs
• Failed jobs
• Retry counts

Worker Metrics

• Health status
• Restart counts
• Concurrency settings

Redis Metrics

• Connection errors
• Operation latency
• Memory usage

🎯 Key Dashboards

Grafana Dashboard Panels

1. Webhook Receive Rate - Incoming webhooks/second
2. Processing Rate - Success vs failures
3. Queue Size - Jobs waiting by queue
4. Failed Jobs - Failed job counts
5. Processing Duration - Latency percentiles
6. Worker Health - All worker status

Prometheus Queries

# Webhook rate
rate(webhooks_received_total[5m])

# Error rate
rate(webhook_processing_failures_total[5m]) / rate(webhooks_received_total[5m])

# p95 latency
histogram_quantile(0.95, rate(webhook_processing_duration_seconds_bucket[5m]))

🚨 Alerts

Pre-configured alerts in monitoring/alerts.yml:

• High webhook failure rate (> 10%)
• Growing queue size (> 500 jobs)
• High failed job count (> 100)
• Worker unhealthy
• Redis connection errors
• Slow processing (p95 > 30s)

🏥 Health Checks

Detailed Health Check

curl http://localhost:3000/api/health | jq

Simple Health Check

curl http://localhost:3000/api/health?simple=true

📚 Documentation

• Full Monitoring Guide - Complete documentation
• Quick Start Guide - 5-minute setup
• Implementation Details - Technical details

🔧 Configuration

Prometheus

Edit monitoring/prometheus.yml to customize:

• Scrape intervals
• Retention periods
• Target endpoints

Grafana

1. Access http://localhost:3001
2. Login: admin/admin
3. Import dashboard from monitoring/grafana-dashboard.json
4. Configure data sources and alerts

Sentry

# Environment variables
SENTRY_DSN=your-dsn
SENTRY_DEV_MODE=false  # Set true for dev

🐛 Troubleshooting

No Metrics in Prometheus

# Check metrics endpoint
curl http://localhost:3000/api/metrics

# Check Prometheus targets
open http://localhost:9090/targets

Grafana Shows No Data

1. Add Prometheus data source
2. URL: http://prometheus:9090
3. Save & Test
4. Import dashboard

Sentry Not Working

# Check DSN is set
echo $SENTRY_DSN

# Check worker logs
npm run workers | grep Sentry

📦 Architecture

┌─────────────┐
│  Webhooks   │
└──────┬──────┘
       │
       ▼
┌─────────────┐     ┌──────────┐
│   Queues    │◄────┤  Redis   │
└──────┬──────┘     └────┬─────┘
       │                 │
       ▼                 ▼
┌─────────────┐     ┌──────────┐
│   Workers   │     │ Exporter │
└──────┬──────┘     └────┬─────┘
       │                 │
       ▼                 ▼
┌─────────────┐     ┌──────────┐
│ Metrics API │────►│Prometheus│
└─────────────┘     └────┬─────┘
       │                 │
       ▼                 ▼
┌─────────────┐     ┌──────────┐
│   Sentry    │     │ Grafana  │
└─────────────┘     └──────────┘

🎓 Best Practices

1. Monitor Key Metrics

   • Webhook failure rate
   • Queue backlog
   • Worker health
   • Processing latency

2. Set Up Alerts

   • Configure notification channels
   • Tune alert thresholds
   • Create escalation policies

3. Regular Reviews

   • Check dashboards daily
   • Review Sentry errors
   • Analyze performance trends

4. Capacity Planning

   • Track growth trends
   • Plan for scaling
   • Optimize resource usage

🔍 What's Monitored

All 5 Webhook Queues

✅ webhook:shopify
✅ webhook:slack
✅ webhook:outlook
✅ webhook:gmail
✅ webhook:meta

Each Queue Tracks

• Receive rate
• Processing rate
• Success/failure ratio
• Queue depth
• Worker health
• Processing latency
• Error types
• Retry counts

🎯 Success Criteria

System is working correctly when:

• ✅ Metrics endpoint returns data
• ✅ Health check shows "healthy"
• ✅ Prometheus scraping successfully
• ✅ Grafana shows charts
• ✅ Workers report healthy status
• ✅ Errors appear in Sentry

📞 Support

For issues or questions:

1. Check documentation in docs/
2. Review troubleshooting section
3. Check service logs: npm run monitoring:logs
4. Test system: npm run monitoring:test

🔄 Updates

To update the monitoring system:

# Pull latest changes
git pull

# Update dependencies
npm install

# Restart monitoring stack
npm run monitoring:stop
npm run monitoring:start

# Restart workers
npm run workers

Status: ✅ Production Ready
Version: 1.0.0
Last Updated: November 19, 2025