Message Queues

Reliable async processing at scale

JustKalm uses a robust message queue architecture for async job processing, with intelligent retry logic, priority queues, and comprehensive dead letter handling.

Queue Metrics

Real-time processing statistics

2.8M

Jobs/Day

99.97%

Success Rate

45ms

Avg Latency

Queue Types

Queue Architecture

High Priority

Real-time valuations
Webhook deliveries
Critical alerts
<100ms SLA

Standard Priority

Batch processing
Email notifications
Data sync jobs
<5s SLA

Background

Analytics aggregation
Report generation
Data cleanup
Best effort

Queue Configuration

# queue-config.yaml - Message queue infrastructure

queues:
  # High-priority valuation queue
  valuations_high:
    broker: redis
    url: redis://redis-cluster.justkalm.svc:6379/0
    concurrency: 50
    prefetch: 10
    visibility_timeout: 300
    max_retries: 5
    priority: high
    
  # Standard processing queue
  valuations_standard:
    broker: redis
    url: redis://redis-cluster.justkalm.svc:6379/1
    concurrency: 100
    prefetch: 20
    visibility_timeout: 600
    max_retries: 3
    priority: normal
    
  # Webhook delivery queue
  webhooks:
    broker: sqs
    url: https://sqs.us-east-1.amazonaws.com/123456789/webhooks
    concurrency: 200
    visibility_timeout: 30
    max_retries: 10
    dlq: webhooks_dlq
    
  # Background jobs
  background:
    broker: rabbitmq
    url: amqp://rabbitmq-cluster.justkalm.svc:5672
    exchange: background_jobs
    concurrency: 25
    prefetch: 5
    max_retries: 3
    
# Redis cluster for pub/sub
redis:
  cluster:
    nodes:
      - redis-0.redis.justkalm.svc:6379
      - redis-1.redis.justkalm.svc:6379
      - redis-2.redis.justkalm.svc:6379