Insurance CRM High-Concurrency Webhook Runbook

Key Architectural Outcomes & Takeaways

✓ Zero-lead-loss ingestion pipeline capable of absorbing 5,000+ simultaneous webhook events per minute
✓ Automated deduplication and idempotency validation using client SHA-256 fingerprinting
✓ Step-by-step triage protocol for clearing consumer backlogs in Redis BullMQ queues
✓ Safe, scripted automated replay from Dead Letter Queues (DLQ) without duplicate lead creation

1. System Topology & SLA Commitments

During Open Enrollment (November 1 through January 15), inbound quote requests and carrier status webhooks surge by over 800%. Our architecture guarantees:

Max Ingestion Latency: < 250ms HTTP 202 Accepted response to external lead sources.
Data Loss Tolerance: 0.00% (All events persist to disk-backed Redis / WAL before acknowledgment).
Processing SLA: 99.5% of leads parsed, deduplicated, and routed to advisory CRM within 60 seconds.

CODE

  +----------------+     +-------------------+     +---------------------+
  | Inbound Leads  | --> | Cloudflare Worker | --> | Ingestion API Tier  |
  | & Carrier Webh |     | (WAF & SSL Term)  |     | (Fastify Node.js)   |
  +----------------+     +-------------------+     +---------------------+
                                                              |
                                                              v (Atomic RPUSH)
                                                   +---------------------+
                                                   | Redis Master Cluster|
                                                   | (BullMQ Queue)      |
                                                   +---------------------+
                                                              |
                                      +-----------------------+-----------------------+
                                      |                                               |
                                      v (Parallel Workers)                            v (Exhausted Jobs)
                           +----------------------+                       +-----------------------+
                           | Advisory CRM Syncer  |                       | Dead Letter Queue     |
                           | & Carrier Dispatch   |                       | (DLQ Inspection)      |
                           +----------------------+                       +-----------------------+

2. Severity Matrix & Incident Triggers

Severity	Condition	Primary Impact	PagerDuty Target
SEV-1	BullMQ backlog > 2,500 jobs OR DLQ inflow > 25/min	High lead latency (> 5 min); dropped carrier webhooks	On-Call Lead Architect
SEV-2	Ingestion API 5xx rate > 2.0% over 5 minutes	Inbound lead rejection; upstream carrier webhook retries	Infrastructure Team
SEV-3	Redis memory utilization > 80%	Risk of eviction or OOM crash	Platform Engineer

3. Triage & Incident Resolution Procedure

Step 3.1: Check Queue Depths and Consumer Health

Execute via internal diagnostic shell:

BASH

# 1. Connect to Redis Cluster
redis-cli -h redis-prod.internal.askthehim.com -p 6379 -a "$REDIS_AUTH_TOKEN"

# 2. Check current queue lengths
HGETALL bull:insurance-leads:meta
LLEN bull:insurance-leads:wait
LLEN bull:insurance-leads:failed

# 3. Check memory pressure and eviction status
INFO memory

Step 3.2: Scale Consumer Worker Replicas

If bull:insurance-leads:wait is growing faster than processing throughput, scale the worker deployment:

BASH

# Scale worker pool from 3 to 10 pods
kubectl scale deployment/lead-consumer-worker --replicas=10 -n production

# Monitor real-time pod initialization
kubectl get pods -n production -l app=lead-consumer-worker -w

4. Idempotent Ingestion & Deduplication Code

To prevent duplicate client records when lead aggregators retry HTTP requests, every job calculates an idempotency digest:

TYPESCRIPT

// src/workers/lead-ingestion/idempotency.ts
import crypto from 'crypto';
import { Redis } from 'ioredis';

export async function isDuplicateSubmission(
  redis: Redis,
  payload: { email: string; phone: string; quoteType: string }
): Promise<boolean> {
  const normalizedPhone = payload.phone.replace(/\D/g, '');
  const normalizedEmail = payload.email.trim().toLowerCase();
  
  const rawFingerprint = `${normalizedEmail}:${normalizedPhone}:${payload.quoteType}`;
  const hash = crypto.createHash('sha256').update(rawFingerprint).digest('hex');
  const cacheKey = `idempotency:lead:${hash}`;

  // Set key if not exists (NX) with 72-hour expiration (259,200 seconds)
  const acquired = await redis.set(cacheKey, 'PROCESSED', 'EX', 259200, 'NX');

  return acquired === null; // true if already exists (duplicate)
}

5. Dead Letter Queue (DLQ) Safe Replay Script

When downstream carrier endpoints recover from an outage, replay failed jobs from the DLQ safely using this automated script:

TYPESCRIPT

// scripts/replay-dlq.ts
import { Queue } from 'bullmq';

async function replayFailedJobs() {
  const queue = new Queue('insurance-leads', {
    connection: { host: process.env.REDIS_HOST, port: 6379 }
  });

  const failedJobs = await queue.getFailed(0, 1000);
  console.log(`Found ${failedJobs.length} failed jobs in DLQ.`);

  for (const job of failedJobs) {
    console.log(`Retrying Job ID ${job.id} (Failed Reason: ${job.failedReason})`);
    await job.retry();
    // Throttle replay to 20 jobs/sec to avoid hammering downstream CRM
    await new Promise(r => setTimeout(r, 50));
  }

  console.log('Replay batch complete.');
  await queue.close();
}

replayFailedJobs().catch(console.error);

6. Post-Incident Review Checklist

Verify all replayed DLQ leads exist in CRM database with status verified.
Confirm no duplicate communication was sent to applicants.
Rotate Redis credentials if authentication logs report abnormal brute-force attempts.
Scale worker replica count back to baseline (3 pods) once wait queue returns to 0.

Continuous Verification Protocol

This document was validated against current carrier API specifications, CMS guidelines, and IRS statutes as of November 1, 2026. Consultations with Charles Boyd are complimentary and independent.

Incident Response & Failover Runbook: High-Concurrency Insurance Lead & Webhook Processing Pipeline