What Broke When BrandMinder Processed 25,000+ Mentions: Lessons in Celery Queues

When building BrandMinder, our initial prototype worked beautifully in development: one user asked a question, 4 AI agents ran sequentially in Python, and output appeared in 15 seconds.

Then we ran our first batch test processing 25,000+ brand mentions across multiple channels. Everything broke.

### 1. What Went Wrong
1. **HTTP Thread Starvation:** Synchronous LLM calls held open WSGI worker threads, causing HTTP 504 timeouts.
2. **API Rate Limiting:** Chained agent calls hit OpenAI/Gemini tier limits simultaneously, causing unhandled cascade failures.
3. **JSON Output Schema Drift:** A single hallucinated trailing comma in an agent response broke downstream processing.

### 2. The Production Refactor
- **Asynchronous Task Graph:** Migrated all multi-agent orchestration to Celery with Redis as the message broker.
- **Exponential Backoff Retries:** Wrapped all upstream LLM calls with automated jittered retry decorators.
- **Pydantic Guardrails:** Enforced strict Pydantic JSON schemas on every agent output boundary. If an agent returned malformed JSON, a self-correcting repair prompt executed automatically before passing state forward.

### 3. Key Takeaway
AI agents are only as reliable as the distributed queues and schema contracts that contain them.