Summary
A complete overhaul of the background job scheduler and database maintenance routines, designed to handle millions of rows without impacting application performance.What’s New
- Orchestrator Command: A single, intelligent command (
jobs:orchestrate) now manages all background tasks, replacing multiple overlapping cron jobs. - Batched Updates: Contribution calculations now use optimized, batched queries, eliminating N+1 query issues and reducing database load by over 99%.
- Time-Budgeted Maintenance: Maintenance tasks (like fingerprinting) are now time-boxed to prevent long-running processes from blocking the database.
- Priority Queues: Implemented a strict priority system (Realtime > Emails > Default > Analytics) to ensure critical tasks run first.
Why It Matters
- Stability: The application remains responsive even during heavy background processing.
- Efficiency: Server resource usage (CPU/RAM) is reduced by ~80% due to fewer concurrent processes.
- Scalability: The new architecture is built to scale with the platform, handling millions of tracks and events efficiently.
Technical Details
- Process Reduction: Reduced from ~6 concurrent processes per minute to 1.
- Query Optimization: Replaced table scans with indexed, cursor-based lookups.
- Backpressure: The scheduler now detects high load and defers non-critical tasks.