Engineering
IgnisMQ: Running Thousands of Ephemeral, Isolated Queues on Aerospike at Scale
Shantanu Tiwari20 November, 2026
Over the past decade, PhonePe has expanded rapidly to serve over 730 million registered users and more than 160 million daily active users. Operating at this immense scale continuously reveals unique distributed systems challenges, particularly around workload isolation and handling sudden traffic spikes. Whenever off-the-shelf architectures fell short of our operational demands, we built focused internal solutions to address them. As part of our open-source contributions, we are sharing these battle-tested systems back with the open-source community.
We begin with IgnisMQ, our brokerless message queue designed for a specific challenge: achieving complete workload isolation across thousands of short-lived, high-throughput queues built on Aerospike.

📦 GitHub: https://github.com/PhonePe/ignisMQÂ
đź“– Docs: https://phonepe.github.io/IgnisMQ/
Why Priority Queues Weren’t Enough
PhonePe’s communication platform sends over 2 billion messages every day across hundreds of concurrent campaigns. Some are critical, time-sensitive alerts; some are large user engagement pushes; others are silent background updates.
Like most teams, we didn’t start with thousands of queues. We started with what usually works: priority queues.
We split traffic by criticality:
- High priority: Urgent alerts and transactional messages
- Medium priority: Time-sensitive engagement campaigns
- Low priority: Bulk promotional pushes
While this model seemed effective in theory, it quickly broke down under real-world production demands.
The Reality: Starvation Inside the Same Priority
Priority queues only protect high-priority traffic from low-priority traffic. They don’t protect campaigns from each other.
Here is what kept happening:
- The Traffic Jam: Two campaigns land in the same priority tier. One is a routine push to 50 million users; the other is a critical alert for 50,000 users. The 50,000 messages end up waiting behind millions of others anyway.
- The Blast Radius: A downstream channel or provider slows down for one specific campaign. Its consumer threads hang waiting for responses. Soon, all consumer workers in that priority tier are choked, stalling every other campaign in that bucket.
- Huge Volume Differences: Campaign sizes vary wildly, from tens of thousands to hundreds of millions of messages. A static set of shared queues simply couldn’t handle those swings without constant manual intervention.

Figure 1: Head-of-line blocking in shared priority queues versus workload isolation in IgnisMQ dedicated lanes.
The Realization
Fixing this didn’t mean adding more priority tiers. It meant changing the model entirely:
Give every single campaign run its own isolated queue.
Spin it up on demand when the campaign starts, assign it a dedicated worker budget, and tear it down the moment it drains. If one campaign runs into downstream throttling or moves slowly, no other campaign feels it.
This architectural shift established our core design requirements:
- Cheap creation and deletion: Spin up and tear down thousands of queues a day without metadata bloat or cluster-wide rebalances.
- Complete isolation: A 50M run never competes for resources with a 50K run.
- Independent flow control: Absorb massive ingestion bursts without overwhelming downstream channels.
- Zero message loss: Pod restarts or consumer crashes must not drop or duplicate messages.
- No new infrastructure: We already run Aerospike at scale; we didn’t want another broker cluster to operate.
Why Not Kafka or RabbitMQ?
Both are exceptional brokers, but they are architected for a modest number of long-lived topics and queues. We needed the exact opposite: thousands of short-lived, disposable queues created and destroyed daily.
| Distributed Need | Apache Kafka | RabbitMQ | IgnisMQ (Aerospike) |
|---|---|---|---|
| Queue Lifecycle | Static, long-lived topics | Long-lived queues | Ephemeral (minutes to hours) |
| Scale to 1,000s of Dynamic Queues | Metadata thrashing on controller; severe partition skew | High broker memory (process per queue); metadata churn | Instant; creating a queue costs just writing a record |
| Noisy Neighbor Isolation | Massive campaigns choke partitions and consumer groups | High-throughput bursts trigger broker-wide flow control | Strict isolation; independent concurrency budget per queue |
| Infrastructure Overhead | Dedicated, stateful cluster to run, scale, and tune | Dedicated, stateful cluster to run, scale, and tune | Zero new infrastructure; reuses existing Aerospike clusters |
Running an entirely new broker fleet just to handle transient queues meant massive operational overhead. Instead, we asked a simpler question:
Why run a dedicated broker when we can turn the storage engine we already operate into a brokerless queue?
The Inspiration
Knowing that dedicated brokers weren’t the right fit for this specific access pattern, we started looking for alternative models. While digging through distributed systems papers, we found our inspiration in a 2013 IEEE INFOCOM paper: HBaseMQ: A distributed message queuing system on clouds with HBase (Chen Zhang and Xue Liu).
It introduced a brilliant, pragmatic concept: building a message queue directly on top of an existing key-value or wide-column database.
The learning was clear. If a queue is ultimately just a few pointers and a sequence of records, we didn’t need a heavy, dedicated message broker at all. We could build the queue natively on the high-performance storage we already operate. We took that inspiration, engineered it for sub-millisecond latencies, and built it on Aerospike.
How It Works: Load, Fire, and Sweep
IgnisMQ is a client library, not a separate server cluster. It embeds directly inside your application services. Aerospike acts as the distributed storage and synchronization engine, while ZooKeeper is used solely to elect a leader for lightweight background sweeps.
To prevent hot-key bottlenecks, every queue is horizontally divided into shards. Within each shard, a queue is simply two atomic counters and a sequence of records in Aerospike:
- The Load Pointer: Marks the next available write slot.
- The Fire Pointer: Marks the next message waiting for delivery.

Figure 2: Architecture of IgnisMQ showing Load, Fire, and Sweep operations along with the Sideline Queue flow.
The Three Primitives
1. Load (Publish)
Publishing requires just two fast Aerospike operations:
- Atomically increment the shard’s load pointer.
- Write the serialized payload into {queue}_SHARD_{n}_{slot}.
Because the slot reservation is atomic, thousands of concurrent publishers across different application pods write in parallel without race conditions or locks.

2. Fire (Consume)
Consumers automatically read and process messages from the queue in three steps:
- Claim & Read: Atomically advance the fire pointer to claim and read the next message.
- Process & Delete: If execution succeeds, the message is marked completed and deleted.
- Sideline on Failure: If execution fails or times out, the message moves to the Sideline Queue for background retries.

3. Sweep (Crash Recovery)
What happens if an application node dies after claiming a fire pointer but before completing the message?
Because the fire pointer has already moved forward, normal consumers will not re-read that slot. Rather than scanning millions of rows or inspecting timestamps, the Sweeper recovers abandoned messages purely through pointer arithmetic:
- IgnisMQ periodically checkpoints where each shard’s fire pointer was positioned.
- At regular intervals, the sweeper evaluates each shard by calculating a watermark position: watermark = firePointerBefore(now – sweepDuration). If abandoned messages are detected, it peeks a window of candidate slots batched across shards.
- Any record slot lower than that checkpoint that still exists in Aerospike was abandoned by a crashed worker. The sweeper moves it to the sideline for a controlled retry.
Zero timestamp indexing. Zero table scans. Idle queues cost zero sweep overhead.

The Built-In Console
Because IgnisMQ has no broker, there is no central server to host a management UI. Instead, we ship a built-in console natively inside the library.
Every service that uses IgnisMQ exposes an /ignisConsole endpoint, which provides:
- Cluster View: Reads directly from Aerospike to show all active queues, depth, and handler configurations.
- Instance View: Shows live metrics, active shovels, and thread pool health for the current process.
- Operational Control: Operators can manually trigger a shovel to replay sidelined messages after fixing a downstream API outage, requiring zero deployments.
IgnisMQ in Production: Scale and Proof of Isolation
The true test of IgnisMQ isn’t just throughput, it’s isolation at extreme scale.
At PhonePe, IgnisMQ routinely handles over 2 billion messages daily. When a large campaign starts, it loads hundreds of millions of messages into Aerospike in a matter of minutes. Because loading is simply an atomic increment and a write, producers are never throttled by consumers.


Figure 4: Production metrics displaying publish spikes, handler execution rates, and stable p99 latencies during peak traffic.
Looking at our production Grafana metrics during a peak window, you can see the publish rate spike vertically as a massive campaign is loaded. Right alongside it, the overall handler executions scale up smoothly to drain the backlog at a controlled rate. The most critical takeaway from this dashboard is the p99 handler latency: even while absorbing a 200-million message ingestion burst, the execution latency for other campaigns remains completely flat. A massive volume spike in one queue does not degrade the execution speed of another.
To monitor this dynamic environment, we couldn’t rely on static dashboards alone. Because thousands of these queues are created and destroyed daily, we needed real-time visibility into the shards.

Figure 5: IgnisMQ console displaying real-time cluster queue depths, active consumers, and shard pointer states.
Since there is no central broker to host a management UI, we ship a built-in console natively inside the library. Every service using IgnisMQ exposes an /ignisConsole endpoint. This gives our operators a live, cluster-wide view of every active queue. It shows exactly where the Load and Fire pointers are per shard, the active thread pool health, and the sideline queue depth.
Contributing & Moving Forward
At PhonePe, we are committed to providing excellent experiences to our customers in the safest and most transparent way by using technology to its full potential.
We hope that the journey of IgnisMQ, its design, and its architecture inspires and informs the broader engineering community. If you’re excited by the hard distributed systems problems we’re solving and the technology we’re building, we invite you to join us on this journey by exploring career opportunities at PhonePe Careers.
We strongly encourage you to use our open-source tools and contribute back. We will be happy to review your contributions, collaborate on new capabilities, and answer your questions on GitHub.