Thanks! That’s a very solid observation and a battle-tested one.
You’re absolutely right ,priority as just a field breaks down under real load. We’ve seen the same failure mode in production: large marketing/batch campaigns flooding the queue and starving latency-sensitive transactional notifications.
I really like the approach of separate Kafka topics per priority:
* `notification.high`
* `notification.normal`
* `notification.low`
This gives a few concrete advantages:
* Hard isolation between transactional and bulk traffic
* Independent consumer groups with different scaling and throughput configs
* Ability to assign more partitions / faster consumers to high-priority topics
* No reliance on consumer-side prioritization, which is brittle under backpressure
In practice, this also simplifies ops:
* High-priority topics can have stricter SLAs and alerts
* Low-priority topics can tolerate lag without impacting user experience
* Backlogs in marketing traffic don’t impact order confirmations, OTPs, etc.
A pattern that’s worked well for us:
* Orchestrator does a routing decision upfront, publishes directly to the right topic
* Channel services subscribe to multiple topics, but with weighted concurrency or even separate deployments for high priority
You can still keep the `priority` field for observability and auditing, but **topic-level separation is the real enforcement mechanism**.
Great call-out , this is exactly the kind of nuance interviewers love because it shows you’ve seen systems fail at scale, not just designed them on paper.
Thanks! That’s a very solid observation and a battle-tested one.
You’re absolutely right ,priority as just a field breaks down under real load. We’ve seen the same failure mode in production: large marketing/batch campaigns flooding the queue and starving latency-sensitive transactional notifications.
I really like the approach of separate Kafka topics per priority:
* `notification.high`
* `notification.normal`
* `notification.low`
This gives a few concrete advantages:
* Hard isolation between transactional and bulk traffic
* Independent consumer groups with different scaling and throughput configs
* Ability to assign more partitions / faster consumers to high-priority topics
* No reliance on consumer-side prioritization, which is brittle under backpressure
In practice, this also simplifies ops:
* High-priority topics can have stricter SLAs and alerts
* Low-priority topics can tolerate lag without impacting user experience
* Backlogs in marketing traffic don’t impact order confirmations, OTPs, etc.
A pattern that’s worked well for us:
* Orchestrator does a routing decision upfront, publishes directly to the right topic
* Channel services subscribe to multiple topics, but with weighted concurrency or even separate deployments for high priority
You can still keep the `priority` field for observability and auditing, but **topic-level separation is the real enforcement mechanism**.
Great call-out , this is exactly the kind of nuance interviewers love because it shows you’ve seen systems fail at scale, not just designed them on paper.