Broadway
Concurrent, multi-stage data ingestion and processing pipeline framework for Elixir. Broadway provides built-in support for message acknowledgment, batching, backpressure, and fault tolerance when consuming from message queues (AWS SQS, RabbitMQ, Kafka, Google Cloud PubSub). Uses GenStage under the hood for demand-driven processing — producers only emit messages when consumers are ready.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
No external network calls — library only. TLS for message queue connections via producer adapter config. No PII in Broadway itself — security is producer/consumer responsibility.
⚡ Reliability
Best When
You're building high-throughput Elixir data pipelines that consume from message queues (SQS, Kafka, RabbitMQ) and need backpressure, batching, and fault tolerance built in.
Avoid When
You just need background jobs — Oban is simpler for task queues. Broadway's value is in multi-stage pipeline processing of high-volume message streams.
Use Cases
- • Build multi-stage agent data processing pipelines that consume from AWS SQS or Kafka with automatic acknowledgment and retry on failure
- • Process LLM API responses in batches using Broadway's batching support — group agent outputs for batch database writes or downstream API calls
- • Implement backpressure-aware agent ingestion pipelines that adapt to downstream processing speed without dropping messages
- • Create fault-tolerant agent event processing that handles failures gracefully — Broadway retries failed messages and continues processing others
- • Fan out agent work from a single message queue to multiple concurrent processing stages with configurable concurrency per stage
Not For
- • Simple queue consumers without batch processing needs — Oban is better for job queues; Broadway shines for high-throughput pipeline processing
- • Non-Elixir stacks — Faust (Python), Kafka Streams (Java), or Lapine (Ruby) for other languages
- • Real-time streaming analytics — Broadway processes discrete messages, not continuous time-series streams
Interface
Authentication
Broadway is a library — auth for message queue producers (SQS, RabbitMQ, Kafka) handled by respective producer adapters using their native auth mechanisms.
Pricing
Broadway is maintained by Dashbit (José Valim's company) and is Apache 2.0 licensed. Free for all use.
Agent Metadata
Known Gotchas
- ⚠ Message acknowledgment happens automatically — agents must not manually ack/nack messages; returning {:failed, message} from handle_message signals Broadway to route to handle_failed
- ⚠ Batching is optional but requires explicit configuration — batch_size and batch_timeout must be tuned per workload; default batch of 1 eliminates batching benefits
- ⚠ Concurrency is per-stage — processor_concurrency and batcher_concurrency are independent settings; under-provisioning either creates bottlenecks in the pipeline
- ⚠ Broadway pipelines are supervised — pipeline crashes trigger supervisor restart; accumulating state inside processors (ETS, Agent) may be lost on crash
- ⚠ Producer adapters have different ack semantics — SQS uses visibility timeout, RabbitMQ uses channel-level acks; read adapter-specific docs for at-least-once vs exactly-once behavior
- ⚠ Testing Broadway pipelines requires Broadway.Topology in test mode — use Broadway.test_message/3 and Broadway.test_batch/3 to inject test messages without real message queue infrastructure
Alternatives
Full Evaluation Report
Detailed scoring breakdown, competitive positioning, security analysis, and improvement recommendations for Broadway.
Scores are editorial opinions as of 2026-03-06.