Spring Kafka
Spring integration for Apache Kafka — provides KafkaTemplate for producing, @KafkaListener for consuming, and Kafka Streams support. Spring Kafka features: KafkaTemplate.send() for producing records, @KafkaListener(topics='agent-events') for consuming, automatic offset management, consumer group coordination, batch consumption mode, dead-letter topics (DeadLetterPublishingRecoverer), retry with exponential backoff (DefaultErrorHandler), Kafka Streams binding via StreamsBuilderFactoryBean, transaction support (KafkaTransactionManager), embedded Kafka for testing (EmbeddedKafkaBroker), and Spring Boot auto-configuration via spring-boot-starter. Key difference from AMQP: Kafka retains all messages (configurable retention), enables replay and multiple independent consumer groups reading same topic.
Score Breakdown
⚙ Agent Friendliness
🔒 Security
Kafka SASL credentials in application.yml must be externalized — use Spring Cloud Config, Vault, or environment variables for agent production deployments. Kafka ACLs control topic read/write per service account — follow least-privilege for agent service topic access. TLS required for agent event data in transit.
⚡ Reliability
Best When
Your agent service produces events that multiple independent consumers need to process, or you need event replay, audit log, or stream processing — Kafka's log-based model enables all consumer groups to read independently.
Avoid When
You need complex routing logic, immediate message deletion after consumption, or sub-10ms task delivery latency.
Use Cases
- • Agent event streaming — KafkaTemplate<String, AgentEvent> kafkaTemplate.send('agent-events', event.agentId(), event) publishes agent events; @KafkaListener(topics = 'agent-events', groupId = 'agent-analytics') AgentAnalyticsService reads independently from audit log service using same topic
- • Agent event replay for new services — Kafka topic retention means new AgentSearchIndexService can replay all historical agent events from offset 0 to build search index; impossible with RabbitMQ which deletes consumed messages
- • Agent audit log — every agent state change event written to Kafka topic with perpetual retention; compliance requires full agent action history; Kafka log serves as immutable audit ledger consumed by multiple downstream systems
- • Embedded Kafka for agent integration tests — @EmbeddedKafka(partitions=1, topics='agent-events') starts in-process Kafka for agent service integration tests; no Docker required in CI; spring.kafka.bootstrap-servers=#{embeddedKafka.brokersAsString} auto-configured
- • Agent Kafka Streams aggregation — StreamsBuilder builds topology that joins agent-requests topic with agent-configs topic, aggregates per agent, and outputs to agent-metrics topic; stateful stream processing for real-time agent KPI computation
Not For
- • Complex message routing — Kafka is a log; no exchange/binding routing logic; for agent message fanout with routing rules use RabbitMQ with topic exchanges
- • Request-reply messaging — Kafka is async fire-and-forget; for synchronous agent RPC use REST or gRPC; Spring Kafka ReplyingKafkaTemplate exists but is complex
- • Low-latency task queues — Kafka's batching and partitioning model adds latency vs RabbitMQ for single-message agent task dispatch; use RabbitMQ for sub-100ms agent task delivery
Interface
Authentication
Kafka auth via SASL (PLAIN, SCRAM, OAUTHBEARER) or mTLS. Configure via spring.kafka.properties.security.protocol=SASL_SSL and sasl.mechanism/jaas.config. Confluent Cloud uses API keys as SASL_SSL credentials.
Pricing
Spring Kafka and Apache Kafka are open source. Managed Kafka services vary in pricing.
Agent Metadata
Known Gotchas
- ⚠ Consumer group offset management — @KafkaListener with same groupId across agent service instances share partition assignment; new consumer group sees all messages from earliest offset by default (spring.kafka.consumer.auto-offset-reset=earliest); redeploying with different groupId causes agent event replay from beginning — coordinate group ID changes
- ⚠ Partition count determines max parallelism — 3 partitions means max 3 active consumers in same group; adding more agent consumer instances beyond partition count is wasted resources; partition count is set at topic creation and changing requires Kafka admin tools; plan agent topic partitions for expected consumer concurrency
- ⚠ DefaultErrorHandler replaces old SeekToCurrentErrorHandler — Spring Kafka 2.8+ deprecated SeekToCurrentErrorHandler; DefaultErrorHandler with BackOff configures retry; agent code migrating from older Spring Kafka must update error handler configuration; old handler bean definitions cause ClassNotFoundException
- ⚠ EmbeddedKafka port conflicts in parallel tests — @EmbeddedKafka default port 9092 conflicts with multiple test classes running in parallel; use spring.embedded.kafka.brokers-property annotation attribute or spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers} with dynamic port; agent test suites with multiple @EmbeddedKafka tests fail in parallel CI
- ⚠ Message deserialization failures bypass error handler — JsonDeserializer failures in consumer record deserialization occur before listener method; DefaultErrorHandler doesn't intercept; configure spring.kafka.consumer.properties.spring.deserializer.key.type and ErrorHandlingDeserializer wrapping actual deserializer to route agent poison pill messages to DLT
- ⚠ Kafka Streams and @KafkaListener in same app require separate factories — Kafka Streams (StreamsBuilderFactoryBean) and standard @KafkaListener consumers need separate consumer/producer configs; sharing spring.kafka.* properties between Streams and listener can cause serialization incompatibility; agent apps using both patterns must configure separate Spring beans
Alternatives
Full Evaluation Report
Comprehensive deep-dive: security analysis, reliability audit, agent experience review, cost modeling, competitive positioning, and improvement roadmap for Spring Kafka.
AI-powered analysis · PDF + markdown · Delivered within 30 minutes
Package Brief
Quick verdict, integration guide, cost projections, gotchas with workarounds, and alternatives comparison.
Delivered within 10 minutes
Score Monitoring
Get alerted when this package's AF, security, or reliability scores change significantly. Stay ahead of regressions.
Continuous monitoring
Scores are editorial opinions as of 2026-03-07.