Kafka Beyond the Basics: Connect, Streams and Real-Time Patterns
A practical guide to Kafka partitions, producers, consumers, Connect, Streams, event sourcing, and CQRS.
Apache Kafka is often introduced as a messaging platform, but that description captures only part of what makes it valuable. At its core, Kafka is a partitioned, append-only log. Applications write events to that log, consumers process them independently, and retained history allows data to be replayed as requirements evolve.
Around that foundation, Kafka Connect moves data between systems, while Kafka Streams transforms and analyzes events as they flow. This article revisits Kafka’s core architecture, then shows how those pieces combine in practical real-time systems.
Kafka’s Core Model: The Partitioned Log
A Kafka topic is a named stream of records divided into partitions. Each partition is an ordered, immutable log, and every record receives an offset identifying its position within that partition.
Ordering is guaranteed within a partition, not across an entire topic.
Producers determine where records are written. Consumers read those records, usually as part of a consumer group. Inside a group, each partition has one active owner, which allows processing to scale horizontally without multiple group members simultaneously handling the same partition.
Partitions affect far more than storage. They define ordering boundaries, consumer parallelism, producer routing, and the number of tasks available to a Kafka Streams application.
What Happens When a Producer Sends a Record?
A producer starts with a record containing a topic, an optional key, and a value. The record is serialized, assigned to a partition, and added to a partition-specific batch. A sender thread later transmits that batch to the broker acting as the partition leader.
Keys matter because records with the same key are typically routed to the same partition, preserving their relative order. Records without keys can be distributed across partitions for throughput.
Several settings control the balance between latency, throughput, and durability:
batch.size controls how much data can accumulate in each partition batch.
linger.ms lets the producer wait briefly so more records can join a batch.
acks determines how much confirmation the producer requires from Kafka.
Idempotence prevents duplicate writes caused by safe retries within a producer session.
With acks=0, the producer does not wait for confirmation. With acks=1, the partition leader confirms the write. With acks=all, the write is acknowledged according to the in-sync replica requirements. The right setting depends on what losing or duplicating an event would mean for the application.
Consumer Groups and Offset Management
Consumer groups are Kafka’s primary mechanism for scaling reads. If a topic has four partitions and a group has two consumers, Kafka can assign two partitions to each consumer. Another group can independently read all four partitions because each group maintains its own progress.
That progress is represented by committed offsets. After a restart or rebalance, a consumer resumes from the last committed position associated with its group.
Commit timing directly affects delivery behavior:
Commit before processing: a crash can cause records to be skipped.
Process before committing: a crash can cause records to be processed again.
Commit offsets inside a supported transaction: consumed offsets and produced output can be committed atomically.
Automatic commits are convenient, while manual commits provide more control. The important question is whether the committed offset accurately represents work the application has completed.
Inside the Kafka Cluster
Each partition has a leader replica. Producers send writes to the leader, while follower replicas copy its log and can take over if the leader becomes unavailable. Kafka tracks an in-sync replica set for every partition. A durable design combines replication, a suitable minimum number of in-sync replicas, and producer acknowledgements.
Kafka also separates cluster metadata management from data processing. In a KRaft-based cluster, a controller quorum manages metadata and leader elections. Brokers handle client requests and store partition data.
On disk, a partition is stored as a directory of log segments and indexes. The active segment receives new records. Closed segments can later be removed by retention or processed by compaction. Retention removes eligible segments by age or size; compaction preserves the latest value for each key, which makes compacted topics useful for changelogs and recoverable state.
Kafka Connect: Move Data Without Writing Another Service
Many Kafka workflows begin or end outside Kafka. Data may start in a relational database, file system, message queue, or object store. Processed records may need to reach a search index, analytics platform, or another database. Kafka Connect provides a framework for these integrations.
Connectors
Connectors define what should be copied and how the work should be divided. Source connectors bring data into Kafka; sink connectors move records from Kafka into external systems.
Tasks
Tasks perform the copying. Depending on the connector and source, work can be distributed across multiple tasks for parallelism.
Workers
Workers host connectors and tasks. In distributed mode, workers coordinate assignments and rebalance work after a failure.
Converters
Converters translate data between Connect’s internal representation and the format stored in Kafka, such as JSON or Avro. Connect also tracks source offsets so integrations can resume after restarts.
Example: MySQL to Elasticsearch
Consider a login table in MySQL. A JDBC source connector can poll for new rows based on a timestamp and publish them to a Kafka topic. An Elasticsearch sink connector can consume that topic and index each record.
MySQL → JDBC source connector → Kafka topic → Elasticsearch sink connector → Elasticsearch
The application team does not need to maintain a custom service for polling, publishing, offset storage, retry handling, and indexing. The integration is represented primarily as connector configuration. This is where Connect is strongest: moving data between established systems with minimal application code.
Kafka Streams: Process Events as They Happen
Kafka Connect moves records. Kafka Streams computes with them. Kafka Streams is a Java library embedded directly in an application. A processing topology can filter, map, group, aggregate, join, and write results to an output topic.
Two abstractions appear frequently:
KStream represents an unbounded sequence in which every record is an event or fact.
KTable represents a changing view in which each key resolves to its latest known value.
For example, a stream of user activity may contain every click. A table derived from customer updates may contain only the current profile for each customer.
Streams applications can maintain local state stores for aggregates and joins. Changelog topics allow another instance to reconstruct that state after a failure. Scaling is achieved by running more application instances, with input partitions determining how work is divided into tasks.
Windows, Joins, and Real-Time Decisions
Streams become especially useful when events must be correlated over time. A windowed aggregation can count transactions during a recent interval, compute averages, or identify an unusual burst of activity. Joins can enrich events with account data, inventory, profiles, or other signals.
Together, windows and joins support clickstream enrichment, market statistics, anomaly detection, and fraud detection.
A Real-Time Fraud Detection Pipeline
A fraud-detection pipeline demonstrates how Connect and Streams complement each other. Connectors can capture changes from operational databases and place account information in Kafka. A payment service can publish transaction events directly. A Streams application can join payments with account data, calculate windowed activity, suppress incomplete results, and produce alerts.
Sink connectors can then send results to systems such as Elasticsearch or object storage.
The responsibilities are intentionally separated:
Producer APIs publish application-owned events.
Kafka Connect integrates Kafka with external systems.
Kafka Streams performs event-driven computation.
Sink connectors deliver results to serving or analytical platforms.
Event Sourcing: Store What Happened
In a traditional state-oriented system, a database row tells us the current result. Event sourcing stores the sequence of events that produced that result. Instead of keeping only a balance of 120, a system may retain AccountOpened, MoneyDeposited, MoneyWithdrawn, and another MoneyDeposited event. The balance is a projection calculated from the event history.
Because Kafka retains ordered events within each partition, a consumer can replay them to rebuild state, add a new view, or correct a processing bug. A compacted topic or Streams state store can maintain the latest projection while the event log preserves history.
CQRS: Separate Writes from Reads
Command Query Responsibility Segregation separates the model used to change state from the models used to answer queries. A command service validates a request and emits an event. Independent processors then build read models designed for search, dashboards, caches, analytics, or query-specific APIs.
Each read model can be rebuilt by replaying the event stream, and reads and writes can scale independently. The trade-off is eventual consistency. A read model may not reflect a newly accepted command immediately, so applications must explicitly handle read-your-own-writes when needed.
CQRS and event sourcing often work together: events form the source of truth, while projections provide convenient views of that truth.
Choosing the Right Kafka Tool
Use producer and consumer clients when:
The application creates domain events.
Custom processing or protocol behavior is required.
Kafka interaction is part of the service’s core logic.
Use Kafka Connect when:
Data needs to move between Kafka and an external system.
A suitable connector already exists.
Configuration and managed offset tracking are preferable to a custom integration service.
Use Kafka Streams when:
Events must be filtered, transformed, joined, or aggregated.
Processing requires state or time windows.
The topology belongs inside a deployable Java application.
Connect handles movement. Streams handles computation. Client APIs provide direct application control.
Final Takeaways
Kafka becomes easier to reason about when every design decision is traced back to partitions. Partitions define ordering, parallelism, consumer ownership, and Streams tasks. Replication, acknowledgements, in-sync replicas, and commit timing determine durability and processing behavior.
Kafka Connect provides a configurable way to move data into and out of Kafka.
Kafka Streams provides a stateful library for processing events in flight.
Event sourcing preserves the history of what happened.
CQRS turns that history into read models optimized for different users and queries.
Kafka is not simply a queue between two applications. It can serve as a durable event log connecting operational systems, real-time computation, and the views derived from both.
Comments