How Apache Flink Works: Real-Time Stream Processing, RocksDB State, and Fault Tolerance

Every time you click an ad, swipe your card, or stream a song, a system somewhere needs to react — in milliseconds, not minutes. Not when the day's data is batch-processed overnight. Right now.

That's what Apache Flink is built for. It powers real-time analytics at Alibaba (4.5 billion events on Singles' Day alone), Uber, Netflix, and LinkedIn. And understanding how it works gives you a mental model for an entire category of modern infrastructure.

This post covers the essentials: what Flink is, how its architecture works, why it needs RocksDB, and why Kafka Streams independently arrived at the exact same design.

The Problem with Processing Data as It Arrives

Batch processing is easy. You wait until all the data has arrived, then process it. A daily report, a weekly model retrain, a monthly invoice — all of these can wait.

Stream processing is hard because the data never stops arriving. You can't wait. You process each event the moment it lands, and you might need to remember things across events — the count of clicks so far, whether you've seen this ID before, the average over the last 60 seconds.

That memory across events is called state, and it's what makes stream processing genuinely difficult.

A simple filter — "drop events where country = 'XX'" — needs no memory. But a deduplication check — "have I seen this click ID before?" — needs to remember every ID it's ever processed. A 1-minute aggregation needs to accumulate events and emit a summary when the minute ends. Both of these require stateful computation.

Flink is a stateful stream processor. That one word — stateful — is what separates it from a simple message consumer.

The Architecture

Flink has two moving parts: a JobManager and a set of TaskManagers.

JobManager is the brain. You submit a job to it, it figures out how to run it in parallel, assigns work to the TaskManagers, and watches for failures. If a worker crashes, the JobManager reassigns its work.

TaskManagers are the workers. Each one runs a slice of your job. If your job is configured to run at parallelism 8, Flink splits the work into 8 parallel streams and distributes them across available TaskManagers.

A Flink job is written as a pipeline — a sequence of operations. Here's one for an ad click tracking system:

Each step runs in parallel across your TaskManagers. Events flow through the pipeline continuously. No waiting for a batch to fill up.

The Hard Part: Where Does the State Live?

Each step in the pipeline might need to remember things:

• Deduplication needs to remember every click ID it's seen in the last 24 hours
• Window aggregation needs to accumulate clicks per ad until the minute ends

Where does that memory go?

The naive answer is: in RAM. Keep a giant dictionary in memory on each worker. Fast to read, fast to write.

The problem: RAM is expensive and limited. A fraud detection job tracking 50 million active user sessions won't fit. And when the worker crashes, everything in memory is gone — you'd have to replay hours of events from scratch to rebuild the state.

You need something that:

1. 02
   Is as fast as memory for hot data (recently accessed keys)
2. 04
   Can spill to disk when state grows beyond RAM
3. 06
   Survives a crash — data on disk persists; data in RAM doesn't

That something is RocksDB.

Why RocksDB

RocksDB is an embedded key-value database built at Facebook. "Embedded" means it runs inside the same process as your Flink TaskManager — no network hop, no separate server.

The clever part is how it balances speed and size using a structure called an LSM tree:

For hot keys — click IDs you've seen recently, ad counts you're actively updating — data sits in the MemTable in RAM. Access is as fast as a dictionary lookup.

For cold keys — click IDs from 12 hours ago that might arrive as duplicates — data has been flushed to disk. Access is slower but still fast because each disk file has a Bloom filter: a structure that can answer "is this key definitely NOT here?" without reading the file. Most lookups skip most files entirely.

For writes — always fast because RocksDB only ever appends. No random writes, no seeking. Append to the MemTable, flush to disk sequentially.

The result: RocksDB can handle state that's 10–100× larger than your available RAM while keeping hot-path operations at near-memory speed. This is why every major stream processing framework — Flink, Kafka Streams, Apache Samza — independently chose it.

Surviving Crashes: Checkpointing

State on local disk survives a process restart but not a machine failure. If the disk dies, the state is gone.

Flink solves this with checkpointing: periodically taking a snapshot of all state across all workers and uploading it to durable storage like S3.

The elegant part: Flink doesn't pause processing to take the snapshot. Instead, it injects invisible markers called barriers into the event stream. When a barrier flows through an operator, that operator snapshots its current state and continues processing. By the time the barrier has passed through every operator in the pipeline, Flink has a consistent point-in-time snapshot of the entire job.

If a worker crashes, Flink restores all workers from the last successful checkpoint and replays any Kafka events that arrived after that checkpoint. No data lost. No reprocessing from the beginning of time.

Incremental snapshots make this efficient: RocksDB's disk files are immutable once written, so Flink only uploads files that are new or changed since the last checkpoint. A 100GB state store might only upload a few MB per checkpoint if most of the data hasn't changed.

Kafka Streams: Same Problem, Same Solution

Kafka Streams is a stream processing library that runs inside your own application — no separate cluster required. You add it as a dependency, write your processing logic, and run it as part of your existing service.

Despite this different architecture, Kafka Streams made the exact same state management decision: local RocksDB, backed by a durable log.

The difference is how durability works:

Kafka Streams' approach is simpler: every state change is written to a Kafka topic, so durability comes from Kafka — which you already have. No S3 bucket needed. No checkpoint configuration.

Flink's approach is faster for large state recovery: restoring from an S3 snapshot is faster than replaying thousands of changelog events, especially when state is large.

The right choice depends on your context:

• Kafka Streams when you want simplicity and already run Kafka — great for microservices that need some stream processing without a separate cluster
• Flink when you need fine-grained control over parallelism, event-time processing, and faster recovery at large state sizes

The Common Thread

Three frameworks, independently built, all converged on the same pattern:

Local RocksDB for hot, low-latency state access + a durable log or snapshot for fault tolerance

This isn't coincidence. Remote state (Redis, Cassandra) introduces network round-trips on every event — at 50,000 events per second, that's 50,000 network calls per second per worker. Purely in-memory state is fast but can't survive crashes. Remote storage (S3 every write) is durable but adds 20–200ms latency per operation.

RocksDB sitting locally on the worker's disk, with async backup to durable storage, hits the right balance: near-memory speed for current operations, crash-safe via periodic persistence, no per-event network cost.

The stream processing space took a few years to figure this out. Now it's the consensus architecture.

Further Reading

If you want to go deeper on any of this:

• Flink Architecture Overview — the official docs are genuinely good
• Flink State Backends — how to configure RocksDB and heap state
• Kafka Streams Architecture — how Kafka Streams handles state and partitioning
• RocksDB Overview — the LSM tree design in detail
• Alibaba's 11.11 with Flink — what 4.5 billion events in a day actually looks like
• Ad Click Aggregator System Design — a concrete Flink use case, step by step