Imagine two restaurant kitchens that serve the same menu.
- Kitchen 1 has a big team that shares everything. Cooks share the same stoves and the same fridge, and a manager keeps handing out orders. It has run for years, everyone knows how it works, and you can hire cooks for it anywhere. But when everyone reaches for the same pan, people wait.
- Kitchen 2 gives every chef their own station with their own stove, their own ingredients and their own orders. Nobody waits for anyone else. It is fast and predictable, but it needs a bigger, more carefully built kitchen, and the cooks are harder to find.
Both kitchens cook the same dishes. They just organise the work very differently. That is the story of Apache Kafka and Redpanda. Both move streams of events between systems, and Redpanda is built to be a drop-in replacement for Kafka. Under the hood they make different engineering bets, and those bets decide when each one wins.
If you are choosing between them, you hit a wall of claims: "10x faster", "no ZooKeeper", "3x fewer nodes", "JVM-free". Few sources explain which claims still hold and which ones are out of date. We spent several weeks studying both systems, the published benchmarks, the independent re-tests, and 13 companies that run Redpanda in production. This post is what we found. So let's start at the beginning.
1. What Is Event Streaming?
Think of a notice board where every machine, app and sensor pins a note the moment something happens: "Order placed." "Temperature is 82 degrees." "User clicked Buy." Other systems read the notes in order, at their own pace, and the notes stay on the board for a while so anyone can re-read them.
That is event streaming. Kafka and Redpanda are both the notice board. A few words you will see everywhere:
- Producer: whatever writes events.
- Consumer: whatever reads them. Several consumers can form a consumer group and share the work.
- Topic: a named stream, like "orders".
- Partition: a topic is split into partitions so many machines can work in parallel. Order is guaranteed inside one partition.
- Offset: the position of an event in a partition, like a page number.
Unlike a traditional message queue, a streaming log keeps events after they are read. That is what lets you replay history, for example to investigate a failure or to test a new rule against last week's data.
Queues such as RabbitMQ throw a message away once it is consumed. Nanit, one of the companies in our research, moved from RabbitMQ to Redpanda for exactly this reason: a queue could not replay old events.
2. What Is Kafka, and What Changed in 4.0?
Apache Kafka is the open-source standard for event streaming. It runs on the JVM (the Java virtual machine) and uses a pool of threads shared across partitions. Writes go through the operating system's page cache, and durability by default comes from replication across brokers, not from an fsync on every write.
For years Kafka also needed a separate coordination service called ZooKeeper, which was one more system to deploy, secure and monitor. That is no longer true. Kafka 4.0, released on March 18, 2025, made KRaft (Kafka's own Raft-based controller) the default and removed ZooKeeper completely. It also added:
- A new consumer rebalance protocol (KIP-848): rebalances no longer need global synchronisation, which makes consumer groups more stable.
- Share groups (early access): several consumers can read one partition, which enables queue-style use.
- Java requirements: brokers need Java 17 or later.
Many older comparisons say "Redpanda is simpler because Kafka needs ZooKeeper". That is out of date. Check the date on anything you read.
3. What Is Redpanda?
Redpanda is a streaming platform that speaks the Kafka protocol but is a different piece of software, written from scratch in C++. Because it speaks the same protocol, existing Kafka producers, consumers and tools can usually connect to it with only a change of address.
It ships as a single binary that includes the broker, Raft-based replication, a schema registry and an HTTP proxy, plus a command-line tool (rpk), a web console and built-in metrics. Around the core sit a few related products:
| Product | What it is |
|---|---|
| Redpanda Streaming | The Kafka-compatible core engine |
| Redpanda Connect | Data pipelines with 300+ connectors, so you do not need a separate Kafka Connect cluster |
| Redpanda SQL | A PostgreSQL-compatible engine to query live topics and Apache Iceberg data together |
| Agentic Data Plane | A control layer for AI agents that sits between the agents and company data |
You can run it yourself (self-managed), as a fully managed cloud service, or in a bring-your-own-cloud (BYOC) setup where Redpanda manages the cluster inside your own cloud account.
Redpanda replaces the Kafka broker, not the Kafka client. Your application code still uses standard Kafka client libraries.
4. How Do They Differ Under the Hood?
From the outside both look the same: partitioned, replicated, append-only logs behind one API. The differences are inside.
| Apache Kafka 4.x | Redpanda | |
|---|---|---|
| Language and runtime | Java on the JVM (brokers need Java 17+) | C++ native binary |
| Threading | Shared thread pools | One pinned thread per CPU core, shared-nothing |
| Disk I/O | Operating system page cache, mostly sequential writes | Direct I/O (DMA) that bypasses the page cache |
| Metadata and coordination | KRaft controller quorum (since 4.0) | Raft controller built in |
| Replication | Leader and followers with in-sync replicas | Raft group per partition, committed by majority |
| Default durability | Replication, no fsync on every write | Raft majority, fsync by default (per one engineering comparison) |
| Packaging | Broker, plus separate Schema Registry and Connect | Single binary with embedded schema registry and tooling |
The JVM matters because of garbage collection: the runtime periodically pauses to reclaim memory. A pause stalls everything on that broker, which shows up as latency spikes at the far end of the distribution. Tuning reduces pauses but cannot remove them. C++ has no garbage collector, so that class of pause does not exist.
Neither design is simply better. Each is optimised for a different failure mode, and the next section explains Redpanda's bet.
5. What Is Thread-per-Core?
Back to the kitchens. In Kitchen 1, many cooks (threads) share the same stoves and fridge (memory). When two reach for the same pan, one waits (a lock), and the manager keeps switching which cook uses which stove (context switching).
Redpanda is Kitchen 2. It uses a C++ framework called Seastar that pins one thread to each CPU core. Each core owns its own memory and its own set of partitions. If core 0 gets a request for a partition that belongs to core 1, it does not reach in and grab it. It passes a message through a lock-free queue, like handing an order slip across the pass.
According to Redpanda's documentation, this gives:
- No context switching: one pinned thread per core, so the operating system has nothing to shuffle.
- No lock contention: cores do not share data, so they do not wait on each other.
- Direct hardware access: DMA for disk I/O, memory allocated up front and pinned to cores, and built-in tuners that configure the host.
The same page notes that this lets Redpanda scale up on one machine before it has to scale out.
The part nobody puts in the pitch. A station per chef only works if you have the space and the stoves. Thread-per-core and direct I/O are bets on fast NVMe drives and dedicated cores. Redpanda's small, direct writes create more random I/O, which independent testing linked to latency problems on long runs. Kafka's page cache turns many small writes into larger sequential ones, which is gentler on the drive.
Redpanda's design is real, and it gives predictable latency in the right setting. It is also a bet on specific hardware.
6. Is Redpanda Really 10x Faster?
"10x faster tail latency" is the most-quoted claim, and almost every copy of it traces back to one vendor benchmark. Before you repeat a number, look at where it came from.
The vendor benchmark. In October 2022 Redpanda published a comparison of Redpanda 22.2.2 against Kafka 3.2.0. It used the OpenMessaging Benchmark with TLS and SASL enabled, two producers, two consumers and a 1:1 read/write ratio.
| Workload | Kafka p99.99 | Redpanda p99.99 | Claimed gap |
|---|---|---|---|
| 50 MB/s | 164.6 ms (3 x i3en.large) | 13.9 ms (3 nodes) | 12x |
| 500 MB/s | 388.7 ms (4 x i3en.3xlarge) | 16.7 ms (3 nodes) | 23x |
| 1 GB/s | 5,509.7 ms (6 x i3en.6xlarge) | 79.6 ms (3 nodes) | 70x |
"p99.99" means the slowest 1 in 10,000 requests. It describes the extreme tail, not the typical request. Two cautions:
- The post disagrees with itself. Its 50 MB/s paragraph says "up to 38% faster" at the tail, while its table says 12x. Its 500 MB/s paragraph gives 10x and 4x, while its table says 23x.
- The versions are old. When we did this research the latest releases were Kafka 4.3.x and Redpanda 26.x.
The same post also notes that with 9 Kafka nodes at 1 GB/s, Kafka's average latency was within 16% of Redpanda's. The gap is mostly in the extreme tail.
The independent re-test. In May 2023 Jack Vanlightly re-ran the tests on three i3en.6xlarge brokers. He was working for a Kafka vendor at the time, so read his work with that in mind, but his methods are public. He found:
- 1 GB/s on 3 nodes did not hold. Redpanda could not drain backlogs under that load or sustain it when a broker went down.
- Many producers hurt. With 50 producers instead of 4, end-to-end latency reached about 24 seconds at 500 to 1000 MB/s.
- Long runs degraded. Over 24 to 36 hours at 1 GB/s, Redpanda's latency rose after about 12 hours (p99 near 3.5 s), which he linked to NVMe write latency. Kafka's improved over the same period.
- Retention matters. Once brokers began deleting old segments, Redpanda's latency stepped up on mid-to-high throughput tests. Below 100 MB/s it did not.
- Keyed records. With keys and 288 partitions, Redpanda topped out near 330 MB/s against a 500 MB/s target. Kafka reached 500.
- Where Redpanda did well. Its most robust results came at low throughput. Kafka never lost on throughput in his tests.
Our reading: Redpanda's latency advantage is genuine for a narrow but common shape (few producers, moderate throughput, shorter tests). Its throughput and hardware-efficiency claims are the contested ones. We found no Redpanda reply to these specific tests, so for current versions the question is open.
7. Does Redpanda Really Cost Less to Run?
If the speed story is contested, the operations story is where Redpanda's case is most consistent. In our research, cost and operational complexity were each cited by 10 of 13 companies, and latency by 5.
The saving is usually not that one Redpanda server is cheaper than one Kafka server. It comes from three places:
- Compute: less CPU lost to garbage collection pauses, context switches and locks, so a latency-sensitive workload can sometimes run on fewer nodes. Redpanda claims up to 3x fewer. Independent testing disputes that at the highest throughput.
- Fewer components: a typical Kafka stack means brokers, controllers, a Schema Registry, a Connect cluster and JVM tuning. Redpanda folds most of that into one binary, with auto-tuners and built-in metrics. Since Kafka 4.0 removed ZooKeeper, the remaining gap is components and tuning, not coordination.
- People: less JVM tuning and fewer alerts. This is hard to measure and easy to overstate, but it was the most common reason in our case studies.
A cost checklist. When you model the cost, include the items that are easy to forget:
- Licence fees: Redpanda Enterprise, or Confluent if you use its Kafka distribution or cloud.
- The number of nodes you need at your real peak, from your own test.
- NVMe headroom. One comparison notes Redpanda may need more over-provisioning at high load than the common 10% baseline.
- Migration effort, especially if you use Kafka Streams.
- Hiring and training. Kafka skills are far easier to find.
Redpanda's saving is real when it replaces a costly managed bill or removes operating effort. It is not automatic.
8. Is Tiered Storage a Redpanda Exclusive?
Imagine a warehouse with a small, fast loading dock for today's parcels and a huge, cheap storage hall for old ones. Tiered storage does this for event data: recent events stay on fast local disks, and older ones move automatically to cheap object storage such as Amazon S3, while remaining readable through the same API.
It is tempting to list this as something only Redpanda offers. It is not:
| Apache Kafka | Redpanda | |
|---|---|---|
| Available | Yes, introduced in 3.6 | Yes |
| How you get it | Disabled by default. Kafka has no built-in remote storage plugin, so you supply your own | Integrated, but an Enterprise-licence feature |
| Known limits | Not supported for compacted topics | Needs the paid licence |
The real difference is a managed, integrated experience (Redpanda, at a price) versus a bring-your-own-plugin one (Kafka, at the cost of your own engineering).
9. Is Redpanda a Drop-In Replacement?
Mostly. Redpanda speaks the Kafka wire protocol, so existing producers, consumers and tools generally connect with only an endpoint change. StoneX described its migration as "we really did just change the endpoint, and off we went." Two practical notes:
- You still use Kafka client libraries in your application. Redpanda replaces the broker, not the client.
- Compatibility is strong, not total. One engineering guide reports that Kafka Streams is not drop-in and migrating it is a refactor, and that some Kafka Connect connectors show edge cases under load or failure because they use internal admin APIs. Test transactions and exactly-once behaviour explicitly.
The ecosystem gap. Kafka's real moat is everything around it: Kafka Streams, Kafka Connect, Flink and Spark integrations, managed offerings on every major cloud, and a huge pool of engineers and answers. The same guide counts over 32,000 Stack Overflow questions for Kafka against tens for Redpanda. Redpanda's answer is Redpanda Connect (300+ connectors) and a smaller, vendor-led community.
Kafka is also moving. Version 4.0 added a new consumer rebalance protocol and early-access share groups, so some pain points that pushed teams toward Redpanda are being addressed upstream.
"Drop-in" means the wire protocol matches. It does not mean your whole toolchain moves for free.
10. What About the Licence?
This is the part that surprises teams late in a project.
| Apache Kafka | Redpanda | |
|---|---|---|
| Licence | Apache 2.0 | Community Edition under the Business Source License (BSL), which converts to Apache 2.0 four years after each code merge |
| Free use | Unrestricted | Free and source-available, but you cannot offer Redpanda as a commercial streaming service to others |
| Paid features | None in the project itself | Enterprise licence needed for Tiered Storage, RBAC, audit logging, continuous data balancing, Iceberg topics, remote read replicas, OIDC and more |
| Trial | Not applicable | 30-day enterprise trial on new clusters, extendable by 30 days |
For most internal use, BSL is fine. It matters if you want to build a hosted product on top, if your organisation only permits OSI-approved licences, or if a feature you assumed was free (tiered storage, role-based access control) turns out to need an Enterprise licence.
Check the licence page against your must-have feature list before you commit.
11. Who Actually Switched, and Why?
We studied 13 companies that publish Redpanda adoption stories: Akamai, NYSE, StoneX, ShareChat, Truecaller, Zafin, Lacework, The Hotels Network, Fortis Games, LiveRamp, chess.com, Nanit and GlobalFoundries. We tagged each by the reasons it gave. Ten cited cost, ten cited operational complexity, five cited latency, and one cited scalability. Several cited more than one.
Most teams moved to cut cost and operating effort. Fewer moved for raw speed. Only NYSE named latency as its single driver.
All of these are vendor-published case studies from customers who chose to be featured, so read the figures as claims. We opened the four most cited:
| Company | Came from | Redpanda deployment | Reported result |
|---|---|---|---|
| Akamai | Confluent Cloud plus object storage | Self-managed Enterprise (Azure, moving to Linode) | About 55% lower cost than Confluent Cloud; 3M+ events/s; three major upgrades with zero downtime; no latency figures given |
| ShareChat | Google Pub/Sub (not Kafka) | Redpanda Cloud BYOC on GCP | About 70% lower cloud spend; about 3 GB/s (5M msgs/s) on fewer than 40 nodes |
| StoneX | Kafka via a managed provider | Self-managed | Latency from double-digit to single-digit ms; 5-6x fewer resources; migration was "just change the endpoint" |
| NYSE | Evaluated other Kafka distributions | Self-managed Enterprise on AWS | Chose it because alternatives "didn't fare well with bursty traffic"; sub-100 ms customer delivery target; no comparative figures given |
Three details change how you should quote these:
- Not every story is Kafka to Redpanda. ShareChat came from Pub/Sub, LiveRamp from batch Hadoop and Spark, and Nanit from RabbitMQ. Those savings say little about Kafka itself.
- Several comparisons are against Confluent Cloud, not self-run Kafka. Akamai's 55% measures a managed-service bill. A well-run self-managed Kafka may land elsewhere.
- Some numbers belong to someone else. The NYSE page's "under 20 microseconds" describes NYSE's own decision systems, not Redpanda's latency.
The strongest signal is the small-team one. Truecaller found that self-managed Kafka needed dedicated engineers, and The Hotels Network lacked the in-house expertise to run it. Both moved to managed Redpanda Cloud, which removed that burden.
12. Other Terms You'll Keep Hearing
- Tail latency (p99, p99.99): how slow the slowest 1% (or 0.01%) of requests are. Averages hide it, and it is where Kafka and Redpanda differ most.
- Garbage collection (GC): the JVM's periodic pause to reclaim memory.
- Raft: a consensus algorithm that lets replicas agree on the order of writes. Redpanda uses it per partition, and Kafka uses it (as KRaft) for cluster metadata.
- KRaft: Kafka's built-in replacement for ZooKeeper.
- ISR (in-sync replicas): the replicas of a Kafka partition that are caught up with the leader.
- Tiered storage: moving old data from local disks to object storage.
- BYOC (bring your own cloud): the vendor manages the cluster, but it runs inside your own cloud account so the data stays with you.
- Backpressure: what happens when consumers cannot keep up with producers. Kafka handles it by letting consumers read at their own pace and measuring consumer lag.
- Schema registry: a service that stores the shape of your events so formats can change safely over time.
- Seastar: the C++ framework behind Redpanda's thread-per-core design.
- Shadow traffic: mirroring production traffic to a new system, to test it without risk.
13. Which One Should You Choose?
Start with your workload and your team, not with a latency chart.
| Your situation | Lean toward | Why |
|---|---|---|
| Greenfield project, small team, no Kafka experience | Redpanda (managed or BYOC) | One binary, simple tooling, less to learn and run |
| Latency is a contractual target at p99 or above, with few producers and moderate throughput | Redpanda, after your own test | This is the shape where its design shines (NYSE, StoneX) |
| Existing Kafka that is well run, with mature monitoring and runbooks | Kafka | Migration cost must beat real gains, and ZooKeeper is no longer an argument |
| Heavy use of Kafka Streams or niche Kafka Connect connectors | Kafka | Streams is not drop-in, and connectors can hit edge cases |
| Many producers, many partitions, keyed routing, sustained 1 GB/s or more | Kafka, or test Redpanda hard | Independent tests found Redpanda weakest here |
| Licence must be OSI-approved, or you plan to host the platform as a product | Kafka | Redpanda Community is BSL |
| You need tiered storage without paying for an enterprise licence | Kafka (bring your own plugin) | Redpanda's tiered storage is Enterprise-only |
| Kafka operations are eating engineering time or blocking hiring | Evaluate Redpanda | The most common reason in our adoption research |
If you do evaluate a move:
- Stand up Redpanda alongside Kafka and mirror production traffic in shadow mode.
- Compare p99 and p99.99 end to end, over a run long enough to hit your retention limit.
- Kill a broker under load, and measure recovery and backlog draining.
- Test exactly-once and transactional flows explicitly.
- Move producers one service at a time, and keep a rollback path.
If your Kafka cluster is stable, your team knows it, and you rely on Streams or a wide connector set, the gain is unlikely to justify the move. Kafka is the default, so hiring, tooling and answers are easier to find.
14. A Reality Check
- A vendor benchmark is a marketing document, even a well-run one. Both sides publish numbers that favour their own design.
- A short test measures a different system than a long one. Redpanda's latency changed over 12-hour runs and once retention kicked in.
- A case study shows one company on one workload, and the savings often compare against a managed-service bill.
- A "drop-in" replacement can still need work for Kafka Streams, some connectors and transactions.
- A feature that exists on both sides can differ in cost and effort. Tiered storage is the example.
- Kafka is not standing still. Kafka 4.0 removed ZooKeeper and changed consumer rebalancing.
Good evaluation means measuring latency, throughput, cost, failure behaviour and team effort on your own workload, and knowing when the stable option is simply the right one. The goal is not to switch. It is to pick the right tool for a real problem.
The Bigger Picture
Back to the two kitchens.
- Kitchen 1 (Kafka) is the proven, shared kitchen: an open licence, a huge ecosystem, and years of operational knowledge.
- Kitchen 2 (Redpanda) is the station-per-chef kitchen: predictable latency and simple operations in the right setting, at the price of a licence, a smaller ecosystem and a bet on specific hardware.
The JVM, thread-per-core, the page cache, KRaft, tiered storage and the licence are not separate topics. They are pieces of one decision. And Kafka 4.0 has narrowed the gap by removing ZooKeeper.
Don't just memorise the claims. Learn where each one came from, what it was measured on, and what can go wrong. The one rule that survived every source we read is to benchmark your own workload, for long enough, with failures included.
Sources
All pages below were opened and read for this post. Information was current as of October 2026.
- Redpanda vs. Kafka: A performance comparison (Redpanda, October 2022; Redpanda 22.2.2 vs Kafka 3.2.0)
- Kafka vs Redpanda Performance: Do the claims add up? (Jack Vanlightly, May 2023)
- How Redpanda Works (Redpanda documentation)
- Redpanda Licenses and Enterprise Features (Redpanda documentation)
- Kafka tiered storage (Apache Kafka documentation)
- Kafka 4.0 unveiled: key changes (Instaclustr)
- Redpanda vs. Kafka: An honest engineering guide (Medium, secondary source)
- Redpanda case studies: Akamai, ShareChat, StoneX, NYSE
- Our internal Redpanda research workbook (adoption analysis for the other nine companies, which we did not re-verify against the vendor pages)
