Dead-letter topics in Spring Kafka

Why does one bad record stall the whole partition?

Kafka's unit of ordering is the partition. One consumer owns a partition and reads it strictly in offset order, because the container only calls poll() again once the previous batch has been processed. If record 1,204 throws every time and your handling just logs and rethrows, the container retries it in place: ten attempts by default, far more if you configured a generous back-off. Offset 1,205 waits the whole time, and lag on that one partition climbs while its siblings keep moving.

That pattern — one partition frozen, the rest healthy — is the tell. A slow consumer needs more capacity; a poison pill needs somewhere else for the bad record to go.

How do you retry a failing record without blocking the partition?

DefaultErrorHandler retries in place: it classifies the exception, and for anything recoverable it re-delivers the same record to your listener according to a BackOff, without advancing the offset. That is still blocking — the partition sits still for the length of the backoff — which is fine for a dependency down for a few seconds, and wrong for a record that will never succeed.

@Bean
public DefaultErrorHandler kafkaErrorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template,
        (record, ex) -> new TopicPartition(record.topic() + "-dlt", record.partition()));

    // 1s, 2s, 4s, 8s, capped at 15s, then give up and recover
    var backOff = new ExponentialBackOffWithMaxRetries(4);
    backOff.setInitialInterval(1_000L);
    backOff.setMultiplier(2.0);
    backOff.setMaxInterval(15_000L);

    var handler = new DefaultErrorHandler(recoverer, backOff);
    // fail fast on programming errors instead of retrying them four times
    handler.addNotRetryableExceptions(IllegalArgumentException.class);
    return handler;
}

Once the backoff is exhausted, the handler calls the recoverer and only then does the container commit the offset. That is the step that matters: without a recoverer of your own, exhausting the back-off hands the record to the default recoverer, which logs it and moves on. The partition is unblocked, but the record is gone and the only trace is a log line. A recoverer turns "give up" into "move on".

What does @RetryableTopic add, and how are the retry topics named?

Blocking retries hold up the partition for as long as the backoff runs — a problem once backoffs reach tens of seconds. Since Spring Kafka 2.7, @RetryableTopic takes the retry off the original partition: each failed attempt republishes to a dedicated retry topic, a listener there picks it up after its own delay, and the original partition stays free throughout.

@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 1000, multiplier = 2.0, maxDelay = 15000),
    autoCreateTopics = "true",
    exclude = { IllegalArgumentException.class }
)
@KafkaListener(topics = "orders", groupId = "orders-service")
public void onOrder(ConsumerRecord<String, Order> record) {
    orderService.process(record.value());
}

@DltHandler
public void onDlt(ConsumerRecord<String, Order> record,
                   @Header(KafkaHeaders.DLT_EXCEPTION_MESSAGE) String message) {
    dltAuditRepository.record(record.topic(), record.partition(), record.offset(), message);
}

Spring creates the topics when autoCreateTopics is on, suffixed with their delay by default: orders becomes orders-retry-1000, orders-retry-2000, ending at orders-dlt. Set topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE for orders-retry-0, orders-retry-1 instead, and give each retry topic at least as many partitions as the original, or you lose per-key ordering. Skip @DltHandler and the record still lands in the DLT; Spring logs it instead of calling your method.

Choose	When
DefaultErrorHandler + BackOff	Short outages, low retry counts, a few seconds total — the partition can afford to wait.
@RetryableTopic	Longer backoffs, downstream systems with their own slow recovery, or many listeners where you do not want one flaky dependency to freeze every partition it touches.

What lands in the dead-letter topic, and what headers does it carry?

DeadLetterPublishingRecoverer does the write to the DLT: by default the original topic name with -dlt appended, same partition number, stamped with headers describing what went wrong and where it came from.

Header (KafkaHeaders)	Holds
DLT_EXCEPTION_FQCN	Fully qualified class name of the exception that caused the failure.
DLT_EXCEPTION_MESSAGE	The exception's message.
DLT_EXCEPTION_STACKTRACE	The full stack trace, as text.
DLT_ORIGINAL_TOPIC	The topic the record was originally consumed from.
DLT_ORIGINAL_PARTITION	The partition it came from.
DLT_ORIGINAL_OFFSET	Its offset on that partition — the key you need to replay it precisely.
DLT_ORIGINAL_TIMESTAMP	Its original produce timestamp.

That is the point of a recoverer over a hand-rolled catch that logs and swallows: a logged exception tells you a record failed once, at the moment you happened to be watching. A DLT record with these headers tells you, permanently, which record, from where, and why — queryable with any Kafka client.

How do you stop a bad deserializer from taking the consumer down first?

All of the above assumes your listener gets called with a deserialized object. A malformed payload — wrong schema, truncated bytes, JSON where Avro is expected — fails inside the deserializer, on the poll thread, before any error handler sees a record to retry. Unhandled, it kills the consumer.

ErrorHandlingDeserializer is the fix: it wraps your real deserializer, catches what it throws, and hands the container a record with a null value and the exception in a header instead of letting it propagate.

@Bean
public ConsumerFactory<String, Order> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class.getName());
    return new DefaultKafkaConsumerFactory<>(props);
}

DefaultErrorHandler treats DeserializationException as fatal by default — no retries, straight to the recoverer, which is right: a payload that will never parse only delays the inevitable. It still reaches your DLT with the original topic, partition and offset, and the exception headers now point at a schema problem rather than a business one.

Does any of this give you exactly-once processing?

No, and it is worth saying plainly. @RetryableTopic and DLT publishing sit on Kafka's ordinary at-least-once semantics: a rebalance, a crash between processing and committing, or a redelivered retry-topic message can all bring the same event to your listener twice. Kafka's transactional producer gives exactly-once for a strict read-process-write pipeline that never leaves Kafka, but once a listener calls a database, an API or another topic outside that transaction, you are back to at-least-once.

The practical answer is an idempotent consumer: make reprocessing safe rather than trying to prevent it, by tracking applied event IDs alongside the business write, in the same transaction.

@Transactional
public void process(PaymentEvent event) {
    if (processedEventRepository.existsById(event.id())) {
        return; // already applied — safe to see this again after a retry or replay
    }
    ledger.apply(event);
    processedEventRepository.save(new ProcessedEvent(event.id(), Instant.now()));
}

This is also what makes replay safe later: to the consumer, a replayed record is just another redelivery.

How do you replay from the DLT without replaying blind?

A DLT record stays unresolved until it is fixed and replayed, or deliberately written off. Replay means producing it back onto its original topic — read from the DLT_ORIGINAL_TOPIC header, not the DLT topic name — with a trail kept outside the Kafka log, since retention will eventually delete both the record and the evidence of what happened to it.

@Service
public class DltReplayService {

    private final KafkaTemplate<String, byte[]> kafkaTemplate;
    private final ReplayAuditRepository auditRepository;

    public DltReplayService(KafkaTemplate<String, byte[]> kafkaTemplate,
                             ReplayAuditRepository auditRepository) {
        this.kafkaTemplate = kafkaTemplate;
        this.auditRepository = auditRepository;
    }

    public void replay(ConsumerRecord<String, byte[]> dltRecord, String replayedBy, String reason) {
        String originalTopic = header(dltRecord, KafkaHeaders.DLT_ORIGINAL_TOPIC);
        int originalPartition = Integer.parseInt(header(dltRecord, KafkaHeaders.DLT_ORIGINAL_PARTITION));

        kafkaTemplate.send(originalTopic, dltRecord.key(), dltRecord.value());

        auditRepository.save(new ReplayAudit(
            originalTopic, originalPartition, dltRecord.offset(),
            replayedBy, reason, Instant.now()));
    }

    private String header(ConsumerRecord<?, ?> record, String name) {
        return new String(record.headers().lastHeader(name).value(), StandardCharsets.UTF_8);
    }
}

Two things make this worth building rather than reaching for a console producer. Replaying preserves the key, so it lands on the same partition and behind the same idempotency check as any other event for that entity. And every replay writes who did it, why, and which DLT offset it came from, into a table nobody can quietly delete — the record an auditor will actually ask for.

What should page someone: DLT lag and retry depth

1. DLT lag. Any consumer-group lag on a *-dlt topic means records are sitting unresolved — nothing reads a DLT automatically, so that lag never falls on its own. Alert on lag above zero sustained past your triage window, not on a threshold count.
2. Retry depth. With @RetryableTopic, watch how far down the retry chain records land. Volume on orders-retry-4000 rather than orders-retry-1000 means attempts are failing repeatedly before they reach the DLT — an earlier signal that a dependency is degrading.

Both are ordinary consumer-group lag metrics, from Micrometer's Kafka bindings or kafka-consumer-groups.sh --describe. The work is deciding these topics deserve their own alert, not lumping them in with the main topic's lag.

How is this governed in a bank?

A dead-letter topic holds customer data that failed to process, which makes it a compliance surface, not just an engineering convenience.

• Retention. Long enough to triage and replay, often longer than the source topic — but every extra day is another day a sensitive payload sits in a second location. Set it against your triage SLA, not the topic default.
• PII in the payload. The DLT carries the failed record unredacted, plus a stack trace that can itself leak field values. Apply the source topic's data classification and ACLs to the DLT too, narrower if anything, and check exceptions don't echo raw values into a header.
• Who may replay. Replay reintroduces an event into a live system — a change, not a read. Route it through the approval any production write gets: named requester, reason, a second approver for anything touching money movement.

None of this is exotic; it is the governance a bank already applies to manual payment corrections, extended to cover a new kind of correction.

Practical takeaways

1. Start with DefaultErrorHandler and a short ExponentialBackOffWithMaxRetries; move to @RetryableTopic only once a backoff would otherwise stall the partition for longer than you can tolerate.
2. Always end the chain in a DeadLetterPublishingRecoverer. Without one, exhausting retries logs the record and skips it: the partition moves on, and the record is gone.
3. Wrap every consumer's deserializer in ErrorHandlingDeserializer, even if you're confident in your schema. It costs nothing when messages are well-formed.
4. Design the listener to be idempotent before you need it to be. It is the same property that makes retries, redeliveries and replays all safe.
5. Alert on DLT lag and retry-topic depth as their own signals, and build a replay tool with an audit trail before you need one under pressure — not while triaging an incident.

My Java and Spring background, including the financial-services systems this pattern comes from, is on my CV. If you're weighing DefaultErrorHandler against @RetryableTopic for a pipeline, or want a replay tool's audit trail reviewed against a bank's change-control process, email me. For the same instinct applied elsewhere, see audit logging for agent tool calls.

Sources

• Spring for Apache Kafka reference — Handling Exceptions (DefaultErrorHandler, ErrorHandlingDeserializer, DeadLetterPublishingRecoverer)
• Spring for Apache Kafka reference — Non-Blocking Retries / Retry Topic
• Spring for Apache Kafka reference — Retry Topic naming
• Apache Kafka documentation — Consumers and consumer groups
• Confluent — Apache Kafka Dead Letter Queue

Sources checked September 2026.