Reliable Messaging with Outbox Pattern

Prerequisites

To understand this topic fully, familiarity with transactions, microservices, event-driven architecture, and idempotency is recommended.

Here are some of my previous blog posts that will be useful:

Introduction

With a microservice architecture, it is very common to do something like the following:

Create/update/delete an entity.
Publish an event to a message broker.

For instance, consider an e-commerce system. Every time we create an Order via the OrderService, we also publish an OrderCreatedEvent so that related services can react accordingly.

Some examples of services that must perform an action when an OrderCreatedEvent is received:

InventoryService needs to update its inventory data.
EmailService sends a confirmation email to the customer.

It's likely we have some code like the following:

/**
 * Some pseudocode to illustrate creating a new order via the OrderService.
 */
 
// Step 1
Database.CreateOrder(orderInfo);
 
// Step 2
MessageBroker.Publish(new OrderCreatedEvent(orderInfo));

/**
 * Some pseudocode to illustrate creating a new order via the OrderService.
 */
 
// Step 1
Database.CreateOrder(orderInfo);
 
// Step 2
MessageBroker.Publish(new OrderCreatedEvent(orderInfo));

Problem

However, like we have discussed in my previous blog posts, things can fail! Suppose the following occurs:

✔️ Order is created successfully.
❌ Failure to send the OrderCreatedEvent.

// ✔️ Step 1
Database.CreateOrder(orderInfo);
 
// ❌ Step 2
MessageBroker.Publish(new OrderCreatedEvent(orderInfo));

// ✔️ Step 1
Database.CreateOrder(orderInfo);
 
// ❌ Step 2
MessageBroker.Publish(new OrderCreatedEvent(orderInfo));

Example without outbox pattern — Example Without Outbox Pattern

Although it might appear as though the Order was created, the OrderCreatedEvent has actually been lost. As a result, the InventoryService would not update its inventory data, and the EmailService would not send the confirmation email to the user.

For mission-critical workflows, this can have serious implications!

Without the Outbox pattern, it is possible to lose events when performing a create/update/delete of an entity. For mission-critical workflows, this can have serious implications!

Solution: The Outbox Pattern

The Outbox pattern addresses this problem, via two main components:

Persisting events with transactions.
Background worker to publish events.

Outbox Pattern

The Outbox pattern eliminates the possibility of losing events by persisting events with transactions, and having a background worker publish these events.

The overall process is outlined by the sequence diagram below:

Example with outbox pattern — Example With Outbox Pattern

Let's explore these concepts a bit further.

Persisting Events With Transactions

First and foremost, events are persisted in the database, leveraging a transaction.

/**
 * Modified pseudocode leveraging the Outbox pattern.
 */
 
Database.StartTransaction();
  Database.CreateOrder(orderInfo);
  Database.PersistEvent(new OrderCreatedEvent(orderInfo));
Database.EndTransaction()

/**
 * Modified pseudocode leveraging the Outbox pattern.
 */
 
Database.StartTransaction();
  Database.CreateOrder(orderInfo);
  Database.PersistEvent(new OrderCreatedEvent(orderInfo));
Database.EndTransaction()

As illustrated by the pseudocode above, the two operations now execute atomically to ensure data consistency.

Background Worker to Publish Events

The background worker periodically queries the persisted events (referred to as the "outbox" entries), publishes them to the message broker, and then marks them as processed. This process can be accomplished via polling.

Considerations

Below are a few considerations that are important to keep in mind when implementing the Outbox pattern.

Order of publishing: Events must be sent in the same order in which the service updates the database. For example, OrderUpdated should not be published before an OrderCreated event.
Idempotent consumers: It is good practice to ensure that consumers are idempotent. In the case that duplicate messages are published, there would be no impact.

Conclusion

Events are persisted in the database alongside the create/update/delete of an entity, leveraging a transaction.
A background worker is responsible for polling the events persisted in the database, publishing to a message broker, and finally marking the events as processed.
The Outbox pattern improves the reliability of a system given that events will not be lost.