There are many flavours of event-driven architecture. I tend to speak and write a lot about event choreography, as 1) it’s a pattern that allows for effective separation of concerns, and 2) it’s a more accessible introduction to event-driven architectures than things like event sourcing. (Although I do make noise about event sourcing, too.)

In particular, in-process event choreography is extremely useful as it allows different parts of the same codebase to take action without being directly coupled together, but still allows for all of the state changes to be committed within a single transaction. Similarly, for distributed systems, event choreography is a powerful pattern to deliver simple, scalable workflows across disparate parts of an organisation.

Choreography versus orchestration

It is, however, important to draw a distinction between choreography and orchestration. If the system that will perform the action makes the decision, it’s choreography. If the system that will perform the action gets told to do it by another system, it’s orchestration.

The key is which system decides that an action should be taken.

• Choreography is when a system takes independent action of its own volution as a result of the action of another system.
• Orchestration is when some kind of process manager instructs a system to take action, usually as a result of an action of another system.

So which should I use? How do I decide?

This brings us to the question of which we should use and when. A simple rule of thumb is if a business process has a name and some state, it’s an entity. When we speak in terms of rules, e.g. “When a customer signs up, send them a welcome email” then choreography is a good candidate. When we speak in terms of process names, e.g. “Your application process is at the ‘Pending approval’ stage” then it’s a good hint that we’re in orchestration territory.

How do we model an orchestrated process?

We’ve already established that an orchestrated process has both state (“Where is this process up to?”) and behaviour (i.e. state transitions). This sounds suspiciously like… a domain entity.

If we accept that our process is a domain entity, many things fall into place. For one, we can query its state, and we can instruct it to change state. It can also refuse to undergo an invalid state transition, e.g. a call to a LoanApplicationProcess.Approve(...) should probably fail if LoanApplicationProcess.CurrentState == LoanApplicationState.NotYetSubmitted.

Doesn’t process orchestration mean I need an enterprise service bus? Should I get one?

No matter what problem you have, an enterprise service bus can always find a way to make it worse.

Thinking purely in terms of domain-driven design, we already have all of the puzzle pieces we need to solve this problem: domain logic and state persistence. This sounds suspiciously like it woudld fit into the normal structure of our domain model. We presumably have a repository implementation of some sort, and a persistence layer.

When our process needs to respond to an external event, we can use a vanilla event handler to hydrate our process instance and dispatch the event to the process.

The only missing piece is when we have a process that is dependent on time, which can be solved quite easily with a metronome which hydrates every non-finished process entity and calls a .TimePasses(DateTimeOffset now) method on it.

In a previous article, I wrote a little about events’ granularity but I wanted to expand on it here a little.

It’s important to understand and accept that the granularity of events will be different across each system - and that there is no One True Way™ to represent a thing that happened. An event’s job is to represent the thing that happened from the perspective of the system that performed the action. In other words, we just lay it out there and let others make of it what they will.

Our system shouldn’t speculate and shouldn’t guess who our subscribers are or what else they might want to know. That’s their job. If we adapt to our n subscribers then we have n reasons to change. If we just announce what happened from our own perspective then we only have one reason to change, i.e. our own system.

In-process events versus long-lived message contracts

NOTE: For the purposes of this article, I’m not talking about event sourcing. We’ll cover that in another article, and we structure, broker and apply our events slightly differently when we’re doing that.

Within a single codebase and a single process, lightweight events (domain events or otherwise) are an excellent way to decouple concerns whilst still ensuring transactional consistency. Events can be simple POCOs, are refactor-friendly, and are able to be changed within the space of a single commit without affecting any downstream applications.

By contrast, a long-lived event is generally written to a durable, persisted store of some description (and for some value of “persisted”) such as Kafka, Rabbit, ASB, Kinesis etc. These event types are generally intended to be consumed by other applications (unless we’re also using event sourcing and persisted events are our primary source of truth), and thus form part of our application’s API contract - and should be treated in the same way as any other part of that contract, namely being versioned and supported for agreed periods of time. We generally publish these event schemas, either via a schema definition like Apache Avro or in a pre-packaged “message DTOs” package or similar.

Reprojecting fine-grained events into coarse-grained events

In this article I’m going to use the term “domain event” to mean an event that occurred within a single domain and which is not visible in that form outside of that domain, and the term “business event” to mean an event that occurred within one domain but that is visible outside of that domain. Those definitions might blur slightly (or a lot) depending on context and perspective but they’ll serve well enough to illustrate the points I’d like to make here.

It’s entirely possible (and reasonable) to map domain events to business events and vice versa. For a concrete example, let’s say that we’re using some simple, in-process domain event brokerage. We’ll use a sample Barista class that looks something like this:

public class Barista
{
    // ...

    public void MakeCoffeeFor(Customer customer, Order order, IEspressoMachine espressoMachine)
    {
        if (!order.IsPaid) throw new OrderNotPaidException("No coffee for you!");
        if (customer.IsShouty) throw new UnrulyCustomerException("Manners maketh [wo]man.");

        var coffee = espressoMachine.PullShots(order.NumberOfShots);
        customer.Accept(coffee);

        DomainEvents.Raise(new BaristaMadeCoffeeForCustomerEvent(this, customer, coffee))
    }
}

Contrived examples notwithstanding, we model our real-world domain by following the AAA pattern and first giving our Barista the autonomy to refuse to make coffee if it hasn’t been paid for, and to refuse service to unruly customers¹. Once our preconditions are satisfied, the coffee is (presumably) made, and a BaristaMadeCoffeeForCustomerEvent domain event is raised. This event is brokered in-process during the same unit of work in which the Barista makes coffee.

Once our unit of work completes, we can map our in-process domain events to business events, which can then be published externally to a Kafka topic or similar. A useful pattern is something along these lines:

public interface IMapToBusinessEvent<TDomainEvent> where TDomainEvent : IDomainEvent
{
    IEnumerable<IBusinessEvent> Map(TDomainEvent domainEvent);
}

with a sample implementation looking something like this:

public class BaristaMadeCoffeeForCustomerEventMapper : IMapToBusinessEvent<BaristaMadeCoffeeForCustomerEvent>
{
    // ...

    public IEnumerable<IBusinessEvent> Map(BaristaMadeCoffeeForCustomerEvent domainEvent)
    {
        yield return new BaristaMadeCoffeeForCustomerBusinessEvent
        {
            Timestamp = _clock.Now,
            BaristaId = domainEvent.Barista.Id,
            CustomerId = domainEvent.Customer.Id,
            Description = domainEvent.Coffee.ToString()
        };
    }
}

This pattern gives us a few options:

1. We can use rich object references in our in-process domain events. This means that we don’t have to resolve the same entity multiple times from a repository just to deal with it in the same unit of work.
2. We can downcast our rich object references (which won’t serialise very well) to just identifier references when we map to the business event. This also allows us to include only the information that should be externally-visible.
3. We can choose to publish multiple business events based on a single domain event.
4. We can choose to not publish any business event at all from a domain event. It’s entirely legitimate that there will be domain events that are purely internal to our application and which we do not want to publish to the broader app ecosystem.
5. We can refactor our internal, in-process domain event whenever we like, as long as we project it into the same shape of externally-visible business event as before.

Reprojecting coarse-grained events into fine-grained events

It’s entirely possible (and very likely) that one system will care about events at a finer granularity than another.

Our contrived barista example

To continue our barista example, let’s look at two domains: ordering and fulfilment. Our ordering system is responsible for recording a customer’s order, taking payment and then announcing, “Order up!” via an OrderPlaced event. According to our ordering system, an order is an order is an order. Our stream might look something like this:

OrderPlaced { customerId: 1, sku: 'FLATWHITE', size: 'M', sugars: 1, milk: 'FULLFAT' }
OrderPlaced { customerId: 2, sku: 'CAPUCCINO', size: 'M', sugars: 0, milk: 'SKIM' }
OrderPlaced { customerId: 3, sku: 'MOCHA', size: 'L', sugars: 2, milk: 'FULLFAT' }
OrderPlaced { customerId: 4, sku: 'TOASTED SANDWICH', fillings: 'SURPRISE ME' }

In the physical world, a barista is likely to project an OrderPlaced event into a bit more granular a mental stream. Specifically, they’re likely to steam more than one coffee’s worth of milk and perhaps attempt to aggregate events into a few streams that might look more like this:

OrderRequiringFullFatMilkPlaced { size: 'M' }
OrderRequiringSkimMilkPlaced { size: 'M' }
OrderRequiringFullFatMilkPlaced { size: 'L' }

But why would they do this?

The reprojection of events in this case is because the barista will take a different action based on the nature of the event. If more full-fat milk is required then they’ll just add more milk to the steaming jug before starting, whereas if skim milk is required then they need to steam that separately. If the order is for a toasted sandwich then the Barista doesn’t care at all so it doesn’t even bother to reproject that OrderPlaced event. They make coffee. Sandwiches are someone else’s problem.

The key point here is that while one system (the ordering system) doesn’t draw a distinction between orders containing different kinds of milk, or even drinks versus food at all, the fulfilment system is likely to take very different actions based on some of the events’ payloads.

A real-world example

Contrived examples aside, let’s consider something more serious: a court order prohibiting a convicted criminal from changing address without first applying to a court to vary its orders.

We have a government agency responsible for knowing where a citizen lives and publishing a CitizenChangedAddressEvent to a Kafka topic whenever such an event occurs. As far as that agency is concerned, this is a legitimate action for a person to take. They’ve moved house; they tell their government².

CitizenChangedAddressEvent: { id: '<some citizen guid>', address: '123 Imaginary Street, Nowheresville' }

The court systems subscribe to this event stream, and care about the CitizenChangedAddressEvent. When a citizen changes address, the system processes that event and performs a series of checks, then potentially projects that event into a more granular event relevant to its own domain:

namespace WhenACitizenChangesAddress
{
    public class ReprojectTheEventStream : IHandleBusEvent<CitizenChangedAddressEvent>
    {
        // ...

        public async Task Handle<CitizenChangedAddressEvent>(CitizenChangedAddressEvent e)
        {
            if ( /* citizen is convicted criminal and has moved across state boundaries */ )
            {
                // ...
            }

            if ( /* citizen is a child subject to a court order */ )
            {
                var child = _childRepository.Get(e.Id);
                DomainEvents.Raise(new ChildSubjectToCourtOrderMovedWithoutConsentEvent(child));
            }
        }
    }
}

resulting in an event stream that could look something like this:

ConvictedCriminalMovedInterstateEvent: { id: '<some citizen id>' }
PersonSubjectToDomesticViolenceOrderMovedTooCloseToVictimEvent:  { id: '<some citizen id>' }
ChildSubjectToCourtOrderMovedWithoutConsentEvent:  { id: '<some citizen id>' }

Within the court system’s domain, these events are likely to be dealt with completely differently. A convicted criminal crossing state boundaries is likely to have a warrant issued for their arrest; a person subject to a domestic violence order who has moved too close to their victim(s) may or may not have a warrant issued; a child who is named in a court order might have a travel ban imposed so that they cannot leave the country.

The court system can then implement specific handlers for these more-specific event types:

namespace WhenAChildSubjectToACourtOrderMovesWithoutConsent
{
    public class PutThemOntoAnAirportWatchList: IHandleEvent<ChildSubjectToCourtOrderMovedWithoutConsentEvent>
    {
        // ...

        public async Task Handle(ChildSubjectToCourtOrderMovedWithoutConsentEvent e)
        {
            var child = _childRepository.Get(e.Id);
            _airportWatchList.Add(child);
        }
    }
}

This seems like a lot of indirection for not much value. Why wouldn’t we just check the original event type in our handler for the CitizenChangedAddressEvent? Well, commonly we want to do more than one thing as a result of an event like this. Let’s add another behaviour:

namespace WhenAChildSubjectToACourtOrderMovesWithoutConsent
{
    public class PutThemOntoAnAirportWatchList: IHandleEvent<ChildSubjectToCourtOrderMovedWithoutConsentEvent>
    {
        // ...

        public async Task Handle(ChildSubjectToCourtOrderMovedWithoutConsentEvent e)
        {
            var child = _childRepository.Get(e.Id);
            _airportWatchList.Add(child);
        }
    }

    public class NotifyAllGuardiansOfTheChild: IHandleEvent<ChildSubjectToCourtOrderMovedWithoutConsentEvent>
    {
        // ...

        public async Task Handle(ChildSubjectToCourtOrderMovedWithoutConsentEvent e)
        {
            var child = _childRepository.Get(e.Id);
            var guardians = _guardianLookupService.FindGuardiansOf(child);
            foreach (var guardian in guardians)
            {
                _alertService.Notify(guardian, $"Child {child}'s address has changed without permission from the court.");
            }
        }
    }
}

Now it becomes obvious that if we were to include the same check for “Does this CitizenChangedAddressEvent mean that a child named in a court order has changed address without consent?” then we would start to see copious copied/pasted code to determine if this was the kind of event to which we should respond. Our code becomes much less testable, its number of dependencies explodes and we’re left with a single, gigantic orchestration method.

By contrast, if we adopt a “When X, do Y” pattern, we get a much more expressive, rule-based system that is much easier to discover. It also makes it much easier to additionally define “Also, when X happens, do Z.”

The point of this reprojection is that the publishing system neither knows nor cares how downstream systems will consume its events. It’s much tidier to reproject the coarse-grained event into finer-grained ones at the point in the ecosystem where differentiation makes a difference to the action that will be taken.

… a facetious look at the nature of events.

Anything that happens, happens.
Anything that, in happening, causes something else to happen, causes something else to happen.
Anything that, in happening, causes itself to happen again, happens again.
It doesn’t necessarily do it in chronological order, though.

– Douglas Adams, Mostly Harmless

Principles of expressing events

• Events are immutable facts about the past.
• The system that undergoes the state change is responsible for publishing the event.
• The system publishing the event is responsible for defining the structure of the event.

I’ll deal with each of these principles individually - and slightly out of order. Please forgive the poetic licence.

Principle: The system that undergoes the state change is responsible for publishing the event.

Whilst modesty is admirable in a human, in software it is confusing. The component that has undertaken the action or undergone the state change should be the one to publish an event saying so. Our software should shout about its own doings, and not expect the town crier to follow it around.

Our code should follow the Assert. Act. Announce. pattern. Specifically, it should assert its preconditions, perform precisely its appointed task (and nothing more) and then announce that that task has been performed.

If we have some code that looks like this:

var cat = CatFactory.SpawnOne();
var mat = MatFactory.WeaveOne();
cat.SitOn(mat);

then our Cat implementation could look something like this:

public class Cat
{
    // ...

    public void SitOn(Mat mat)
    {
        // Assert
        if (_remainingLives <= 0) throw new DeadCatException("Dead cats can't sit on things.");

        // Act
        _currentlySittingOn = mat;
        mat.BeSatUponBy(this);

        // Announce
        DomainEvents.Raise(new CatSatOnMatEvent(this, mat))
    }
}

Crucially, the Cat is the one that has undergone the state change so it is the one that emits the event. (Side note: the Mat might also change state by virtue of being sat upon, in which case it should emit its own event.)

Principle: The system publishing the event is responsible for defining the structure of the event.

… and for honouring that contract.

The system that underwent the state change is the one that knows what happened. It is therefore the best placed to define the structure of the event representing that state change. Crucially, the publishing system should not attempt to cater to downstream subscribers who might desire additional information.

Consider the following event:

public class CatSatOnMatEvent
{
    public Instant Timestamp {get; set; }
    public Guid CatId { get; set; }
    public Guid MatId { get; set; }
}

This structure is quite terse but describes exactly what happened and only what happened. This is a Good Thing.

What happens, though, when a downstream subscriber wants to know the cat’s name? We end up with something like this:

public class CatSatOnMatEvent
{
    public Instant Timestamp {get; set; }
    public Guid CatId { get; set; }
    public string CatName { get; set; } // not a good idea
    public Guid MatId { get; set; }
}

What could be so harmful about that? Let’s go one further and suggest that another downstream subscriber wants to know the colour of the mat upon which the cat sat:

public class CatSatOnMatEvent
{
    public Instant Timestamp {get; set; }
    public Guid CatId { get; set; }
    public string CatName { get; set; } // not a good idea
    public Guid MatId { get; set; }
    public Color MatColor { get; set; } // also not a good idea
}

All of a sudden, we discover that our publisher will be responsible for marshalling all sorts of additional information just in case a subscriber wants it. The first subscriber doesn’t care about the colour of the mat and the second doesn’t care about the name of the cat, yet the publisher ends up being obliged to collate both every time it publishes an event just in case a subscriber wants it. This gives the publisher potentially as many reasons to change as there are subscribers - exactly the opposite of what the Single Responsibility Principle advocates.

What the publishing system should be doing is including only the information 1) about which it is authoritative; and 2) necessary to convey the essence of what happened. The system that cares about the name of the cat can then call a service to retrieve the name of the cat; likewise for the system that cares about the colour of the mat.

The publishing system is responsible for honouring that event contract for as long as agreed with its downstream subscribers. For an in-process event a refactoring could be done in a single commit, but for a business event published to a Kafka stream it might need to honour the contract for weeks, months or even years. It is the responsibility of the publisher to honour this contract for as long as its SLA specifies. (Site note: contract tests and/or schema registries will come in useful here.)

Principle: Events are immutable facts about the past.

An event should describe a thing that happened. It must have actually happened at a specific time. Events are expressed in the simple past tense.

The cat sat on the mat.

This is a good event. It’s expressed in the simple past tense. It unequivocally happened. Anything that may have happened after this event does not change the correctness of this event.

The cat is on the mat.

This sentence is in the present tense.

We could validly fire this event infinite times per second provided that there were, indeed, an appropriate cat sitting upon an appropriate mat. It would not be helpful. It would be much more helpful if we were to announce (once) at the moment that the cat were to first sit on the mat that it had done so.

The cat was sitting on the mat.

This is in the past progressive tense. It’s in the past, true, but it describes something that was happening and continuing to happen. Provided that the statement was once true (i.e. that the cat, once upon a time, sat upon the mat), the statement will always continue to be true. While a true statement, it is not helpful.

The cat sits on the mat.

Does it? When? Is it there now? Does it just generally sit on the mat? Do all cats sit on all mats?

The cat will sit on the mat.

Are we sure? How sure? Are we sure that it’s going to sit on the mat at precisely the timestamp on the event? It is a cat, after all. What if it gets hungry? Or bored?

This event is neither immutable (it might or might not happen) nor about the past.

The cat would have sat on the mat.

… but it didn’t because cats are fickle. This is not a thing that happened, therefore it’s not a good candidate for an event.

The cat should sit on the mat in ten minutes.

This is a command, not an event. It’s a valid command but it’s not a fact about the past.

Some other considerations

Commands should not be dressed up as passive-aggressive events.

We want our event publishers to observe the Assert. Act. Announce. pattern. Specifically, the publisher of an event should actively not care about whether any downstream consumers take action as a result of the event.

When an upstream system requires a downstream system to take a specific action as a result of something that happened, it should send an explicit command (either synchronous or asynchronous) to that system rather than announcing an event and then waiting expectantly.

Dinner is served, cat!

This is a passive-agressive event. The publisher of the event is waiting expectantly for their cat to come and eat its dinner, without having expressed that expectation.

Cat was requested to eat dinner.

This is also a passive-aggressive event. It expects a specific action to be taken and it is likely that the publisher of the event is waiting for that action to be completed.

Cat! Come and eat your dinner!

This is a perfectly acceptable command; it’s just not an event. It is an imperative instruction for an entity (cat) to undertake an action (come and eat dinner). It is entirely reasonable to send such a command; just not to present it as an event.

Think about your events’ granularity.

It is useful to structure events according to business events that actually happened and that people would talk about in real terms. It is likely that downstream systems will take different actions based on different events and don’t want to process “catch-all” events that they then have to further filter.

For a more serious example, consider entity-named events (e.g. CustomerEvent) that describe something vaguely related to a customer and, worse, generic CustomerUpdated or event just CustomerUpdate “events” that neither describe what happened or even assert that something did, in fact, happen.

A downstream system will most likely take very different actions as a result of a CustomerCancelledAccountEvent (perhaps sending an apology letter) than a CustomerChangedEmailAddressEvent (sending an alert email to the old address), and a different action again as a result of a CustomerConvictedOfFraudEvent (perhaps suspending shipments to that customer). If all of those happenings are wrapped up in a single CustomerUpdated or CustomerEvent then 1) too much responsibility is pushed onto each subscriber; and 2) the event payload will bloat in order to accommodate all of the different information required for all the downstream subscribers.

Statements about the future are not events.

We see many examples of these kinds of anti-patterns in transaction-processing systems.

For example, future-dated transactions are inserted into a table in the same way as past-dated transactions (strike one: not about the past). Almost invariably they are then updated with an IsProcessed flag or similar when they are actually processed (strike two: not immutable).