The Narrative

In the biblical narrative, Adam and Eve are tempted by the serpent. They give in and are expelled from the Garden of Eden into a world of suffering.

In software engineering, the people designing a new distributed system are tempted by the greatest lie of them all: “Let’s release something ASAP; we will fix/decide later.” They give in and condemn the project to be a source of exponentially compounding rot and an endless incident generator.

This is the Original Sin-an idyllic garden lost because of a fruit. Consistent development velocity and the reliability of the software over its whole lifetime are lost to moving fast for the first couple of weeks and then losing that speed anyway.

At its core, the software engineering issue boils down to the following: the absence of explicit foundational choices results in implicit and ever-compounding system deficiencies that take more and more effort to amend as time goes on and the system keeps evolving.

If we take a little time to make explicit choices about how our system will deal with the challenges all distributed systems face, we should end up in a spot that makes everyone happy: entire classes of bugs are mitigated, the software can be adequately extended even after years of evolution, and the company earns a great reputation for making reliable products.

With that in mind, here is the list of the 10 things that make or break distributed software, organized roughly by a combination of two factors: how likely you are to have issues if you ignore them and how hard the solution is to retrofit later.

1. Adversarial List

These are the things each of your business domains needs to withstand. You need to know what unique problems your software faces before you can decide on the solutions.

For example, if you’re developing financial software, you will have very strict requirements. You will need to properly deal with duplicate deliveries, clock skew, cascading failures, partial writes, slow nodes, and many other things. Having that list makes it clear that you need strong idempotency, a linearizable consistency model, a single source of truth, excellent observability, strict access control, etc.

2. Request Lifecycle

This specifies how a single request behaves as it traverses the system and encounters trouble.

Define your error model (common error envelope, error codes, retryability, server vs. client, user-facing vs. internal errors, mapping between protocols), timeouts, retry policy, deadlines, cancellation, backoff, and jitter.

3. Idempotency

Define how you’re going to deal with duplicate requests/messages. Exactly-once delivery is a unicorn, so in the real world assume at-least-once delivery and turn it into effectively-once processing through proper idempotency.

4. Consistency Model

This is the CAP/PACELC decision. You need to choose what kind of consistency you need: linearizable, causal, read-your-writes, eventual. And that decision must be made for each of your business domains separately. Achieving stricter levels of consistency comes at a cost, so it’s wise to choose the less strict one if the domain can tolerate it.

5. Data Ownership

Generally, each entity should have exactly one writer (its owner). Everyone else caches or reads and learns of changes through a defined propagation path. It’s the prerequisite for reconciliation to even mean something. “Make X agree with Y” is undefined unless you’ve decided which is the source of truth.

6. Time & Causality

Decide how you order events when it matters for correctness. Logical clocks (Lamport), hybrid-logical clocks (HLC), or vector clocks? Ordinary wall clocks will fail you, so don’t use them to decide the order and causality of events. Only use them for display and rough metrics.

7. Backpressure Control

This will dictate what the system does under sustained aggregate overload to prevent cascading collapse. You need to make decisions about max queue sizes, load shedding, circuit breakers, and rate limiting.

8. Observability

Sooner rather than later, you will want to know what’s happening in your system, answer all sorts of questions about it, and know if it’s healthy. All of your services must propagate the correct context and share the same understanding of observability for you to be able to have end-to-end insights.

Standardize structured logging, tracing, metrics, and automated alerting.

9. API Evolution

In a distributed system, you can never atomically upgrade all nodes. Without an evolution policy, every change risks a partial-deploy outage.

So decide on the evolution-friendly serialization format, backward- and forward-compatibility policy, deprecation schedule, etc.

10. Security & Isolation

Think about where the trust perimeter sits and how services authenticate each other. Are all the services in the “trusted network,” or do you need specific rules with zero trust by default?