This article was originally published on my website — https://kislayverma.com/programming/design-review-checklist-for-distributed-systems/

本文最初发布在我的网站上-https: //kislayverma.com/programming/design-review-checklist-for-distributed-systems/

Distributed system design is a hard problem, made all the worse because the design process gives no direct feedback. Problems stemming from faulty design often show up as scalability problems, resilience problems or data issues. However, solving those problems is often the equivalent of addressing the symptoms and not the disease — we may be able to patch up the system enough to keep it running, but the underlying design issues remain and can be triggered again under different circumstances. It takes a lot of effort, not to mention organizational wrangling, to be able to analyze design related root causes when the system is failing in production.

分布式系统设计是一个难题，由于设计过程没有直接反馈 ，因此使情况更糟。 由错误的设计引起的问题通常表现为可伸缩性问题，弹性问题或数据问题。 但是，解决这些问题通常等同于解决症状，而不是解决疾病-我们也许能够对系统进行足够的修补以使其运行，但是潜在的设计问题仍然存在，并且可以在不同情况下再次触发。 系统生产失败时，要分析与设计相关的根本原因，需要花费很多精力，更不用说组织机构的争执了。

As with an earlier post on code reviews for distributed systems, this article is a simplistic checklist of things that I look out for when reviewing design of a distributed system functionality (anything which requires multiple systems to work together).

与之前有关分布式系统代码审查的帖子一样，本文是检查分布式系统功能(所有需要多个系统协同工作的项目)的设计时要注意的事情的简单清单。

I think of distributed design issues in terms of three buckets : Consistency-vs-Availability, Domain Coupling, and Observability. The first two often leak into each other because a distributed system is like a complex mesh and each design choice impacts multiple other things. Each bucket is a huge huge topic in its own right, so the following guidelines represent a minimum level of scrutiny that I believe should be imposed on any design.

我从三个方面来考虑分布式设计问题： 一致性-相对-可用性 ， 域耦合和可观察性 。 前两个经常相互泄漏，因为分布式系统就像一个复杂的网格，并且每个设计选择都会影响多个其他方面。 每个存储桶本身就是一个巨大的巨大主题，因此以下准则代表了对任何设计都应施加的最低审查级别。

Depending on the use-case and context of the problem, you should go far deeper into specific aspects after these basics have been checked off. Conversely, if there’s a problem with these, be extremely cautious.

根据问题的用例和上下文，在检查了这些基础知识之后，您应该更深入地研究特定方面。 相反，如果存在这些问题，请格外小心。

一致性或可用性 (Consistency or Availability)

I should remind you upfront that when I say “system” in this article, I mean a collection of independent systems which collaborate in different ways to deliver the final user experience. And when we talk about consistency versus availability, we are talking about all the involved systems.

我应该提前提醒您，当我在本文中说“系统”时，是指一组独立的系统，它们以不同的方式进行协作以提供最终的用户体验。 当我们谈论一致性与可用性时，我们是在谈论所有涉及的系统。

CAP Theorem tells us that we can choose any two out of consistency, availability, and partition tolerance for our system. However, network partitions are a fact of life, hence the true choice of CAP theorem is between consistency and availability — we can have an “AP” system (available, aka continues to work under partition) or a “CP” system (fails if all involved components are not alive and well).

CAP定理告诉我们，我们可以从系统的一致性，可用性和分区容忍度中选择任意两个。 但是，网络分区已成事实，因此，CAP定理的真正选择是在一致性和可用性之间—我们可以拥有一个“ AP”系统(可用，又可以在分区下继续工作)或一个“ CP”系统(如果失败，所有涉及的组件都还没有运行良好)。

A fundamental rule of software architecture is that all software fails. So let’s say that we need consistency across three components for a feature to work as designed. This makes the feature brittle because if any one of the components is down, the feature does not work. We now have the dreaded “Single Point of Failure” — in fact we have three of them!!! The more we try to make the entire system consistent, the more we make it susceptible to failure at the slightest glitch. The more components that must stay in sync, the worse this gets.

软件体系结构的基本规则是所有软件都将失败 。 因此，假设我们需要三个组件之间的一致性才能使功能按设计工作。 这会使功能部件变脆，因为如果任何一个组件出现故障，该功能部件将无法使用。 现在，我们有了可怕的“ 单点故障” ，实际上，我们有三个！！！ 我们越努力使整个系统保持一致，就越容易使它在最小的故障下容易发生故障。 必须保持同步的组件越多，情况就越糟。

Fortunately, there is a way out.

幸运的是，有一条出路。

When looking at individual systems, CP and AP can be binary choices (e.g. MySQL is consistent, Cassandra is not), but when looking at a distributed system, they can start taking on shades of grey. Each component may be consistent (order, inventory and payment), but the overall system can still be designed to be eventually consistent. This gives us the necessary leeway to add some availability to our system. So the main guideline here is to design the OVERALL system for availability, such that the complete functionality can be achieved over time despite individual subsystems being intermittently unavailable.

当查看单个系统时，CP和AP可以是二进制选择(例如，MySQL是一致的，而Cassandra不是)，但是当查看分布式系统时，它们可以开始呈现灰色阴影。 每个组件可能是一致的(订单，库存和付款)，但是整个系统仍可以设计为最终一致的。 这为我们提供了必要的回旋余地，以增加我们系统的可用性。 因此，这里的主要指导原则是设计OVERALL系统的可用性，以便尽管各个子系统间歇性地不可用，但随着时间的推移可以实现完整的功能。

使用异步消息传递进行通信 (Use asynchronous message passing for communication)

This is the most powerful weapon we have for taking off the pressure of consistency in favour of availability and features as a major guideline in the Reactive Manifesto. Consider making the communication between components asynchronous (message passing over a message broker) instead of making it a request-response style API call. If synchronous communication is the crack cocaine of Silicon Valley, synchronous API calls are the crack cocaine of the distributed system design. Consider this — if two systems don’t have to be consistent, then why should we do anything immediately (as the synchronous communication model demands). A request-response model creates a form of temporal coupling (“serve this request right now!”) between the caller and callee, which causes the former to fail if the latter is unavailable at a time, which can then cause cascading failures in its callers and so on. Asynchronous communication allows the called system to process requests at its own pace, thereby taking off the pressure of availability.

这是我们缓解一致性压力以支持可用性和功能的最强大武器，这是《 React式宣言》的主要指南。 考虑使组件之间的通信异步(通过消息代理传递的消息)，而不是使其成为请求-响应样式的API调用。 如果同步通信是硅谷的可卡因 ，那么同步API调用就是分布式系统设计的可卡因。 考虑一下-如果两个系统不必保持一致，那么为什么我们要立即做任何事情(如同步通信模型所要求的)。 请求-响应模型在调用者和被调用者之间创建了一种时间耦合形式(“立即为该请求服务！”)，如果后者一次不可用，则会导致前者失败，从而导致其级联失败。来电者等等。 异步通信允许被叫系统按照自己的节奏处理请求，从而减轻了可用性的压力。

While asynchronous messaging is a powerful tool, there are several things that MUST be kept in mind when adopting it.

尽管异步消息传递是一个功能强大的工具，但在采用异步消息传递时必须牢记一些事情。

1. Define the minimal acceptable user experience — For every end-user experience, define the absolute minimal consistent experience. E.g. If a user wins an online game, must we credit bonus points, award him a new position on the leaderboard, notify all his friends, and send him a notification in an all-or-nothing manner? As should be obvious, the more we can agree to do outside the core, consistent experience, the less likely it is that we will encounter system failure. We must be ruthless about this when discussing requirements and do only the minimum needed to support that — everything else should be done asynchronously.

   定义最小可接受的用户体验 -为每种最终用户体验定义绝对的最小一致体验。 例如，如果用户赢得了在线游戏，我们是否必须积分点积分，在排行榜上授予他新的职位，通知他的所有朋友并以全有或全无的方式向他发送通知？ 显而易见的是，我们越愿意同意在核心，一致的经验之外进行工作，遇到系统故障的可能性就越小。 在讨论需求时，我们必须对此保持冷酷无情，并仅做最少的支持工作即可，其他一切都应异步进行。

2. Explicitly guarantee eventual consistency : A single user action or client request can modify data across multiple components, and the design should guarantee that all these systems will come into consensus with some specified time — even if it is via a distributed rollback.

   明确保证最终的一致性 ：单个用户操作或客户请求可以修改多个组件之间的数据，并且设计应保证所有这些系统在某个指定的时间达成共识-即使是通过分布式回滚也是如此。

3. Guarantee SLA for consistency systemically : This point is worth calling out all by itself. We may have a plan for eventual consistency, but without the mechanism for setting and enforcing a set time frame on it, it is impossible to detect failure from slow processing. Since we cannot determine whether an event threw an error during processing or whether it got dropped in the network, an explicit hard bound in time is necessary to maintain the eventual consistency guarantee.

   系统地保证SLA的一致性 ：值得一提。 我们可能有一个最终一致性的计划，但是如果没有在其上设置和执行设置时间框架的机制，就不可能从缓慢的处理中检测到故障。 由于我们无法确定事件是在处理过程中引发错误还是在网络中丢失，因此必须有明确的时间限制来维持最终的一致性保证。

域耦合 (Domain Coupling)

Good distributed system design essentially hinges on separating distinct things from each other at the right level of abstraction. This line of separation is called the domain boundary and is identified by a unique language of communication and interfaces for functionality unique to that domain. E.g. A message broker domain encapsulates messages, delivery guarantees, storage media etc. Payments domain encapsulates transactions, payment gateways etc.

好的分布式系统设计实质上取决于在正确的抽象级别上将不同的事物彼此分离。 此分隔线称为域边界，并通过该域唯一功能的通信和接口的唯一语言进行标识。 例如，消息代理域封装消息，传递保证，存储介质等。支付域封装交易，支付网关等。

While this broadly maps to the concept of bounded-context in microservice language, the idea is applicable broadly and recursively. E.g. Payments can be a domain, inside which payment gateways and transactions can be sub-domains, and so on.

尽管这广泛地映射到微服务语言中的有界上下文的概念，但是该思想可以广泛且递归地应用。 例如，付款可以是一个域，在其中付款网关和交易可以是子域，依此类推。

The main idea of designing a distributed system in terms of domain is in terms of decoupling domains at the same level, while building cohesion at the level of their parent domain (if any).e.g. In the example above, we would try to keep payment gateways and transactions as decoupled from each other as possible in terms of implementation, but being part of the same domain should dictate a certain consistency in terminology and data models.

在域方面设计分布式系统的主要思想是在相同级别解耦域，同时在其父域(如果有)级别建立凝聚力。例如，在上面的示例中，我们将尝试保持付款网关和事务在实现方面应尽可能彼此分离，但它们属于同一域，则应在术语和数据模型上规定一定的一致性。

1. Create domain boundaries : With the huge uptick in adoption of microservices, it becomes very important to identify the domain of each component in a manner decoupled from the underlying technical implementation. It is important to identify which components belong to which domain and how external systems talk to these systems. We might use multiple services to handle a shipment (tracking service, scanning service, audit service etc), but external users should work with a cohesive “logistics” domain entity and API. A very good tool for building domain boundaries are API gateways which can abstract the internal details of a domain behind higher order APIs.

   创建域边界 ：随着微服务的采用Swift增加，以与底层技术实现脱节的方式标识每个组件的域变得非常重要。 重要的是要确定哪些组件属于哪个域以及外部系统如何与这些系统通信。 我们可能使用多种服务来处理货件(跟踪服务，扫描服务，审计服务等)，但是外部用户应使用具有凝聚力的“物流”域实体和API。 API网关是构建域边界的一个非常好的工具，它可以在高阶API之后抽象域的内部细节。

2. Use standard domain language to communicate between systems : Chatter between two components should be in terms of existing entities + their states + actions possible on them instead of some newly created constructs which belong to neither domain. We can use constructs from either side of the communication for this, depending on whether we want events or messages. If you find that communicating between two components requires creation of some special language, it “might” mean that there is some problem in the way these two components are separated or perhaps we are missing another component that should exist between these two.

   使用标准的域语言在系统之间进行通信 ：两个组件之间的颤抖应以现有实体+它们的状态+对它们可能采取的动作为依据，而不是一些不属于这两个域的新创建的构造。 为此，我们可以使用通讯双方的构造，具体取决于我们需要事件还是消息。 如果您发现两个组件之间的通信需要创建某种特殊的语言，则“可能”表示这两个组件分离的方式存在问题，或者我们可能错过了这两个组件之间应该存在的另一个组件。

3. Separate multi-domain/multi-component actions into workflows — A very common way domain coupling occurs is when one component starts taking end-to-end control of a multi-component workflow. This means that this one domain now knows about various other domains, their behaviour, and the nature of a “workflow” outside of its own boundary. This awareness makes the component coupled to the existing workflows and hence difficult to evolve.

   将多域/多组件操作分离到工作流中 -域耦合发生的一种非常常见的方式是，当一个组件开始对多组件工作流进行端到端控制时。 这意味着该域现在知道其他各个域，它们的行为以及其自身边界之外的“工作流”的性质。 这种意识使组件耦合到现有的工作流程，因此很难进行开发。

4. If our features invoke multiple components, we should separate this orchestration out of the core services modelling the domain into stateful orchestration components. This can be dedicated orchestrating services or some sort of BPM systems. Statefulness means that we get benefits like retries, error reports, SLA etc from one place.
   如果我们的功能调用了多个组件，则应将此编排从建模该域的核心服务中分离为有状态的编排组件。 这可以是专用的编排服务或某种BPM系统。 有状态性意味着我们可以从一个地方获得重试，错误报告，SLA等好处。
5. Using choreography to build workflows is also a viable option if we cannot use explicit orchestration for everything, but we should use some way of tracking task completion SLA to reduce the brittleness for long-lived workflows.
   如果不能对所有事物都使用显式编排，则使用编排来构建工作流也是一个可行的选择，但是我们应该使用某种方式来跟踪任务完成SLA，以减少长寿命工作流的脆弱性。

6. Model business processes across components : A corollary to the above point is that we should model businesses processes end-to-end regardless of technical boundaries. Since we are already decoupling domains by offloading orchestration to workflows, it doesn’t make sense to build workflows within narrow team boundaries. We should make the workflows encompass as much of the business process as a whole as we can — this will build central repositories of business knowledge and provide deep visibility into the state of operations.

   对跨组件的业务流程进行建模 ：得出以上结论的推论是，无论技术范围如何，我们都应端到端对业务流程进行建模。 由于我们已经通过将业务流程转移到工作流中来分离域，因此在狭窄的团队边界内构建工作流没有任何意义。 我们应该使工作流尽可能涵盖整个业务流程，这将建立业务知识的中央存储库，并提供对操作状态的深入可见性。

7. Prefer events over messages : A further component on decoupling domains is to prefer the use of pub-sub style events rather than targeted messages. While this is contingent on many other factors, it further enhances agnosticity towards other domains because the publisher doesn’t care about consumers in the pub-sub model, hence coupling between publisher and consumer domain is eliminated.

   优先于消息而不是消息 ：解耦域的另一个组件是更喜欢使用发布订阅样式事件，而不是目标消息。 尽管这取决于许多其他因素，但由于发布者不在乎发布-订阅模型中的消费者，因此它进一步增强了对其他域的不可知性，因此消除了发布者与消费者域之间的耦合。

可观察性 (Observability)

In the words of Charity Majors, Observability is the ability to answer new questions about a system without having to peek inside it. I think of observability in two flavours : technical and business. We should be able to explain the technical state of the system, and we should be able to determine if it is doing what it is supposed to do from a business metrics perspective.

用慈善专业人士的话说， 可观察性是回答有关系统的新问题而不必窥视系统内部的能力 。 我认为可观察性有两种：技术和业务。 我们应该能够解释系统的技术状态，并且我们应该能够从业务指标的角度确定系统是否正在执行应做的工作。

1. Use event data to build metrics : The basic unit of work inside any system is an event of some sort occurring explained in the system’s own domain language. The event can be of any type (e.g. ORDER_CREATED, REQUEST_RECEIVED, ERROR_RESPONSE_RETURNED). An idea worth pursuing is that instead of the common logs-traces-metrics approach, we consider the entire system information being modelled and emitted as events and stored outside in an analytical database for deriving intelligence. The benefit of having raw event data is that in sync with the domain modelling approach (something happened), and raw data can be used to derive new metrics at any time. This is way better than trawling through some combination of unstructured text logs, spans, traces, and arbitrary Prometheus/Statsd type metrics.

   使用事件数据来构建指标 ：任何系统内的基本工作单元都是以系统自身的域语言解释的某种事件。 该事件可以是任何类型(例如，ORDER_CREATED，REQUEST_RECEIVED，ERROR_RESPONSE_RETURNED)。 值得追求的想法是，我们考虑将整个系统信息建模并作为事件发出，并存储在分析数据库中以获取情报，而不是使用通用的对数跟踪度量标准方法。 拥有原始事件数据的好处是与领域建模方法(发生了某些事情)同步，并且原始数据可随时用于导出新指标。 这比通过非结构化文本日志，跨度，迹线和任意Prometheus / Statsd类型指标的某种组合进行拖网更好。

2. Store raw data in a central place : The only way to honour the definition of Observability given above is to store raw data from systems because the moment we start dealing only in pre-defined metrics, we are bound to them and lose the ability to answer any “new” questions. One common argument against raw data is volume, but there are ways to mitigate that (sampling etc.), while the ability to diagnose your systems in face of new failure modes is priceless. And since the overall system is distributed, having the observability data also in distributed silos will be much less powerful than bringing all of it together to derive holistic insights.

   将原始数据存储在一个中央位置 ：遵守上面给出的Observability定义的唯一方法是存储系统中的原始数据，因为从开始仅按预定义指标进行处理的那一刻起，我们就被束缚在其中，并且丧失了回答的能力。任何“新”问题。 反对原始数据的一个常见论点是数量，但是有许多方法可以减轻这种压力(采样等)，而面对新的故障模式来诊断系统的能力是无价的。 而且，由于整个系统是分布式的，因此将可观测性数据也包含在分布式孤岛中，其功能将远不如将所有这些聚集在一起以获取整体见解。

一般准则 (General Guidelines)

1. Use asynchronous frameworks for implementation — I have written before about how we can use asynchronous programming to scale our systems significantly. So when invoking a remote system, we should ensure that we are doing it in an asynchronous manner so as to not block application threads. This is a bit of early implementation/design choice and if your application framework does not allow this or the base framework of the application is not designed for this then you are out of luck. If you have the option of going asynchronous, always take it — your system and your team will thank you for it when that unexpected spurt of traffic comes.

   使用异步框架进行实现 —我之前写过关于如何使用异步编程来显着扩展系统的文章 。 因此，在调用远程系统时，我们应确保以异步方式进行操作，以免阻塞应用程序线程。 这是早期实现/设计的选择，如果您的应用程序框架不允许这样做，或者应用程序的基础框架不是为此目的而设计的，那么您就不走运了。 如果您可以选择异步，请始终采用它-当流量突然激增时，您的系统和团队将感谢您。

2. Know your callers — By definition, no one is in charge in a distributed system. As a result, we should take as many precautions for internal systems as we might take for external systems. The least of these is to enforce rate limits if you can, but at least know your callers. This will help isolate the origin of trouble when things go wrong — identifying sources of tracking using IP addresses when the system is on fire is not fun.

   了解您的呼叫者 -根据定义，分布式系统中没有人负责。 因此，对于内部系统，我们应该采取与对外部系统一样多的预防措施。 这些中的至少一项是在可能的情况下强制执行速率限制，但至少要了解您的呼叫者。 当出现问题时，这将有助于隔离故障根源-当系统着火时使用IP地址识别跟踪的来源并不有趣。

3. Know when to fail — I have spoken a lot about how to make our systems more available under varying conditions, but there are situations when failing is better than doing the wrong thing (Someday I will write an article about how our message passing based design resulted in way more orders than we had inventory for). Consistency is a perfectly acceptable choice in many scenarios, and we should be careful to not overcompensate against them.

   知道何时失败 -我已经讲了很多关于如何在各种情况下使我们的系统更可用的方法，但是在某些情况下，失败比做错事要好(总有一天，我会写一篇文章，介绍基于消息传递的设计是如何产生的以比我们库存更多的订单)。 在许多情况下，一致性是一个完全可以接受的选择，我们应该注意不要过度补偿它们。

I hope you find these guidelines useful in reducing the most commonly found issues in distributed systems design. I would to hear if you have some other considerations that you find simple to apply but very effective — we can add them here!

我希望您发现这些准则有助于减少分布式系统设计中最常见的问题。 我想听听您是否有其他一些考虑因素，您认为它们很容易应用但非常有效-我们可以在此处添加它们！

Read also : Code review checklist for distributed systems

另请参阅： 分布式系统的代码审查清单

To keep up with more articles on software craftsmanship, technology and platforms, join my mailing list. I post new articles once or twice a week.

要了解有关软件Craft.io，技术和平台的更多文章，请加入我的邮件列表 。 我每周发布一次或两次新文章。