WHITE PAPER How To Resolve Network Partitions In Seconds With Lightbend Enterprise Suite Strategies To Seamlessly Recover From Split Brain Scenarios Table Of Contents Executive Summary And Key Takeaways The Question Is When, Not If, The Network Will Fail Distributed Systems Raise Network Complexity Reactive Systems Can Heal Themselves, But Not Network Partitions .5 The Problem High-Level Solution .11 Four Strategies To Resolve Network Partitions 14 Strategy - Keep Majority 15 Strategy - Static Quorum 15 Strategy - Keep Oldest 16 Strategy - Keep Referee 17 Split Brain Resolution From Lightbend 17 Akka SBR .17 Cluster Management SBR 18 The Benefits 21 Serve Customers Better 21 Eliminate Expensive Downtime 21 Immediate Time-To-Value 21 Summary 21 HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE Executive Summary And Key Takeaways In the era of highly-distributed, real-time applications, network issues often result in so-called “split brain” scenarios, where one or more nodes in a cluster becomes suddenly unresponsive This results in data inconsistencies between distributed services that can cause cascading failures and downtime While the industry has turned to Reactive systems to solve issues of application/service level resilience, elasticity, and consistency, network partitions occur outside of the area of concern addressed by these architectures Given the inevitability of network partitions and the impossibility of truly eradicating them, the best solution is to have predetermined strategies that fit business requirements for dealing with this recurring issue quickly and with minimal disruption to overall system responsiveness Lightbend Enterprise Suite, the commercial component of Lightbend Reactive Platform, offers four such strategies as part of its Split Brain Resolver (SBR) feature These strategies, called “Keep Majority,” “Static Quorum,” “Keep Oldest,” and “Keep Referee” can be seamlessly executed in a matter of seconds if and when network partitions occur Users can take advantage of the SBR feature either in development at the code level or at the production/ operations level By avoiding data inconsistencies, failures, and downtime, users of Lightbend Enterprise Suite can better serve their customers, leading to increased retention, growth, and market share HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE The Question Is When, Not If, The Network Will Fail “The network is reliable.” One Of The Fallacies Of Distributed Computing Network issues are unavoidable in today’s complex environments To put it more colloquially: networks are flaky Most users are aware of this and even accustomed to it, and are willing to handle random unresponsiveness from time to time However, user tolerance for network flakiness has limits If a specific website or app repeatedly experiences problems, patience wears thin As network issues mount, it becomes increasingly likely that a user will interact with the offending website or app less often or even abandon it altogether This is not to say that users are the only ones impacted by network issues In a world full of APIs and interconnected systems, network problems affecting one system can easily impact other connected or dependent systems Users interacting with one of those applications through a front-end will likely notice the problem within a short amount of time, but in some cases, it might take many hours or even days for such problems to become apparent For the sake of convenience, this paper will use user experience to highlight the pernicious effects of network issues Most websites and apps access a database or have some form of a data persistence layer Communication from the application layer to the persistence layer is often over a network Thus, for the duration of network issues, problems, and outages, the application becomes unable to perform its normal duties and user experience starts to suffer Network problems can also span a variety of locations They can be widespread across the entire network, occur locally within data centers, or even arise in a single router or on-premise server A complete outage is the nightmare scenario, but even small network hiccups can result in lost revenue For example, high network traffic often creates very slow response times In many cases, these slowdowns are actually worse than broken connections because, even with proper monitoring tools, the offending issue is non-obvious and difficult to diagnose and fix Distributed Systems Raise Network Complexity With distributed systems, various application components (e.g individual microservices and Fast Data pipelines) communicate with each other via some form of messaging One component asks another component for some information A component may communicate to other components a variety of information, such as a state change of a business entity A component may delegate work to other components and wait for a response from the delegates, and so on and so forth HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE E A D B C Figure - Component Messaging Given that, distributed applications can serve to make network issues better or worse, depending on how well they are designed Poorly-designed systems crumble when network problems occur Well-designed systems recover gracefully when the impacted components stop responding or respond very slowly The latest breed of distributed systems are designed from the ground up to be fully prepared for inevitable network issues These systems typically have well-defined default or compensating actions that activate when needed This allows the overall application to continue to function for users even when an application component stops working This new breed of systems is known as Reactive systems Reactive Systems Can Heal Themselves, But Not Network Partitions Reactive systems are designed to maintain a level of responsiveness at all times, elastically scaling to meet fluctuations in demand, and remain highly resilient against failures with built-in self-healing capabilities Lightbend, a leader in the Reactive movement, codified the Reactive principles of responsiveness, resilience, and elasticity, all backed by a message-driven architecture, with the Reactive Manifesto in 2013 Since then, the topic of Reactive has gone from being a virtually unacknowledged technique for building applications—used by only fringe projects within a select few corporations—to becoming part of the overall platform strategy for some of the biggest companies across the world Compared to a traditional system, in which small failures can cause a system-wide crash, Reactive systems are designed to isolate the offending application or cluster node and restart a new instance somewhere else However, at the overall network level, which may span across the entire globe, there exists HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE a fundamental problem with network partitions in distributed systems: it’s impossible to tell if an unresponsive node is the result of a partition in the network (known as a “split brain” scenario) or due to an actual machine crash Network partitions and node failures are indistinguishable to the observer: a node may see that there is a problem with another node, but it cannot tell if it has crashed and will never be available again or if there is a network issue that might heal after some time Processes may also become unresponsive for other reasons, such as overload, CPU starvation, or long garbage collection pauses, leading to further confusion As such, even the most well-designed Reactive systems require additional tooling to quickly and decisively tackle large scale network issues The next section explores how networking problems, in particular network partitions, impact Reactive systems HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE The Problem “The network is homogeneous.” One Of The Fallacies Of Distributed Computing In Reactive systems, challenges arise when heterogeneous software components–such as collaborating groups of individual microservices–exchange important messages with each other Important messages must be delivered and processed, and any failure to deliver and process an important message will result in some form of inconsistent state When the network fails in a distributed system environment, this effectively causes a partition between the systems on each side of the network outage In most cases, the network has failed while all of the systems are still running The systems on each side of the network outage can no longer communicate across the partition It is as if an impenetrable wall has been placed between the systems on both sides of the network outage This is known as a split brain scenario Figure - Network Partition As shown in Figure 2, the network between the left and right nodes is broken The connections between the nodes on each side of the partition are cut To illustrate the impact of network partitions, let’s consider two examples In the first one, we’ll look at an order processing system - one that consists of just two microservices: order and customer The responsibility of the order microservice is to create new orders and the customer microservice is responsible for reserving customer credit HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE When users interact with this system and place an order, the order service creates a new order and sends an order created message to the customer service The customer service receives the order created message and reserves the credit It then sends a customer credit reserved message back to the order service The order service receives the message and changes the order state to approved Let’s now consider the impact of a network partition on this system To begin with, the order service sends the customer service an order created message The customer service then receives the message and reserves the credit as it should Order Customer Order Created Figure - Send Message Successfully The customer service then attempts to send a credit reserved message back to the order service But suddenly the the customer service falls off the network and the message cannot be sent Credit Reserved Order Customer Figure - Send Message Fails The order service never hears back from the customer service, so it resends the order created message again It receives no response, so it retries repeatedly to send the message HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE Order Customer Retry Order Created Figure - Message Send Retry Loop While the order service is caught in this retry loop, the network detects that the customer service is offline and efforts begin to bring it back online When that eventually occurs, the order service successfully sends the order created message and the customer service receives it But a naive implementation of the customer service would then reserve the credit again, which is the incorrect course of action Credit Reserved Order Order Created Customer Figure - Message Sent Again As demonstrated by this example, in the absence of a network partition handling strategy, businesses must make sure to incorporate a robust at-least-once delivery mechanism into the design Unfortunately, implementing such a mechanism is not trivial For example, the common retry loop approach is brittle and has a number of complexities that, if not handled properly, will result in the system converting to an inconsistent state In these circumstances and many others that are beyond the scope of this paper, the only viable option is to have an effective network partition / split brain resolution strategy HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE In the second illustrative example, we will use an in-person meeting with a group of seven coworkers To begin with, all seven members of the meeting are freely communicating back and forth Figure - People in Meeting Suddenly, a wall appears that splits the meeting in two, dividing the co-workers into one group of four people and another of three The wall is solid and soundproof, preventing any communication between the two groups No one on either side of the wall can ascertain what the other group is planning, making collaboration impossible Figure - Wall Splits the Group Let’s say that this is a very important meeting and it must continue, regardless of the presence of a giant impenetrable wall What should each group do? Both groups could sit idly and wait for the wall to disappear Or they could try determine a strategy that would get the entire group back together Most responses to this situation would result in some confusion and disruption to the meeting What if each group decided to continue the meeting on their own? That could result in decisions being made by these two split groups based on incomplete information, information that is only known to people on the other side of the wall In either case, a network partition is creating a number of issues, affecting the team’s velocity and ultimately its business agility HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 10 High-Level Solution Now, imagine that magic walls suddenly appearing in the middle of meetings was common, and even expected among the group To overcome this, the co-workers designed a plan of action in advance: when a wall appears in a meeting, the smaller group leaves their chairs and moves to the side where the larger group is located In our previous example, the wall created one group of four (Group A) and one group of three (Group B) Group B knows they are in the minority, so they move to the other side of the partition Group A knows they are in the majority, so they all stay seated and wait for Group B to arrive on their side of the wall Figure - Group B Joins Group A A key point here is that both groups had to independently arrive at the same conclusion The majority stays where they are and the minority group moves to rejoin the majority This works for an odd number of people, but what if there is an even split? Say there were eight in the meeting and the split was four and four In this situation, tie-breaking rules could help Perhaps the plan is to go to the side with the highest-ranking employee Returning to the first example with the order and customer microservices we see that a network partition that cuts off communication between them will interrupt the normal order processing workflow HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 11 Order Customer Figure 10 - Network Partition Between Services When that happens, just as with the meeting room example, each side needs to independently detect and resolve the issue In other words, the system must be capable of running on each side of the partition, detecting that there is a problem, and deciding which side stays up and which side shuts down The winning side should also be capable of restarting all the processes that were running on the losing side Figure 11 - Cluster Network Partition Now consider a more realistic example Say there are 20 microservices running on a cluster of five nodes (a node, in this case, could be a real server or VM), with four microservices running on each node (see Figure 11 above) The partition has cut off three of the nodes on one side and two nodes on the other side of the partition In order to detect and resolve a network partition in an environment like this, there are number of things that must occur: The cluster must be aware of all nodes within it A service discovery mechanism needs to know what is running and where it is running HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 12 Each node must be constantly checking to see if it can talk to the other nodes in the cluster When a network partition occurs, each of the monitoring components on each node needs to determine which nodes are still accessible and which ones are not In an environment where all of the above is in place, here’s what should happen when the network partition occurs: the node level monitoring detects that three nodes can still communicate with each other on one side of the partition and the other two nodes detect that they can only communicate with each other Figure 12 - Split Brain Recovery The two nodes on the minority side each shut down the microservices that are currently running on those nodes The three nodes on the majority side move the eight processes that were running on the minority side over and begin to run them, if there is sufficient capacity on the majority side to run them See Figure 12 above If there is insufficient capacity to host these additional microservices, it will be necessary to add one or more nodes to the majority side without administrator intervention, which in turn requires that the system possess the ability to scale automatically In short, based on the environment, its capabilities, and any constraints it is subject to, there can be multiple strategies to resolve issues caused by network partitions The next section explores four such strategies HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 13 Four Strategies To Resolve Network Partitions In order to detect and resolve a split brain condition, a system needs to have a reasonable degree of self-awareness That is, it needs to know the composition of the runtime environment and what processes are running in that environment Using a microservice system as an example, this system is composed of multiple servers that are running a collection of microservices and other associated application components, such as front-end containers This self-awareness also needs to include some form of monitoring of the runtime environment that is capable of detecting network partitions This process has to occur independently on the nodes on each side of the partition Within a reasonably functional self-aware system, all of the necessary capabilities to detect and resolve a network partition or split brain condition are present For the distributed Reactive applications it manages, Lightbend Enterprise Suite, the commercial component of Lightbend Reactive Platform, includes a powerful and easy-to-use Split Brain Resolver (SBR) feature Users can take advantage of this feature in two different ways - one that is more suited for development, and one that is more suited for production/operations Each implementation is discussed in more detail later Once a split brain condition is detected, there are a number of recovery options that can be used to resolve the problem The SBR feature includes four effective resolution strategies from which users can select: Keep Majority Static Quorum Keep Oldest Keep Referee SBR offers four strategies because there is no “one-size-fits-all” solution to the split brain condition Every strategy has a network partition-triggered failure scenario where it makes the “right” decision, and there is a failure scenario where it makes the “wrong” decision Users must select the strategy that is best suited for the characteristics of the specific services, nodes, and clusters running in their system Regardless of the strategy chosen, SBR allows users to recover from split brain scenarios in a matter of seconds Even for a large 1000-node cluster, SBR allows for recovery in about 65 seconds, based on Lightbend’s test data (see Table 1) HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 14 Cluster size (nodes) Failure Detection (seconds) >> Stable After (seconds) >> Removal Margin (seconds) >> Total Time To Resolution (seconds) 10 20 50 100 1000 5 5 5 10 13 17 20 30 10 13 17 20 30 19 25 31 39 45 65 Table - Default configuration for total failover latency in clusters Strategy - Keep Majority The Keep Majority strategy was discussed in the previous examples as one approach to resolve split brain conditions This approach is simple and fairly intuitive When a network partition causes a split brain condition, the nodes on the majority side of the partition continue to run and the nodes on the minority side shut down In order to function properly, the decision-making process must happen independently on each side of the partition In the event of an even split, a tie-breaking rule is used to decide which side stays up and which side shuts down To break ties in Keep Majority, SBR keeps up the side with the lowest IP address When to use Keep Majority: This strategy is appropriate for systems in which the number of nodes changes dynamically and often A cluster of nodes that employs some form of auto-scaling is one example At any point in time the number of nodes may change, scaling up as traffic increases and scaling down as traffic declines When to avoid Keep Majority: This strategy is less effective in scenarios where more than half the nodes on a given cluster go down If Keep Majority is used here, the remaining nodes will shut down because they are in the minority Also, if there are more than two partitions, all of the nodes will shut down because none have a majority Example Use Case: A retail or eCommerce site that scales up during peak traffic Strategy - Static Quorum The Static Quorum strategy is similar to the Keep Majority strategy The difference is a configuration setting that specifies the minimum number of nodes needed for a majority: a quorum size HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 15 When a network partition occurs, the side with a node count of at least the configured quorum size continues running For example, in a cluster of nine nodes with a quorum size of five, in a five-to-four split on each side of the partition, the five-node side stays up When to use Static Quorum: This strategy is based on simple addition and is therefore easy to understand When problems occur, it is clear how the decision was made on which nodes continued running and which nodes shut down When to avoid Static Quorum: Unlike Keep Majority, this strategy is not as flexible One limitation is that the cluster must not exceed quorum size * - If there are successive failures and new nodes are not added to replace failed downed nodes, this will result in the remaining nodes not meeting the quorum requirement For example, consider a cluster of nine nodes with a quorum size of five First, there is a six to three split, which results in a six node cluster staying up A second split occurs within the six node cluster, with four and two nodes on each side of the split In this case, neither side has a quorum of five, resulting in all clusters being shut down Example Use Case: Systems used in Financial Services that must meet consistency requirements Strategy - Keep Oldest Keep Oldest is used when cluster singletons are located on the oldest node in the cluster When a network partition occurs, the nodes on the side that includes the oldest node continue running The cluster singleton approach is used when there can only be one instance of a given application or resource running in a cluster at any time A cluster singleton may be an application or resource that has licensing restrictions or is expensive to start up When to use Keep Oldest: This strategy is most useful in systems where it is vital to keep cluster singletons running with minimal disruption When to avoid Keep Oldest: This strategy could result in a significant reduction of the number of available nodes when network partitions occur For example, in a nine node cluster, a sevento-two split with the oldest node on the minority side would result in two nodes staying up and seven nodes shutting down Example Use Case: A single expensive application or resource that needs to be kept running as often as possible HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 16 Strategy - Keep Referee Keep Referee is a good strategy when shutting down a particular node would cause the entire system to cease functioning properly A node that hosts strictly licensed software or a node that has specific firewall restrictions are potential examples When to use Keep Referee: This strategy is best for systems hosting a special node that must always be running When to avoid Keep Referee: In Keep Referee, the referee node is a single point of failure If this node goes down, the entire cluster will shut down with it Example Use Case: Networks forced to have a single special node due to licensing, firewalls, or other related constraints Split Brain Resolution From Lightbend Lightbend customers that have access to Lightbend Enterprise Suite as part of their subscription can take advantage of the SBR feature in two different ways, based on how they’re using Lightbend Enterprise Suite in their environment The first solution is oriented towards development Lightbend Enterprise Suite includes an implementation of SBR for the Akka actor model toolkit Users can use this to add SBR resolution functionality to their applications during development The second implementation of SBR is available as part of Lightbend Enterprise Suite’s cluster management system and is oriented towards production/operations use Users, including both developers and architects, must understand these implementations in more detail in order to decide which one to use Akka SBR The Akka toolkit allows for building Akka actor-based Reactive applications that run in a cluster of nodes These nodes are Java Virtual Machines (JVMs) that may run on a single host, such as a single VM or physical server or on a cluster of hosts An Akka cluster is a distributed system and is therefore vulnerable to network partitions When an Akka cluster experiences a network partition or split brain, SBR goes to work to resolve the issue often within seconds As described in the previous sections, SBR is configured to use one of four strategies: Keep Majority, Static Quorum, Keep Oldest, or Keep Referee If the current cluster conditions are right, the Akka cluster nodes on the winning side of the partition will continue to run, while nodes on the losing side shut down HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 17 Each node in an Akka cluster runs actors When nodes leave the cluster, as when one or more nodes are shut down by SBR, this results in all of the actors located on the departing nodes being terminated Note that the way an actor system responds to the departure of cluster nodes is application-dependent and is independent of the SBR feature Persistence and sharding, both of which are parts of the Akka toolkit, determine that behavior These solutions are implemented to work in a clustered environment In the event of a node leaving the cluster, the actors that went down with the departing nodes are recreated on other nodes in the cluster Both the persistence and sharding solutions are designed to automatically self-heal Custom actor systems that are implemented to be distributed across clusters also need to react appropriately when nodes leave the cluster The Akka toolkit includes many features that may be used to build highly-resilient systems As an example, particular actors can be created that will be notified when nodes leave and join the cluster These notifications can then be used to react appropriately when the topology of a cluster changes Users are therefore advised to use one of these solutions in conjunction with SBR For more details about persistence, sharding, and other aspects of the Akka toolkit, please refer to Akka documentation Cluster Management SBR Akka Cluster While Akka clusters provide a platform for running and managing actor systems, Lightbend Enterprise Suite’s cluster-management functionality delivers a clustered runtime environment for running and managing application systems (a collection of services.) Actor System Actor Figure 13 - Akka Cluster Nodes and Actors HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 18 As previously discussed, an Akka actor system is composed of a cluster of JVMs Actors reside within each of the cluster nodes (see Figure 13) Again, as previously discussed, Akka SBR detects and resolves network partitions by identifying which nodes are on the winning side and the losing side of a partition Various strategies are used to restore actors that were previously resident on the losing side, as shown in Figure 14 where the rectangles represent cluster node JVMs, and the circles represent actors Figure 14 - Cluster Nodes and Actors with Network Partition In contrast, a Lightbend application cluster is composed of Linux nodes Each node hosts running containers In this environment, SBR also detects and resolves network partitions by identifying the nodes on the winning and losing sides of a partition The running containers on the losing side are restarted on the winning side See Figure 15 where the rectangles represent Linux nodes, and the circles represent running containers Note that Figures 14 and 15 are conceptually similar The point is that logically the same SBR processes are used for both Akka and cluster management The difference is the level of granularity Akka is more fine-grained with actors running in JVMs, while cluster management is more coarse-grained with containers running on Linux nodes SBR is built into Lightbend Enterprise Suite’s cluster management feature When a Lightbend application cluster experiences a network partition, the resolution process goes to work using the Keep Majority strategy The nodes on each side of the partition determine which one is the winning side and which one is the losing side The losing side shuts down its services, while the winning side attempts to restart the services that were running on the losing side The primary limitation is capacity: does the winning side have sufficient capacity to run all of the services that were running on the losing side? HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 19 Figure 15- Service Orchestration Cluster and Containers After the losing side’s cluster nodes have stopped all of the running services, they then enter a recovery mode While in this recovery mode, the nodes constantly attempt to reconnect with the nodes on the winning side When the network starts working again, the losing side nodes rejoin the cluster of nodes on the winning side Next, let’s discuss the impact on organizations that use Lightbend’s Enterprise Suite to resolve network partitions using the SBR feature HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 20 The Benefits Serve Customers Better Network partitions are guaranteed to happen, and there is no one solution to resolve this issue In today’s highly competitive, fast-moving environment, users that experience frequent outages, extended downtime, or recurring performance degradations are susceptible to defections The Split Brain Resolver from Lightbend minimizes the impact of network partitions on users and helps businesses serve them better, improving retention and helping attract new users Eliminate Expensive Downtime By automatically applying pre-configured resolution strategies in just seconds, recovering failed nodes no longer requires time-consuming manual intervention by operations staff, freeing them to focus on other operational matters Immediate Time-To-Value As an out-of-the-box feature, SBR delivers immediate value Further, it is built using Lightbend’s deep Akka expertise, so enterprises can be assured of utilizing best-in-class network partition resolution strategies Summary In the era of highly-distributed, real-time applications, network issues often result in so-called “split brain” scenarios, where one or more nodes in a cluster becomes suddenly unresponsive This causes data inconsistencies between distributed services that can lead to cascading failures and downtime Lightbend offers a Split Brain Resolver feature that can help resolve network partitions in just seconds This feature is available to Lightbend customers as part of Lightbend Enterprise Suite and can be taken advantage of either in development at the code level or at the production/operations level Lightbend customers that use the SBR feature can avoid expensive downtime and serve their customers better HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 21 Contact us to discuss the best way to add Split Brain Resolver - and other Lightbend Enterprise Suite features to your Akka, Lagom and Play-based Reactive applications CONTACT US Lightbend (Twitter: @Lightbend) provides the leading Reactive application development platform for building distributed systems Based on a message-driven runtime, these distributed systems, which include microservices and fast data applications, can scale effortlessly on multi-core and cloud computing architectures Many of the most admired brands around the globe are transforming their businesses with our platform, engaging billions of users every day through software that is changing the world Lightbend, Inc 625 Market Street, 10th Floor, San Francisco, CA 94105 | www.lightbend.com ... organizations that use Lightbend s Enterprise Suite to resolve network partitions using the SBR feature HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 20 The Benefits Serve Customers... HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 13 Four Strategies To Resolve Network Partitions In order to detect and resolve a split brain condition, a system... better HOW TO RESOLVE NETWORK PARTITIONS IN SECONDS WITH LIGHTBEND ENTERPRISE SUITE 21 Contact us to discuss the best way to add Split Brain Resolver - and other Lightbend Enterprise Suite features