April 24, 2024

 

  • We’re rolling out MySQL Raft with the goal to finally exchange our present MySQL semisynchronous databases. 
  • The most important win of MySQL Raft was simplification of the operation and making MySQL servers deal with promotions and membership. This gave the provable security of Raft and diminished vital operational ache.
  • Making MySQL server a real distributed system additionally has opened up prospects in downstream methods to leverage it. A few of these concepts are beginning to take form.

At Meta, we run one of many largest deployments of MySQL on this planet. The deployment powers the social graph together with many different providers, like Messaging, Advertisements, and Feed. Over the previous couple of years, we’ve got applied MySQL Raft, a Raft consensus engine that was built-in with MySQL to construct a replicated state machine. We’ve got migrated a big portion of our deployment to MySQL Raft and plan to totally exchange the present MySQL semisynchronous databases with it. The mission has delivered vital advantages to the MySQL deployment at Meta, together with larger reliability, provable security, vital enhancements in failover time, and operational simplicity — all with equal or comparable write efficiency.

Background

To permit for prime availability, fault tolerance, and scaling reads, Meta’s MySQL datastore is a massively sharded, geo-replicated deployment with tens of millions of shards, holding petabytes of information. The deployment contains hundreds of machines working over a number of areas and knowledge facilities throughout a number of continents.

Beforehand, our replication resolution used the MySQL semisynchronous (semisync) replication protocol. This was an information path–solely protocol. The MySQL major would use semisynchronous replication to 2 log-only replicas (logtailers) inside the major area however exterior of the first’s failure area. These two logtailers would act as semisynchronous ACKer (An ACK is an acknowledgment to the first that the transaction has been regionally written).  This is able to enable the information path to have very low latency (sub-millisecond) commits and would supply excessive availability/sturdiness for the writes. Common MySQL primary-to-replica asynchronous replication was used for wider distribution to different areas.

The management aircraft operations (e.g., promotions, failover, and membership change) could be the duty of a set of Python daemons (henceforth known as automation). Automation would do the required orchestration to advertise a brand new MySQL server in a failover location as a major. The automation would additionally level the earlier major and the remaining replicas to duplicate from the brand new major. Membership change operations could be orchestrated by one other piece of automation known as MySQL pool scanner (MPS). So as to add a brand new member, MPS would level the brand new reproduction to the first and add it to the service discovery retailer. A failover could be a extra advanced operation wherein the tailing threads of the logtailers (semisynchronous ACKers) could be shut all the way down to fence the earlier useless major.

Why was MySQL Raft obligatory?

Up to now, to assist assure security and keep away from knowledge loss in the course of the advanced promotion and failover operations, a number of automation daemons and scripts would use locking, orchestration steps, a fencing mechanism, and SMC,  a service discovery system. It was a distributed setup, and it was tough to perform this atomically. The automation grew to become extra advanced and tougher to take care of over time as increasingly more nook circumstances wanted to be patched.

We determined to take a very totally different strategy. We enhanced MySQL and made it a real distributed system. Realizing that management aircraft operations like promotions and membership adjustments had been the set off of most points, we wished the management aircraft and knowledge aircraft operations to be a part of the identical replicated log. For this, we used the well-understood consensus protocol Raft. This additionally meant that the supply of fact of membership and management moved contained in the server (mysqld). This was the only greatest contribution of bringing in Raft as a result of it enabled provable correctness (security property) throughout promotions and membership adjustments into the MySQL server.

The Raft library and the MySQL Raft plugin

Our implementation of Raft for MySQL relies on Apache Kudu. We enhanced it considerably for the wants of MySQL and our deployment. We printed this fork as an open supply mission, kuduraft.

Among the key options that we added to kuduraft are:

  • FlexiRaft — help for 2 totally different intersecting quorums: the information quorum and the chief election quorum
  • Proxying — the flexibility to make use of a proxy intermediate node to scale back community bandwidth
  • Compression — the place we compress binary log (transaction) payloads as soon as earlier than distribution
  • Log abstraction — to help totally different bodily logfile implementations
  • Major ban — the flexibility to forestall some entities from being major quickly

We additionally needed to make comparatively large adjustments to MySQL replication to interface with Raft. For this, we created a brand new closed supply MySQL plugin known as MyRaft. MySQL would interface with MyRaft by way of the plugin APIs (comparable APIs had been used for semisync as effectively), whereas we created a separate API for MyRaft to interface again with MySQL server (callbacks).

MySQL Raft replication topologies

A Raft ring would include a number of MySQL cases (4 in the diagram) in numerous areas. The communication round-trip time (RTT) between these areas would vary from 10 to 100 milliseconds. A number of of those MySQLs (sometimes three) had been allowed to develop into primaries, whereas the remainder of them had been solely allowed to be pure learn replicas (non-primary-capable). The MySQL deployment at Meta additionally has a long-standing requirement for terribly low latency commits. The providers that use MySQL as a retailer (e.g., the social graph) want or have been designed to such extraordinarily quick writes.

To fulfill this requirement, the configuration of FlexiRaft would use solely in-region commits (single area dynamic mode). To allow this, every major succesful area would have two extra logtailers (witnesses or log-only entities). The information quorum for writes could be 2/3 (2 ACKs out of the 1 MySQL + 2 logtailers). Raft would nonetheless handle and run a replicated log throughout all of the entities (1 primary-capable MySQL + 2 logtailers ) * 3 areas + (non-primary-capable MySQL) * 3 areas = 12 entities.

Raft roles: The chief, because the identify suggests, is the chief in a time period of the replicated log. A frontrunner in Raft would even be the first in MySQL and the one accepting shopper writes. The follower is a voting member of the ring and passively receives messages (AppendEntries) from the chief. A follower could be a reproduction in MySQL’s standpoint and could be making use of the transactions to its engine. It might not enable direct writes from consumer connections (read_only=1 is ready). A learner could be a non-voting member of the ring, e.g., the three MySQLs in non-primary-capable areas (above). It might be a reproduction in MySQL’s standpoint.

Replicated log

For replication, MySQL has traditionally used the binary log format. This format is central to MySQL’s replication, and we determined to protect this. From the Raft perspective, the binary log grew to become the replicated log. This was finished through the log abstraction enchancment to kuduraft. The MySQL transactions could be encoded as a collection of occasions (e.g., Replace Rows occasion) with a begin and finish for every transaction. The binary log would even have acceptable headers and would sometimes finish with an ending occasion (Rotate occasion). 

We needed to tweak how MySQL manages its logs internally. On a major, Raft would write to a binlog. That is no totally different from what occurs in customary MySQL. In a reproduction, Raft would additionally write to a binlog as a substitute of to a separate relay log in customary MySQL. This created simplicity for Raft as there was just one namespace of log recordsdata that Raft could be involved about. If a follower had been promoted to chief, it might seamlessly return into its historical past of logs to ship transactions to lagging members. The reproduction’s applier threads would choose up transactions from the binlog after which apply them to the engine. Throughout this course of, a brand new log file, the apply log, could be created. This apply log would play an necessary position in crash restoration of replicas however is in any other case a nonreplicated log file.

So, in abstract:

In customary MySQL:

  • Major writes to binlog and sends binlog to replicas.
  • Replicas obtain in relay log and apply the transactions to the engine. Throughout apply, a brand new replica-only binlog is created.

In MySQL Raft:

  • Major writes to binlog through Raft, and Raft sends binlog to followers/replicas.
  • Replicas/followers obtain in binlog and apply the transactions to the engine. An apply log is created throughout apply.
  • Binlog is the replicated log from the Raft standpoint.

Write transaction on MySQL major utilizing Raft

The transaction would first be ready within the engine. This is able to occur within the thread of the consumer connection. The act of getting ready the transaction would contain interactions with the storage engine (e.g., InnoDB or MyRocks) and generate an in-memory binlog payload for the transaction. On the time of commit, the write would cross by way of group commit/ordered_commit stream. GTIDs could be assigned, after which Raft would assign an OpId (time period:index) to the transaction. At this level, Raft would compress the transaction, retailer it in its LogCache, and write by way of the transaction to a binlog file. It might asynchronously begin delivery the transaction to different followers to get ACKs and attain consensus.

The consumer thread, which is in “commit” of the transaction, could be blocked, ready for consensus from Raft. When Raft would get two out of three in-region votes, consensus commit could be reached. Raft would additionally ship the transaction to all out-of-region members however would ignore their votes due to an algorithm known as FlexiRaft (described under). On consensus commit, the consumer thread could be unblocked, and the transaction would proceed and decide to the engine. After engine commit, the write question would end and return to the shopper. Quickly after, Raft would additionally asynchronously ship a commit marker (OpId of present commit) to downstream followers in order that they’ll additionally apply the transactions to their database. 

Crash restoration

Adjustments needed to be made to crash restoration to make it work seamlessly with Raft. Crashes can occur at any time within the lifetime of a transaction and therefore the protocol has to make sure consistency of members. Listed below are some key insights on how we made it work.

  1. Transaction was not flushed to binlog: On this case, the in-memory transaction payload (nonetheless in mysqld course of reminiscence as an in-memory buffer) could be misplaced and the ready transaction in engine could be rolled again on course of restart. Since there was no additional uncommitted transaction within the Raft log, no reconciliation with different members must be finished.
  2. Transaction was flushed to binlog however by no means reached different members: Mysqld acts as a transaction coordinator and runs a two-phase commit protocol between the engine and the replicated binlog because the individuals. On crash restoration, the ready transaction in engine (e.g., InnoDB or MyRocks) could be rolled again (engine had not reached commit). Raft would undergo failover, and a brand new chief could be elected. This chief wouldn’t have this transaction in its binlog and henceforth would truncate this transaction from the erstwhile chief’s binlog due to to the next time period (by pushing a No-Op message), when the erstwhile chief joins again the ring.
  3. Transaction was flushed to binlog and reached to subsequent chief. Present chief died earlier than committing to the engine: Just like no. 2 above, the ready transaction within the engine could be rolled again. The erstwhile chief would be part of the Raft ring as a follower. On this case, the brand new chief would have this transaction in its binlog and therefore no truncation would occur, for the reason that logs would match. When the commit marker is shipped by the brand new chief, the transaction could be reapplied once more from scratch.

Raft-initiated state machine transitions

Failover and common upkeep operations can set off management adjustments in Raft. After a pacesetter is elected, the MyRaft plugin would attempt to to transition the accompanying MySQL into major mode. For this, the plugin would orchestrate a set of steps. These callbacks from Raft → MySQL would abort in-flight transactions, roll again in-use GTIDs, transition the engine aspect log from apply-log to binlog, and finally set the correct read_only settings. This mechanism is advanced and presently not open sourced.

FlexiRaft

Because the Raft paper and Apache Kudu supported solely a single international quorum, it will not work effectively at Meta, the place rings had been giant however the knowledge path quorum wanted to be small.

To avoid this challenge, we innovated on FlexiRaft, borrowing concepts from Flexible Paxos.

At a excessive stage, FlexiRaft permits Raft to have a unique knowledge commit quorum (small) however take a corresponding hit on the chief election quorum (giant). By following provable ensures of quorum intersection, FlexiRaft ensures that the longest log guidelines of Raft and the suitable quorum intersection will assure provable security.

FlexiRaft helps single area dynamic mode. On this mode, members are grouped collectively by their geo-region. The present quorum of Raft is dependent upon who the present chief is (therefore the identify “single area dynamic”). The information quorum is almost all of voters within the chief’s area. Throughout promotions, if phrases are steady, the Candidate will intersect with the final recognized chief’s area. FlexiRaft would additionally be sure that the quorum of the Candidate’s area can be attained, in any other case the following No-Op message might get caught. If within the uncommon case the phrases aren’t steady, Flexi Raft would strive to determine a rising set of areas which have to be intersected with for security or, within the worst case, would fall again to the N area intersection case of Versatile Paxos. Because of pre-elections and mock elections, the incidences of time period gaps are uncommon.

Management aircraft operations (promotions and membership adjustments)

With a view to serialize promotion and membership change occasions within the binlog, we hijacked the Rotate Occasion and Metadata occasion of the MySQL binary log format. These occasions would carry the equal of No-Op messages and add-member/remove-member operations of Raft. Apache Kudu didn’t have help for joint consensus, therefore we solely enable one-at-a-time membership adjustments (you possibly can change the membership by just one entity in a single spherical to observe the foundations of implicit quorum intersection).

Automation

With the implementation of MySQL Raft, we reached a really clear separation of considerations for the MySQL deployment. The MySQL server could be chargeable for security through Raft’s replicated state machine. The no-data-loss assure could be provably enshrined within the server itself. Automation (Python scripts, daemons) would provoke management aircraft operations and monitor the well being of the fleet. It might additionally exchange members or do promotions through Raft throughout upkeep or when a bunch failure was detected. From time to time, automation might additionally change the regional placement of MySQL topology. Altering the automation to adapt to Raft was an enormous enterprise, spanning a number of years of growth and rollout effort.

Throughout extended upkeep occasions, automation would set management ban data on Raft. Raft would disallow these banned entities from turning into chief or evacuate them promptly on inadvertent election. The automation would additionally promote away from these areas into different areas.

Studying from rollouts and challenges encountered alongside the way in which

Rolling out Raft to the fleet was an enormous studying for the crew. We initially developed Raft on MySQL 5.6 and needed to migrate to MySQL 8.0.

One of many key learnings was that whereas correctness was simpler to cause with Raft, the Raft protocol in itself doesn’t assist a lot within the concern of availability. Since our MySQL knowledge quorum was very small (two out of three in-region members), two unhealthy entities within the area might just about shatter the quorum and produce down availability. The MySQL fleet undergoes a superb quantity of churn every single day (as a result of upkeep, host failures, rebalancing operations), so initiating and doing membership adjustments promptly and accurately had been a key necessities for fixed availability. A big a part of the rollout effort was centered at doing logtailer and MySQL replacements promptly in order that the Raft quorums had been wholesome.

We needed to improve kuduraft to make it extra sturdy for availability. These enhancements weren’t a part of the core protocol however may be thought of as engineering add-ons to it. Kuduraft has the help for pre-elections, however pre-elections are finished solely throughout a failover. Throughout a swish switch of management, the designated Candidate strikes on to an actual election, bumping the time period. This results in caught leaders (kuduraft doesn’t do auto step-down). To deal with this downside, we added a mock elections function, which was much like pre-elections however occurred solely upon a swish switch of management. Since this was an async operation, it didn’t enhance promotion downtimes. A mock election would weed out circumstances the place an actual election would partially succeed and get caught.

Dealing with byzantine failures: Raft’s membership listing is taken into account to be blessed by Raft itself. However in the course of the provisioning of latest members, or due to races in automation, there could possibly be weird circumstances of two totally different Raft rings intersecting. These zombie membership nodes needed to be weeded out and shouldn’t be in a position to talk with one another. We applied a function to dam RPCs from such zombie members to the ring. This was, in some methods, a dealing with of a byzantine actor. We enhanced the Raft implementation after noticing these uncommon incidents that occurred in our deployment.  

Monitoring the MySQL Raft rollout

Whereas launching MySQL Raft, one of many targets was to scale back operational complexity for on-calls, in order that engineers might root-cause and mitigate points. We constructed a number of dashboards, CLI instruments, and scuba tables to watch Raft. We added copious logging to MySQL, particularly across the space of promotions and membership adjustments. We created CLIs for quorum and voting reviews on a hoop, which assist us shortly establish when and why a hoop is unavailable (shattered quorum). The funding within the tooling and automation infrastructure went hand-in-hand and may need been an even bigger funding than the server adjustments. This funding paid off big-time and diminished operational and onboarding ache.

Quorum Fixer

Though it’s undesirable, quorums do get shattered every so often, resulting in availability loss. The everyday case is when automation doesn’t detect unhealthy cases/logtailers within the ring and doesn’t exchange them shortly. This could occur due to poor detection, employee queue overload, or a scarcity of spare host capability. Correlated failures, when a number of entities within the quorum go down on the identical time, are much less typical. These don’t occur usually, as a result of the deployments attempt to isolate failure domains throughout crucial entities of the quorum by way of correct placement selections. Lengthy story quick: At scale, sudden issues occur, regardless of present safeguards. Instruments have to be accessible to mitigate such conditions in manufacturing. We constructed Quorum Fixer in anticipation of this.

Quorum Fixer is a handbook remediation device authored in Python that squelches the writes on the ring. It does out-of-band checks to determine the longest log entity. It forcibly adjustments the quorum expectations for a pacesetter election inside Raft, in order that the chosen entity turns into a pacesetter. After profitable promotion, we reset the quorum expectation again, and the ring sometimes turns into wholesome.

It was a acutely aware choice to not run this device robotically, as a result of we wish to root trigger and establish all circumstances of quorum loss and repair bugs alongside the way in which (not have them silently be fastened by automation).

Rolling out MySQL Raft

Transitioning from semisynchronous to MySQL Raft over an enormous deployment is tough. For this we created a device (in Python) known as enable-raft. Allow-raft orchestrates the transition from semisynchronous to Raft by loading the plugin and setting the suitable configs (mysql sys-vars) on every of the entities. This course of entails a small downtime for the ring. The device was made sturdy over time and might roll out Raft at scale in a short time. We’ve got used it to soundly roll out Raft.

Testing and shadow workflow

For sure, doing a change within the core replication pipeline of MySQL is a really tough mission. Since knowledge security is at stake, testing was key for confidence. We leveraged shadow testing and failure injection considerably in the course of the mission. We’d inject hundreds of failovers and elections on check rings earlier than each RPM package deal supervisor rollout. We’d set off replacements and membership adjustments on the check belongings to set off the crucial code paths.

Lengthy-running testing with knowledge correctness checks had been additionally key. We’ve got automation that runs nightly on the shards, making certain consistency of primaries and replicas. We’re alerted to any such mismatch, and we debug it.

Efficiency

The efficiency of the write path latency for Raft was equal to semisync. The semisync equipment is barely less complicated and therefore anticipated to be leaner, nonetheless we optimized Raft to get the identical latencies as semi-sync. We optimized kuduraft to not add any extra CPU to the fleet despite pulling in lots of extra tasks that beforehand had been exterior the server binary.

Raft made order-of-magnitude enhancements to promotions and failover occasions. Swish promotions, that are the majority of management adjustments within the fleet, improved considerably, and we are able to sometimes end a promotion in 300 milliseconds. Within the semisync setups, for the reason that service discovery retailer could be the supply of fact, the purchasers noticing the end of promotion could be for much longer, resulting in extra elevated finish consumer downtimes on a shard.

Raft sometimes does a failover inside 2 seconds. It’s because we heartbeat for Raft well being each 500 milliseconds and begin an election when three successive heartbeats fail. Within the semisync world, this step was orchestration heavy and would take 20 to 40 seconds. Raft thereby gave a 10x enchancment in downtimes for failover circumstances. 

Subsequent steps

Raft has helped remedy issues with the operational administration of MySQL at Meta by offering provable security and ease. Our targets of getting a fingers off-management of MySQL consistency, and having instruments for the uncommon circumstances of availability loss, are largely met. Raft now opens up vital alternatives sooner or later, as we are able to deal with enhancing the providing to the providers that use MySQL. One of many asks’ from our service house owners is to have configurable consistency. Configurable consistency will enable the house owners on the time of onboarding, to pick out whether or not the service wants X-region quorums or quorums that ask for copies in some particular geographies (e.g., Europe and the USA). FlexiRaft has seamless help for such configurable quorums, and we plan to begin rolling out this help sooner or later. Such quorums will correspondingly result in larger commit latencies, however use circumstances have to have the ability to trade-off between consistency and latency (e.g., PACELC theorem).

Due to the proxying function (potential to ship messages utilizing a multihop distribution topology), Raft can even save community bandwidth throughout the Atlantic. We plan to make use of Raft to duplicate from the USA to Europe solely as soon as, after which use Raft’s proxying function to distribute inside Europe. This may enhance latency, however will probably be nominal on condition that the majority of the latency is within the cross-Atlantic switch and the additional hop is way shorter.

Among the extra speculative concepts in Meta’s database deployments and distributed consensus house are about exploring leaderless protocols, like Epaxos. Our present deployments and providers have labored with the assumptions that include robust chief protocols, however we’re beginning to see a trickle of necessities the place providers would profit from extra uniform write latency within the WAN. One other concept that we’re contemplating is to disentangle the log from the state machine (the database) right into a disaggregated log setup. This may enable the crew to handle the considerations of the log and replication individually from the considerations of the database storage and SQL execution engine.

Acknowledgements

Constructing and deploying MySQL Raft at Meta scale wanted vital teamwork and administration help. We wish to acknowledge the next folks for his or her position in making this mission successful. Shrikanth Shankar, Tobias Asplund, Jim Carrig, Affan Dar and David Nagle for supporting the crew members throughout this journey. We’d additionally prefer to thank the in a position Program Managers of this mission Dan O and Karthik Chidambaram who stored us on observe.

The engineering effort concerned key contributions from a number of present and previous crew members together with Vinaykumar Bhat, Xi Wang, Bartlomiej Pelc, Chi Li, Yash Botadra, Alan Liang, Michael Percy, Yoshinori Matsunobu, Ritwik Yadav, Luqun Lou, Pushap Goyal, Anatoly Karp and Igor Pozgaj.