Cloudflare's Meerkat and the Use of QuePaxa for Global Consensus
Cloudflare has introduced Meerkat, a distributed consensus service designed to manage its global control plane, using a new algorithm called QuePaxa to address challenges in wide-area networks.
Cloudflare recently introduced Meerkat, a distributed consensus service designed to manage its internal global control plane. Meerkat is powered by a new consensus algorithm called QuePaxa, published by Tennage & Băsescu et al. in 2023. Unlike traditional consensus algorithms such as Raft, Multi-Paxos, or Viewstamped Replication, which rely on a single leader process, QuePaxa is designed to function effectively in wide-area networks with high and variable latency, such as the Internet. This is crucial for Meerkat, which operates globally and faces challenges with network volatility and unpredictable latency (Cloudflare Blog). [1]
Traditional consensus algorithms often experience performance degradation when network conditions worsen. For example, in 2020, a partial failure in a core Cloudflare network switch caused a consensus-powered system to slow down by 80x for six hours. While the system did not lose data, the slowdown was unacceptable for critical operations. This highlights the need for a consensus algorithm that can make progress despite network issues, which is one of the key design goals of QuePaxa (Cloudflare Blog). [1]
QuePaxa addresses the limitations of traditional consensus algorithms by eliminating the need for a single leader and reducing reliance on fault detection. Instead, it allows proposers to cooperate in reaching consensus through random priority assignments and proposal adoption. This enables the system to continue functioning even when the leader fails or becomes slow. Unlike leader-based systems, which require complex recovery processes, QuePaxa avoids such dependencies, ensuring that the system remains available even during network volatility (Cloudflare Blog). [1]
Meerkat is a distributed consensus system designed for global scale, with clusters potentially spanning continents and connected by networks characterized by high and variable latency. This setup is unusual for consensus systems, which are traditionally deployed in smaller, more local networks with stricter latency requirements. The system is built to ensure that replicas agree on an ordered log of events, maintaining consistency even in the face of network volatility and server failures. Unlike traditional consensus systems, Meerkat does not rely on a single leader to manage the consensus process, allowing it to maintain performance and availability in wide-area networks. This global deployment environment is a key factor in Meerkat’s decision to adopt QuePaxa as its underlying consensus algorithm. [1]
Advertisement
QuePaxa, the consensus algorithm powering Meerkat, was developed by Tennage & Băsescu et al. in 2023 and is designed to function in asynchronous environments, where network latencies and connectivity issues are unpredictable. Unlike traditional consensus algorithms like Raft, Paxos, or Multi-Paxos, which depend on a single leader and predictable network behavior, QuePaxa avoids relying on a leader to make progress. This makes it particularly suitable for systems like Meerkat, which must operate in environments with frequent network disruptions and varying latencies. The algorithm is structured to handle these conditions by allowing proposers to cooperatively work towards a consensus, reducing the risk of system stalls caused by leader failures or network issues. [1]
In 2020, a partial failure of a core Cloudflare network switch caused a consensus-powered system to degrade significantly, with the dashboard operating 80x slower than usual for six hours. This incident highlighted the vulnerability of traditional consensus systems to network disruptions, even when no data was lost. The system remained unavailable for writes until the issue was resolved, underscoring the limitations of existing algorithms in global-scale deployments. This experience was a critical factor in Cloudflare’s decision to adopt QuePaxa, which is designed to maintain progress despite network volatility and avoid the pitfalls of leader-based systems. [1]
QuePaxa addresses the challenges of destructive interference and fault detection by eliminating the need for a single leader and reducing reliance on precise timing assumptions. Instead of having proposers compete for consensus, QuePaxa allows them to cooperatively converge on a proposal, with priorities assigned randomly to ensure fairness and avoid blocking. This approach reduces the risk of system stalls caused by leader failures or network delays. Additionally, QuePaxa uses a hedging timeout mechanism, which allows clients to send requests to multiple replicas simultaneously, ensuring that the system can continue making progress even if the leader is unavailable or slow. This mechanism avoids the pitfalls of traditional fault detection, which can lead to unnecessary downtime and system stalls. [1]
Meerkat’s implementation of QuePaxa is structured around four phases, which allow the system to handle consensus decisions in a round-based manner. The first phase, Propose and Adopt, involves proposers randomly selecting priorities and broadcasting their proposals to recorders. If a majority of recorders support a proposal, the proposer can decide on it immediately. Subsequent phases ensure that the system safely converges on a consensus decision, even in the presence of network delays or proposer failures. This multi-phase approach allows Meerkat to maintain performance and availability in global-scale deployments, where traditional consensus systems often struggle with network volatility and leader failures. [1]
Advertisement
The Cali Reporter newsletter
The stories worth your time.
California, the country, and what comes next. Join our free newsletter for reporting that keeps you informed.
More in Technology
- Google Play's Closed Testing Requirements and the Importance of Feedback
Google Play's closed testing rule emphasizes the need for developers to gather meaningful feedback from testers, not just track app installs. This approach ensures that real user experiences are evaluated and addressed…
- World Diabetes Day 2026 Focuses on Early Diagnosis and Awareness
The 2026 World Diabetes Day campaign emphasizes the importance of early diagnosis, equitable access to care, and reducing diabetes stigma, according to the WHO. The theme 'Early action on diabetes' highlights the need…
- Understanding Memory Usage on Linux VPS
This article explains how to accurately interpret memory usage on a Linux VPS, focusing on the importance of the 'available' metric and the potential pitfalls of relying on the 'free' column.