Learning on etcd

etcd client architecture

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

etcd server has proven its robustness with years of failure injection testing. Most complex application logic is already handled by etcd server and its data stores (e.g. cluster membership is transparent to clients, with Raft-layer forwarding proposals to leader). Although server components are correct, its composition with client requires a different set of intricate protocols to guarantee its correctness and high availability under faulty conditions. Ideally, etcd server provides one logical cluster view of many physical machines, and client implements automatic failover between replicas. This documents client architectural decisions and its implementation details.

Client feature matrix

Mon, 01 Jan 0001 00:00:00 +0000

Features

Feature	`clientv3-grpc1.14`	`jetcd v0.0.2`
Automatic retry	Yes	.
Retry backoff	Yes	.
Automatic failover	Yes	.
Load balancer	Round-Robin	·
`WithRequireLeader(context.Context)`	Yes	.
`TLS`	Yes	Yes
`SetEndpoints`	Yes	.
`Sync` endpoints	Yes	.
`AutoSyncInterval`	Yes	.
`KeepAlive` ping	Yes	.
`MaxCallSendMsgSize`	Yes	.
`MaxCallRecvMsgSize`	Yes	.
`RejectOldCluster`	Yes	.

KV

Feature	`clientv3-grpc1.14`	`jetcd v0.0.2`
`Put`	Yes	.
`Get`	Yes	.
`Delete`	Yes	.
`Compact`	Yes	.
`Do(Op)`	Yes	.
`Txn`	Yes	.

For details, see the KV API reference.

Data model

Mon, 01 Jan 0001 00:00:00 +0000

etcd is designed to reliably store infrequently updated data and provide reliable watch queries. etcd exposes previous versions of key-value pairs to support inexpensive snapshots and watch history events (“time travel queries”). A persistent, multi-version, concurrency-control data model is a good fit for these use cases.

etcd stores data in a multiversion persistent key-value store. The persistent key-value store preserves the previous version of a key-value pair when its value is superseded with new data. The key-value store is effectively immutable; its operations do not update the structure in-place, but instead always generate a new updated structure. All past versions of keys are still accessible and watchable after modification. To prevent the data store from growing indefinitely over time and from maintaining old versions, the store may be compacted to shed the oldest versions of superseded data.

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

The v3 protocol uses gRPC as its transport instead of a RESTful interface like v2. This new protocol provides an opportunity to iterate on and improve the v2 design. For example, v3 auth has connection based authentication, rather than v2’s slower per-request authentication. Additionally, v2 auth’s semantics tend to be unwieldy in practice with respect to reasoning about consistency, which will be described in the next sections. For v3, there is a well-defined description and implementation of the authentication mechanism which fixes the deficiencies in the v2 auth system.

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

The name “etcd” originated from two ideas, the unix “/etc” folder and “d"istributed systems. The “/etc” folder is a place to store configuration data for a single system whereas etcd stores configuration information for large scale distributed systems. Hence, a “d"istributed “/etc” is “etcd”.

etcd is designed as a general substrate for large scale distributed systems. These are systems that will never tolerate split-brain operation and are willing to sacrifice availability to achieve this end. etcd stores metadata in a consistent and fault-tolerant way. An etcd cluster is meant to provide key-value storage with best of class stability, reliability, scalability and performance.

etcd3 API

Mon, 01 Jan 0001 00:00:00 +0000

This document is meant to give an overview of the etcd3 API’s central design. It is by no means all encompassing, but intended to focus on the basic ideas needed to understand etcd without the distraction of less common API calls. All etcd3 API’s are defined in gRPC services, which categorize remote procedure calls (RPCs) understood by the etcd server. A full listing of all etcd RPCs are documented in markdown in the gRPC API listing.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

KV API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

etcd is a consistent and durable key value store with mini-transaction support. The key value store is exposed through the KV APIs. etcd tries to ensure the strongest consistency and durability guarantees for a distributed system. This specification enumerates the KV API guarantees made by etcd.

APIs to consider

Read APIs
- range
- watch
Write APIs
- put
- delete
Combination (read-modify-write) APIs
- txn

etcd specific definitions

Operation completed

An etcd operation is considered complete when it is committed through consensus, and therefore “executed” -- permanently stored -- by the etcd storage engine. The client knows an operation is completed when it receives a response from the etcd server. Note that the client may be uncertain about the status of an operation if it times out, or there is a network disruption between the client and the etcd member. etcd may also abort operations when there is a leader election. etcd does not send abort responses to clients’ outstanding requests in this event.

Learner

Mon, 01 Jan 0001 00:00:00 +0000

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

A newly joined etcd member starts with no data, thus demanding more updates from leader until it catches up with leader’s logs. Then leader’s network is more likely to be overloaded, blocking or dropping leader heartbeats to followers. In such case, a follower may election-timeout to start a new leader election. That is, a cluster with a new member is more vulnerable to leader election. Both leader election and the subsequent update propagation to the new member are prone to causing periods of cluster unavailability (see Figure 1 below).