etcd

etcd client architecture

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

etcd server has proven its robustness with years of failure injection testing. Most complex application logic is already handled by etcd server and its data stores (e.g. cluster membership is transparent to clients, with Raft-layer forwarding proposals to leader). Although server components are correct, its composition with client requires a different set of intricate protocols to guarantee its correctness and high availability under faulty conditions. Ideally, etcd server provides one logical cluster view of many physical machines, and client implements automatic failover between replicas. This documents client architectural decisions and its implementation details.

Install

Mon, 01 Jan 0001 00:00:00 +0000

System requirements

The etcd performance benchmarks run etcd on 8 vCPU, 16GB RAM, 50GB SSD GCE instances, but any relatively modern machine with low latency storage and a few gigabytes of memory should suffice for most use cases. Applications with large v2 data stores will require more memory than a large v3 data store since data is kept in anonymous memory instead of memory mapped from a file. For running etcd on a cloud provider, see the Example hardware configuration documentation.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

The naming of metrics follows the suggested Prometheus best practices. A metric name has an etcd or etcd_debugging prefix as its namespace and a subsystem prefix (for example wal and etcdserver).

How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

How to Set Up a Demo etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Reading from etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the get subcommand to read from etcd:

$ etcdctl --endpoints=$ENDPOINTS get foo
foo
Hello World!
$

where:

foo is the requested key
Hello World! is the retrieved value

Or, for formatted output:

$ etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo
{"header":{"cluster_id":289318470931837780,"member_id":14947050114012957595,"revision":3,"raft_term":4,
"kvs":[{"key":"Zm9v","create_revision":2,"mod_revision":3,"version":2,"value":"SGVsbG8gV29ybGQh"}]}}
$

where write-out="json" causes the value to be output in JSON format (note that the key is not returned).

Reading from etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the get subcommand to read from etcd:

$ etcdctl --endpoints=$ENDPOINTS get foo
foo
Hello World!
$

where:

foo is the requested key
Hello World! is the retrieved value

Or, for formatted output:

$ etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo
{"header":{"cluster_id":289318470931837780,"member_id":14947050114012957595,"revision":3,"raft_term":4,
"kvs":[{"key":"Zm9v","create_revision":2,"mod_revision":3,"version":2,"value":"SGVsbG8gV29ybGQh"}]}}
$

where write-out="json" causes the value to be output in JSON format (note that the key is not returned).

Reading from etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the get subcommand to read from etcd:

$ etcdctl --endpoints=$ENDPOINTS get foo
foo
Hello World!
$

where:

foo is the requested key
Hello World! is the retrieved value

Or, for formatted output:

$ etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo
{"header":{"cluster_id":289318470931837780,"member_id":14947050114012957595,"revision":3,"raft_term":4,
"kvs":[{"key":"Zm9v","create_revision":2,"mod_revision":3,"version":2,"value":"SGVsbG8gV29ybGQh"}]}}
$

where write-out="json" causes the value to be output in JSON format (note that the key is not returned).

Writing to etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the put subcommand to write a key-value pair:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

where:

foo is the key name
"Hello World!" is the quote-delimited value

Writing to etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the put subcommand to write a key-value pair:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

where:

foo is the key name
"Hello World!" is the quote-delimited value

Writing to etcd

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcdctl

Procedure

Use the put subcommand to write a key-value pair:

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

where:

foo is the key name
"Hello World!" is the quote-delimited value

How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Get keys by prefix

$ etcdctl --endpoints=$ENDPOINTS get PREFIX --prefix

Global Options

--endpoints=[127.0.0.1:2379], gRPC endpoints

Options

--prefix, get a range of keys with matching prefix

Example

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Get keys by prefix

$ etcdctl --endpoints=$ENDPOINTS get PREFIX --prefix

Global Options

--endpoints=[127.0.0.1:2379], gRPC endpoints

Options

--prefix, get a range of keys with matching prefix

Example

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

How to get keys by prefix

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Get keys by prefix

$ etcdctl --endpoints=$ENDPOINTS get PREFIX --prefix

Global Options

--endpoints=[127.0.0.1:2379], gRPC endpoints

Options

--prefix, get a range of keys with matching prefix

Example

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3

etcdctl --endpoints=$ENDPOINTS get web --prefix

How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

How to delete keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Add or delete keys

del to remove the specified key or range of keys:

etcdctl del $KEY [$END_KEY]

Options

--prefix[=false]: delete keys with matching prefix
--prev-kv[=false]: return deleted key-value pairs
--from-key[=false]: delete keys that are greater than or equal to the given key using byte compare
--range[=false]: delete range of keys without delay

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key

etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl.
A running etcd cluster.

Terminology

Here are definations of some key terms used in the Example below.

Terms	Defination
etcdctl	The command line tool for interacting with the etcd server.
`txn` command	`txn` command is an abbreviation for “transaction”. It reads multiple etcd requests from standard input and applies them as a single atomic transaction. A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false. View etcdctl key-value commands for more information.
`compare`	The `compare` clause within a transaction (`txn`) serves as a conditional check that determines whether the transaction’s operations should proceed. It ensures changes are only applied if the current state of the key-value store matches expected conditions, thereby maintaining data consistency and preventing conflicts in concurrent environments. To see how the command is structured, view Perform a transaction section below.

Transactions

txn to process all the requests in one transaction:

How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl.
A running etcd cluster.

Terminology

Here are definitions of some key terms used in the Example below.

Terms	Definition
etcdctl	The command line tool for interacting with the etcd server.
`txn` command	`txn` command is an abbreviation for “transaction”. It reads multiple etcd requests from standard input and applies them as a single atomic transaction. A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false. View etcdctl key-value commands for more information.
`compare`	The `compare` clause within a transaction (`txn`) serves as a conditional check that determines whether the transaction’s operations should proceed. It ensures changes are only applied if the current state of the key-value store matches expected conditions, thereby maintaining data consistency and preventing conflicts in concurrent environments. To see how the command is structured, view Perform a transaction section below.

Transactions

txn to process all the requests in one transaction:

How to make multiple writes in a transaction

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl.
A running etcd cluster.

Terminology

Here are definitions of some key terms used in the Example below.

Terms	Definition
etcdctl	The command line tool for interacting with the etcd server.
`txn` command	`txn` command is an abbreviation for “transaction”. It reads multiple etcd requests from standard input and applies them as a single atomic transaction. A transaction consists of list of conditions, a list of requests to apply if all the conditions are true, and a list of requests to apply if any condition is false. View etcdctl key-value commands for more information.
`compare`	The `compare` clause within a transaction (`txn`) serves as a conditional check that determines whether the transaction’s operations should proceed. It ensures changes are only applied if the current state of the key-value store matches expected conditions, thereby maintaining data consistency and preventing conflicts in concurrent environments. To see how the command is structured, view Perform a transaction section below.

Transactions

txn to process all the requests in one transaction:

How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

How to watch keys

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Watching keys

watch to get notified of future changes:

etcdctl watch $KEY [$END_KEY]

Options

-i, --interactive[=false]: interactive mode
--prefix[=false]: watch on a prefix if prefix is set
--rev=0: Revision to start watching
--prev-kv[=false]: get the previous key-value pair before the event happens
--progress-notify[=false]: get periodic watch progress notification from server

Options inherited from parent commands

--endpoints="127.0.0.1:2379": gRPC endpoints

Examples

etcdctl --endpoints=$ENDPOINTS watch stock1
etcdctl --endpoints=$ENDPOINTS put stock1 1000

etcdctl --endpoints=$ENDPOINTS watch stock --prefix
etcdctl --endpoints=$ENDPOINTS put stock1 10
etcdctl --endpoints=$ENDPOINTS put stock2 20

How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

How to create lease

Mon, 01 Jan 0001 00:00:00 +0000

lease to write with TTL:

etcdctl --endpoints=$ENDPOINTS lease grant 300
# lease 2be7547fbc6a5afa granted with TTL(300s)

etcdctl --endpoints=$ENDPOINTS put sample value --lease=2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS get sample

etcdctl --endpoints=$ENDPOINTS lease keep-alive 2be7547fbc6a5afa
etcdctl --endpoints=$ENDPOINTS lease revoke 2be7547fbc6a5afa
# or after 300 seconds
etcdctl --endpoints=$ENDPOINTS get sample

How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

# another client with the same name blocks
etcdctl --endpoints=$ENDPOINTS lock mutex1

How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

LOCK acquires a distributed mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated.

Prerequisites

Install etcd and etcdctl

Creating a lock

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

Options

endpoints - defines a comma-delimited list of machine addresses in the cluster.
ttl - time out in seconds of lock session.

How to create locks

Mon, 01 Jan 0001 00:00:00 +0000

LOCK acquires a distributed mutex with a given name. Once the lock is acquired, it will be held until etcdctl is terminated.

Prerequisites

Install etcd and etcdctl

Creating a lock

lock for distributed lock:

etcdctl --endpoints=$ENDPOINTS lock mutex1

Options

endpoints - defines a comma-delimited list of machine addresses in the cluster.
ttl - time out in seconds of lock session.

How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Ensure etcd and etcdctl is installed.
Check for active etcd cluster.

Conduct Leader election

The etcdctl command is used to conduct leader elections in an etcd cluster. It makes sure that only one client can become leader at a time.

etcdctl --endpoints=$ENDPOINTS elect <election-name> [proposal]

etcdctl --endpoints=$ENDPOINTS elect election-name p1

Options

--endpoints : $ENDPOINTS

Address of each etcd cluster members.

election-name string

A string identifier for the election. All participants competing for leadership must use the same election name.

How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Ensure etcd and etcdctl is installed.
Check for active etcd cluster.

Conduct Leader election

The etcdctl command is used to conduct leader elections in an etcd cluster. It makes sure that only one client can become leader at a time.

etcdctl --endpoints=$ENDPOINTS elect <election-name> [proposal]

etcdctl --endpoints=$ENDPOINTS elect election-name p1

Options

--endpoints : $ENDPOINTS

Address of each etcd cluster members.

election-name string

A string identifier for the election. All participants competing for leadership must use the same election name.

How to conduct leader election in etcd cluster

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Ensure etcd and etcdctl is installed.
Check for active etcd cluster.

Conduct Leader election

The etcdctl command is used to conduct leader elections in an etcd cluster. It makes sure that only one client can become leader at a time.

etcdctl --endpoints=$ENDPOINTS elect <election-name> [proposal]

etcdctl --endpoints=$ENDPOINTS elect election-name p1

Options

--endpoints : $ENDPOINTS

Address of each etcd cluster members.

election-name string

A string identifier for the election. All participants competing for leadership must use the same election name.

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd --logger=zap
{"level":"info","ts":"2021-09-20T08:19:31.340-0400","caller":"etcdmain/etcd.go:110","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd
{"level":"info","ts":"2021-09-17T09:19:32.783-0400","caller":"etcdmain/etcd.go:72","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd
{"level":"info","ts":"2021-09-17T09:19:32.783-0400","caller":"etcdmain/etcd.go:72","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:

Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Follow these instructions to locally install, run, and test a single-member cluster of etcd:

Install etcd from pre-built binaries or from source. For details, see Install.

Important: Ensure that you perform the last step of the installation instructions to verify that etcd is in your path.

Launch etcd:

$ etcd
{"level":"info","ts":"2021-09-17T09:19:32.783-0400","caller":"etcdmain/etcd.go:72","msg":... }
⋮

Note: The output produced by etcd are logs — info-level logs can be ignored.

From another terminal, use etcdctl to set a key:

How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Check Overall Status

endpoint status to check the overall status of each endpoint specified in --endpoints flag:

etcdctl endpoint status (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check Health

endpoint health to check the healthiness of each endpoint specified in --endpoints flag:

etcdctl endpoint health (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check KV Hash

endpoint hashkv to check the KV history hash of each endpoint specified in --endpoints flag:

How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Check Overall Status

endpoint status to check the overall status of each endpoint specified in --endpoints flag:

etcdctl endpoint status (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check Health

endpoint health to check the healthiness of each endpoint specified in --endpoints flag:

etcdctl endpoint health (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check KV Hash

endpoint hashkv to check the KV history hash of each endpoint specified in --endpoints flag:

How to check Cluster status

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Install etcd and etcdctl

Check Overall Status

endpoint status to check the overall status of each endpoint specified in --endpoints flag:

etcdctl endpoint status (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check Health

endpoint health to check the healthiness of each endpoint specified in --endpoints flag:

etcdctl endpoint health (--endpoints=$ENDPOINTS|--cluster)

Options

--cluster[=false]: use all endpoints from the cluster member list

Check KV Hash

endpoint hashkv to check the KV history hash of each endpoint specified in --endpoints flag:

Overview

Mon, 01 Jan 0001 00:00:00 +0000

The etcd v3 API is designed to give users a more efficient and cleaner abstraction compared to etcd v2. There are a number of semantic and protocol changes in this new API.

To prove out the design of the v3 API the team has also built a number of example recipes, there is a video discussing these recipes too.

Design

Flatten binary key-value space
Keep the event history until compaction
- access to old version of keys
- user controlled history compaction
Support range query

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Auth

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Snapshot a database

snapshot to save point-in-time snapshot of etcd database:

etcdctl --endpoints=$ENDPOINT snapshot save DB_NAME

Global Options

etcdctl

--endpoints=[127.0.0.1:2379], gRPC endpoints

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

etcdutl

-w, --write-out string set the output format (fields, json, protobuf, simple, table) (default "simple")

Example

ENDPOINTS=$HOST_1:2379
etcdctl --endpoints=$ENDPOINTS snapshot save my.db

Snapshot saved at my.db

etcdutl --write-out=table snapshot status my.db

+---------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+---------+----------+------------+------------+
| c55e8b8 | 9 | 13 | 25 kB |
+---------+----------+------------+------------+

How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Snapshot a database

snapshot to save point-in-time snapshot of etcd database:

etcdctl --endpoints=$ENDPOINT snapshot save DB_NAME

Global Options

etcdctl

--endpoints=[127.0.0.1:2379], gRPC endpoints

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

etcdutl

-w, --write-out string set the output format (fields, json, protobuf, simple, table) (default "simple")

Example

ENDPOINTS=$HOST_1:2379
etcdctl --endpoints=$ENDPOINTS snapshot save my.db

Snapshot saved at my.db

etcdutl --write-out=table snapshot status my.db

+---------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+---------+----------+------------+------------+
| c55e8b8 | 9 | 13 | 25 kB |
+---------+----------+------------+------------+

How to save the database

Mon, 01 Jan 0001 00:00:00 +0000

Pre-requisites

Snapshot a database

snapshot to save point-in-time snapshot of etcd database:

etcdctl --endpoints=$ENDPOINT snapshot save DB_NAME

Global Options

etcdctl

--endpoints=[127.0.0.1:2379], gRPC endpoints

Snapshot can only be requested from one etcd node, so --endpoints flag should contain only one endpoint.

etcdutl

-w, --write-out string set the output format (fields, json, protobuf, simple, table) (default "simple")

Example

ENDPOINTS=$HOST_1:2379
etcdctl --endpoints=$ENDPOINTS snapshot save my.db

Snapshot saved at my.db

etcdutl --write-out=table snapshot status my.db

+---------+----------+------------+------------+
| HASH | REVISION | TOTAL KEYS | TOTAL SIZE |
+---------+----------+------------+------------+
| c55e8b8 | 9 | 13 | 25 kB |
+---------+----------+------------+------------+

Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.4.37 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.5.21 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release v3.6.0 or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

Install

Mon, 01 Jan 0001 00:00:00 +0000

Requirements

Before installing etcd, see the following pages:

Install pre-built binaries

The easiest way to install etcd is from pre-built binaries:

Download the compressed archive file for your platform from Releases, choosing release main or later.
Unpack the archive file. This results in a directory containing the binaries.
Add the executable binaries to your path. For example, rename and/or move the binaries to a directory in your path (like /usr/local/bin), or add the directory created by the previous step to your path.

Feature Gates

Mon, 01 Jan 0001 00:00:00 +0000

This page contains an overview of the various feature gates an administrator can specify on etcd.

See feature stages for an explanation of the stages for a feature.

Overview

Feature gates are a set of key=value pairs that describe etcd features. You can turn these features on or off using the --feature-gates command line flag on etcd.

etcd lets you enable or disable a set of feature gates. Use -h flag to see a full set of feature gates. To set feature gates, use the --feature-gates flag assigned to a list of feature pairs in commandline:

Feature Gates

Mon, 01 Jan 0001 00:00:00 +0000

This page contains an overview of the various feature gates an administrator can specify on etcd.

See feature stages for an explanation of the stages for a feature.

Overview

Feature gates are a set of key=value pairs that describe etcd features. You can turn these features on or off using the --feature-gates command line flag on etcd.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

Do clients have to send requests to the etcd leader?

Raft is leader-based; the leader handles all client requests which need cluster consensus. However, the client does not need to know which node is the leader. Any request that requires consensus sent to a follower is automatically forwarded to the leader. Requests that do not require consensus (e.g., serialized reads) can be processed by any cluster member.

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

listen-client-urls and listen-peer-urls specify the local addresses etcd server binds to for accepting incoming connections. To listen on a port for all interfaces, specify 0.0.0.0 as the listen IP address.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

What is etcd?

etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.

How do you pronounce etcd?

etcd is pronounced /ˈɛtsiːdiː/, and means “distributed etc directory.”

Do clients have to send requests to the etcd leader?

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

What is etcd?

etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.

How do you pronounce etcd?

etcd is pronounced /ˈɛtsiːdiː/, and means “distributed etc directory.”

Do clients have to send requests to the etcd leader?

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

What is etcd?

etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.

How do you pronounce etcd?

etcd is pronounced /ˈɛtsiːdiː/, and means “distributed etc directory.”

Do clients have to send requests to the etcd leader?

How to migrate etcd from v2store to v3store

Mon, 01 Jan 0001 00:00:00 +0000

migrate to transform etcd v2store to v3store.

Learn more about the difference between the etcd data storage format and the etcd versioning.

etcd v2.Y releases store data in the v2store, which supports the legacy API v2. When upgrading to etcd v3.Y releases, the cluster may continue to carry this v2store data until it is explicitly migrated into the newer v3store, which is required for full API v3 support and long-term compatibility.

This document focus on how to migrate existing data from the v2store to the v3store. For a detailed guideline of migrating from API v2 to API v3, please refer to Migrate applications from using API v2 to API v3.

How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

How to Add and Remove Members

Mon, 01 Jan 0001 00:00:00 +0000

member to add,remove,update membership:

# For each machine
TOKEN=my-etcd-token-1
CLUSTER_STATE=new
NAME_1=etcd-node-1
NAME_2=etcd-node-2
NAME_3=etcd-node-3
HOST_1=10.240.0.13
HOST_2=10.240.0.14
HOST_3=10.240.0.15
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 \
	--listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 \
	--listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} \
	--initial-cluster-token ${TOKEN}

Then replace a member with member remove and member add commands:

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Note that third-party libraries and tools (not hosted on etcd-io main repository) mentioned below are not tested or maintained by the etcd team. Before using them, users are recommended to read and investigate them.

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2
etcdloadtest - A command line load test client for etcd version 3.0 and above.
etcd-workbench - A free and powerful ui client for etcd v3. Provides desktop application and web packages.

Libraries

The sections below list etcd client libraries by language.

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-workbench - A free and powerful ui client for etcd v3. Provides desktop application and web packages.

Libraries

The sections below list etcd client libraries by language.

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcdhelper - An intellij platform plugin for etcd.
etcd-workbench - A free and powerful ui client for etcd v3. Provides desktop application and web packages.

Libraries

The sections below list etcd client libraries by language.

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
etcd-tui - A modern terminal user interface (TUI) for interacting with your etcd database. Navigate keys, view values, filter data, and manage your etcd cluster directly from your terminal.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcd-defrag - An easier to use and smarter etcd defragmentation tool.
etcdhelper - An intellij platform plugin for etcd.
etcd-workbench - A free and powerful ui client for etcd v3. Provides desktop application and web packages.

Libraries

The sections below list etcd client libraries by language.

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
etcd-tui - A modern terminal user interface (TUI) for interacting with your etcd database. Navigate keys, view values, filter data, and manage your etcd cluster directly from your terminal.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcd-defrag - An easier to use and smarter etcd defragmentation tool.
etcdhelper - An intellij platform plugin for etcd.
etcd-workbench - A free and powerful ui client for etcd v3. Provides desktop application and web packages.

Libraries

The sections below list etcd client libraries by language.

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Note that third-party libraries and tools (not hosted on https://github.com/etcd-io) mentioned below are not tested or maintained by the etcd team. Before using them, users are recommended to read and investigate them.

Tools

etcdctl - A command line client for etcd
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcdloadtest - A command line load test client for etcd version 3.0 and above.
etcd-tui - A modern terminal user interface (TUI) for interacting with your etcd database. Navigate keys, view values, filter data, and manage your etcd cluster directly from your terminal.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-manager - A modern, efficient, multi-platform and free etcd 3.x GUI & client tool. Available for Windows, Linux and Mac.
etcd-backup-restore - Utility to periodically and incrementally backup and restore the etcd.
etcd-druid - A Kubernetes operator to deploy etcd clusters and manage day-2 operations.
etcdadm - A command-line tool for operating an etcd cluster.
etcd-defrag - An easier to use and smarter etcd defragmentation tool.
etcdhelper - An intellij platform plugin for etcd.

Libraries

The sections below list etcd client libraries by language.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

If any part of the etcd project has bugs or documentation mistakes, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency, the heartbeat interval and election timeout settings may need tuning.

The network isn’t the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

The protocol uses a new discovery token to bootstrap one unique etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if it fails halfway, it must not be used to bootstrap another etcd cluster.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery token and endpoint list.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery token and endpoint list.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the capnslog library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
- Examples:
  - A failure to allocate disk space for WAL
Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the zap library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

DebugLevel logs are typically voluminous, and are usually disabled in production.
- Examples:
  - Send a normal message to a remote peer
  - Write a log entry to disk
InfoLevel is the default logging priority.
- Examples:
  - Startup configuration
  - Start to do snapshot
  - Add a new node into the cluster
  - Add a new user into auth subsystem
WarnLevel logs are more important than Info, but don’t need individual human review.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the zap library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

DebugLevel logs are typically voluminous, and are usually disabled in production.
- Examples:
  - Send a normal message to a remote peer
  - Write a log entry to disk
InfoLevel is the default logging priority.
- Examples:
  - Startup configuration
  - Start to do snapshot
  - Add a new node into the cluster
  - Add a new user into auth subsystem
WarnLevel logs are more important than Info, but don’t need individual human review.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the zap library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

DebugLevel logs are typically voluminous, and are usually disabled in production.
- Examples:
  - Send a normal message to a remote peer
  - Write a log entry to disk
InfoLevel is the default logging priority.
- Examples:
  - Startup configuration
  - Start to do snapshot
  - Add a new node into the cluster
  - Add a new user into auth subsystem
WarnLevel logs are more important than Info, but don’t need individual human review.

Golang modules

Mon, 01 Jan 0001 00:00:00 +0000

The etcd project (since version 3.5) is organized into multiple golang modules hosted in a single repository.

There are following modules:

go.etcd.io/etcd/api/v3 - contains API definitions (like protos & proto-generated libraries) that defines communication protocol between etcd clients and server.
go.etcd.io/etcd/pkg/v3 - collection of utility packages used by etcd without being specific to etcd itself. A package belongs here only if it could possibly be moved out into its own repository in the future. Please avoid adding here code that has a lot of dependencies on its own, as they automatically becoming dependencies of the client library (that we want to keep lightweight).

Golang modules

Mon, 01 Jan 0001 00:00:00 +0000

The etcd project (since version 3.5) is organized into multiple golang modules hosted in a single repository.

There are following modules:

go.etcd.io/etcd/api/v3 - contains API definitions (like protos & proto-generated libraries) that defines communication protocol between etcd clients and server.
go.etcd.io/etcd/pkg/v3 - collection of utility packages used by etcd without being specific to etcd itself. A package belongs here only if it could possibly be moved out into its own repository in the future. Please avoid adding here code that has a lot of dependencies on its own, as they automatically becoming dependencies of the client library (that we want to keep lightweight).

Golang modules

Mon, 01 Jan 0001 00:00:00 +0000

The etcd project (since version 3.5) is organized into multiple golang modules hosted in a single repository.

There are following modules:

go.etcd.io/etcd/api/v3 - contains API definitions (like protos & proto-generated libraries) that defines communication protocol between etcd clients and server.
go.etcd.io/etcd/pkg/v3 - collection of utility packages used by etcd without being specific to etcd itself. A package belongs here only if it could possibly be moved out into its own repository in the future. Please avoid adding here code that has a lot of dependencies on its own, as they automatically becoming dependencies of the client library (that we want to keep lightweight).

Data model

Mon, 01 Jan 0001 00:00:00 +0000

etcd is designed to reliably store infrequently updated data and provide reliable watch queries. etcd exposes previous versions of key-value pairs to support inexpensive snapshots and watch history events (“time travel queries”). A persistent, multi-version, concurrency-control data model is a good fit for these use cases.

etcd stores data in a multiversion persistent key-value store. The persistent key-value store preserves the previous version of a key-value pair when its value is superseded with new data. The key-value store is effectively immutable; its operations do not update the structure in-place, but instead always generate a new updated structure. All past versions of keys are still accessible and watchable after modification. To prevent the data store from growing indefinitely over time and from maintaining old versions, the store may be compacted to shed the oldest versions of superseded data.

Data model

Mon, 01 Jan 0001 00:00:00 +0000

Data model

Mon, 01 Jan 0001 00:00:00 +0000

Data model

Mon, 01 Jan 0001 00:00:00 +0000

etcd client design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Client Design

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Introduction

etcd client design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Client Design

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Introduction

etcd client design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Client Design

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Introduction

etcd client design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Client Design

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Introduction

etcd learner design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Learner

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

1. New Cluster member overloads Leader

A newly joined etcd member starts with no data, thus demanding more updates from leader until it catches up with leader’s logs. Then leader’s network is more likely to be overloaded, blocking or dropping leader heartbeats to followers. In such case, a follower may election-timeout to start a new leader election. That is, a cluster with a new member is more vulnerable to leader election. Both leader election and the subsequent update propagation to the new member are prone to causing periods of cluster unavailability (see Figure 1).

etcd learner design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Learner

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

1. New Cluster member overloads Leader

etcd learner design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Learner

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

1. New Cluster member overloads Leader

etcd learner design

Mon, 01 Jan 0001 00:00:00 +0000

etcd Learner

Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

1. New Cluster member overloads Leader

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

The v3 protocol uses gRPC as its transport instead of a RESTful interface like v2. This new protocol provides an opportunity to iterate on and improve the v2 design. For example, v3 auth has connection based authentication, rather than v2’s slower per-request authentication. Additionally, v2 auth’s semantics tend to be unwieldy in practice with respect to reasoning about consistency, which will be described in the next sections. For v3, there is a well-defined description and implementation of the authentication mechanism which fixes the deficiencies in the v2 auth system.

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

etcd API

Mon, 01 Jan 0001 00:00:00 +0000

This document is meant to give an overview of the v3 etcd APIs central design. This should not be mistaken with etcd v2 API, deprecated in etcd v3.5. It is by no means all encompassing, but intended to focus on the basic ideas needed to understand etcd without the distraction of less common API calls. All etcd APIs are defined in gRPC services, which categorize remote procedure calls (RPCs) understood by the etcd server. A full listing of all etcd RPCs are documented in markdown in the gRPC API listing.

etcd API

Mon, 01 Jan 0001 00:00:00 +0000

etcd API

Mon, 01 Jan 0001 00:00:00 +0000

etcd3 API

Mon, 01 Jan 0001 00:00:00 +0000

This document is meant to give an overview of the etcd3 API’s central design. It is by no means all encompassing, but intended to focus on the basic ideas needed to understand etcd without the distraction of less common API calls. All etcd3 API’s are defined in gRPC services, which categorize remote procedure calls (RPCs) understood by the etcd server. A full listing of all etcd RPCs are documented in markdown in the gRPC API listing.

etcd persistent storage files

Mon, 01 Jan 0001 00:00:00 +0000

This document explains the etcd persistent storage format: naming, content and tools that allow developers to inspect them. Going forward the document should be extended with changes to the storage model. This document is targeted at etcd developers to help with their data recovery needs.

Prerequisites

The following articles provide helpful background information for this document:

etcd data model overview: https://etcd.io/docs/v3.5/learning/data_model
Raft overview: https://raft.github.io/raft.pdf (especially “5.3 Log replication” section).

Overview

Long leaving files

File name High level purpose

./member/snap/db

bbolt b+tree that stores all the applied data, membership authorization information & metadata. It’s aware of what's the last applied WAL log index ("consistent_index").

./member/snap/0000000000000002-0000000000049425.snap
./member/snap/0000000000000002-0000000000061ace.snap

Periodic snapshots of legacy v2 store, containing:

basic membership information
etcd-version

etcd persistent storage files

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

The following articles provide helpful background information for this document:

etcd data model overview: https://etcd.io/docs/v3.6/learning/data_model
Raft overview: https://raft.github.io/raft.pdf (especially “5.3 Log replication” section).

Overview

Long leaving files

File name High level purpose

./member/snap/db

bbolt b+tree that stores all the applied data, membership authorization information & metadata. It’s aware of what's the last applied WAL log index ("consistent_index").

./member/snap/0000000000000002-0000000000049425.snap
./member/snap/0000000000000002-0000000000061ace.snap

Periodic snapshots of legacy v2 store, containing:

basic membership information
etcd-version

etcd persistent storage files

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

The following articles provide helpful background information for this document:

etcd data model overview: https://etcd.io/docs/v3.6/learning/data_model
Raft overview: https://raft.github.io/raft.pdf (especially “5.3 Log replication” section).

Overview

Long leaving files

File name High level purpose

./member/snap/db

bbolt b+tree that stores all the applied data, membership authorization information & metadata. It’s aware of what's the last applied WAL log index ("consistent_index").

./member/snap/0000000000000002-0000000000049425.snap
./member/snap/0000000000000002-0000000000061ace.snap

Periodic snapshots of legacy v2 store, containing:

basic membership information
etcd-version

etcd API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

etcd is a consistent and durable key value store. The key value store is exposed through gRPC Services. etcd ensures the strongest consistency and durability guarantees for a distributed system. This specification enumerates the API guarantees made by etcd.

APIs to consider

KV APIs
- Range
- Put
- Delete
- Transaction
Watch APIs
- Watch
Lease APIs
- Grant
- [Revoke]
- Keep alive

KV API allows for direct reading and manipulation of key value store. Watch API allows subscribing to key value store changes. Lease API allows assigning a time to live to a key.

etcd API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

APIs to consider

KV APIs
- Range
- Put
- Delete
- Transaction
Watch APIs
- Watch
Lease APIs
- Grant
- [Revoke]
- Keep alive

KV API allows for direct reading and manipulation of key value store. Watch API allows subscribing to key value store changes. Lease API allows assigning a time to live to a key.

etcd API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

APIs to consider

KV APIs
- Range
- Put
- Delete
- Transaction
Watch APIs
- Watch
Lease APIs
- Grant
- [Revoke]
- Keep alive

KV API allows for direct reading and manipulation of key value store. Watch API allows subscribing to key value store changes. Lease API allows assigning a time to live to a key.

etcd API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

APIs to consider

KV APIs
- Range
- Put
- Delete
- Transaction
Watch APIs
- Watch
Lease APIs
- Grant
- [Revoke]
- Keep alive

KV API allows for direct reading and manipulation of key value store. Watch API allows subscribing to key value store changes. Lease API allows assigning a time to live to a key.

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

The name “etcd” originated from two ideas, the unix “/etc” folder and “d"istributed systems. The “/etc” folder is a place to store configuration data for a single system whereas etcd stores configuration information for large scale distributed systems. Hence, a “d"istributed “/etc” is “etcd”.

etcd is designed as a general substrate for large scale distributed systems. These are systems that will never tolerate split-brain operation and are willing to sacrifice availability to achieve this end. etcd stores metadata in a consistent and fault-tolerant way. An etcd cluster is meant to provide key-value storage with best of class stability, reliability, scalability and performance.

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Users mostly interact with etcd by putting or getting the value of a key. This section describes how to do that by using etcdctl, a command line tool for interacting with etcd server. The concepts described here should apply to the gRPC APIs or client library APIs.

The API version used by etcdctl to speak to etcd may be set to version 2 or 3 via the ETCDCTL_API environment variable. By default, etcdctl on master (3.4) uses the v3 API and earlier versions (3.3 and earlier) default to the v2 API.

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

etcd v3 uses gRPC for its messaging protocol. The etcd project includes a gRPC-based Go client and a command line utility, etcdctl, for communicating with an etcd cluster through gRPC. For languages with no gRPC support, etcd provides a JSON gRPC gateway. This gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Using gRPC gateway

The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON. The following examples use curl, but any HTTP/JSON client should work all the same.

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

Using gRPC gateway

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

Using gRPC gateway

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

Using gRPC gateway

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd provides a gRPC resolver to support an alternative name system that fetches endpoints from etcd for discovering gRPC services. The underlying mechanism is based on watching updates to keys prefixed with the service name.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client and given a target for resolution:

import (
	"go.etcd.io/etcd/clientv3"
	etcdnaming "go.etcd.io/etcd/clientv3/naming/resolver"

	"google.golang.org/grpc"
)

...

cli, err := clientv3.NewFromURL("http://localhost:2379")
if err != nil {
	// ...
}
r, err := etcdnaming.NewBuilder(cli)
if err != nil {
	// ...
}
conn, gerr := grpc.Dial("my-service", grpc.WithResolvers(r), grpc.WithBlock(), ...

Managing service endpoints

The etcd resolver treats all keys under the prefix of the resolution target following a “/” (e.g., “my-service/”) with JSON-encoded go-grpc naming.Update values as potential service endpoints. Endpoints are added to the service by creating new keys and removed from the service by deleting keys.

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

Note that this feature is experimental because it depends on the google.golang.org/grpc/resolver package, which is still experimental in grpc-go.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client:

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

Note that this feature is experimental because it depends on the google.golang.org/grpc/resolver package, which is still experimental in grpc-go.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client:

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

Note that this feature is experimental because it depends on the google.golang.org/grpc/resolver package, which is still experimental in grpc-go.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client:

Embedding etcd in a Go Application

Mon, 01 Jan 0001 00:00:00 +0000

The etcd embed go package provides a simple way to embed an etcd server directly into your application.

For more details, see the embed package documentation.

Embedding etcd in a Go Application

Mon, 01 Jan 0001 00:00:00 +0000

The etcd embed go package provides a simple way to embed an etcd server directly into your application.

For more details, see the embed package documentation.

Embedding etcd in a Go Application

Mon, 01 Jan 0001 00:00:00 +0000

The etcd embed go package provides a simple way to embed an etcd server directly into your application.

For more details, see the embed package documentation.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server.

Storage size limit

The default storage size limit is 2 GiB, configurable with --quota-backend-bytes flag. 8 GiB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

Storage size limit

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

Storage size limit

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

Storage size limit

etcd features

Mon, 01 Jan 0001 00:00:00 +0000

This document provides an overview of etcd features to help users better understand the features and related deprecation process. If you are interested in knowing about how features are developed in the etcd, please see these development guidelines.

The etcd features fall into three stages, experimental, stable, and unsafe. You can get the list of features by running etcd --help.

Experimental

In order to get early feedback, any new feature is usually added as an experimental feature. The experimental feature can be identified by looking at the flag name, which should have --experimental as a prefix. Please consider the following points while using an experimental feature:

etcd features

Mon, 01 Jan 0001 00:00:00 +0000

The etcd features fall into three stages, experimental, stable, and unsafe. You can get the list of features by running etcd --help.

Experimental

etcd features

Mon, 01 Jan 0001 00:00:00 +0000

The etcd features fall into three stages, experimental, stable, and unsafe. You can get the list of features by running etcd --help.

Experimental

etcd features

Mon, 01 Jan 0001 00:00:00 +0000

The etcd features fall into three stages, experimental, stable, and unsafe. You can get the list of features by running etcd --help.

Experimental

Configuration options

Mon, 01 Jan 0001 00:00:00 +0000

etcd is configurable through a configuration file, various command-line flags, and environment variables.

A reusable configuration file is a YAML file made with name and value of one or more command-line flags described below. In order to use this file, specify the file path as a value to the --config-file flag or ETCD_CONFIG_FILE environment variable. The sample configuration file can be used as a starting point to create a new configuration file as needed.

Configuration options

Mon, 01 Jan 0001 00:00:00 +0000

You can configure etcd through the following:

Command-line flags
Environment variables: every flag has a corresponding environment variable that has the same name but is prefixed withETCD_ and formatted in all caps and snake case. For example, --some-flag would be ETCD_SOME_FLAG.
Configuration file

Caution: If you mix-and-match configuration options, then the following rules apply.

Command-line flags take precedence over environment variables.
If you provide a configuration file all command-line flags and environment variables are ignored.

Command-line flags

Flags are presented below using the format --flag-name DEFAULT_VALUE.

Configuration options

Mon, 01 Jan 0001 00:00:00 +0000

You can configure etcd through the following:

Command-line flags
Environment variables: every flag has a corresponding environment variable that has the same name but is prefixed withETCD_ and formatted in all caps and snake case. For example, --some-flag would be ETCD_SOME_FLAG.
Configuration file

Caution: If you mix-and-match configuration options, then the following rules apply.

Command-line flags take precedence over environment variables.
If you provide a configuration file all command-line flags and environment variables are ignored.

Command-line flags

Flags are presented below using the format --flag-name DEFAULT_VALUE.

Configuration options

Mon, 01 Jan 0001 00:00:00 +0000

You can configure etcd through the following:

Command-line flags
Environment variables: every flag has a corresponding environment variable that has the same name but is prefixed with ETCD_ and formatted in all caps and snake case. For example, --some-flag would be ETCD_SOME_FLAG.
Configuration file

Caution: If you mix-and-match configuration options, then the following rules apply.

Command-line flags take precedence over environment variables.
If you provide a configuration file all command-line flags and environment variables are ignored.

Command-line flags

Flags are presented below using the format --flag-name DEFAULT_VALUE.

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Authentication was added in etcd 2.1. The etcd v3 API slightly modified the authentication feature’s API and user interface to better fit the new data model. This guide is intended to help users set up basic authentication and role-based access control in etcd v3.

Special users and roles

There is one special user, root, and one special role, root.

User `root`

The root user, which has full access to etcd, must be created before activating authentication. The idea behind the root user is for administrative purposes: managing roles and ordinary users. The root user must have the root role and is allowed to change anything inside etcd.

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Special users and roles

There is one special user, root, and one special role, root.

User `root`

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Special users and roles

There is one special user, root, and one special role, root.

User `root`

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Special users and roles

There is one special user, root, and one special role, root.

User `root`

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication.

To get up and running, first have a CA certificate and a signed key pair for one member. It is recommended to create and sign a new key pair for every member in a cluster.

For convenience, the cfssl tool provides an easy interface to certificate generation, and we provide an example using the tool here. Alternatively, try this guide to generating self-signed key pairs.

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication. Note that etcd doesn’t enable RBAC based authentication or the authentication feature in the transport layer by default to reduce friction for users getting started with the database. Further, changing this default would be a breaking change for the project which was established since 2013. An etcd cluster which doesn’t enable security features can expose its data to any clients.

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication.

Warning

etcd doesn’t enable RBAC based authentication or the authentication feature in the transport layer by default to reduce friction for users getting started with the database. Further, changing this default would be a breaking change for the project which was established since 2013. An etcd cluster which doesn’t enable security features can expose its data to any clients.

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, the IPs of the cluster members may be unknown ahead of time. In these cases, the etcd cluster can be bootstrapped with the help of a discovery service.

Once an etcd cluster is up and running, adding or removing members is done via runtime reconfiguration. To better understand the design behind runtime reconfiguration, we suggest reading the runtime configuration design document.

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Run etcd clusters as a Kubernetes StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet.

Example Manifest

This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes.

If you copy the contents of the manifest into a file named etcd.yaml, it can be applied to a cluster with this command.

$ kubectl apply --filename etcd.yaml

Upon being applied, wait for the pods to become ready.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 24m
etcd-1 1/1 Running 0 24m
etcd-2 1/1 Running 0 24m

The container used in the example includes etcdctl and can be called directly inside the pods.

Run etcd clusters as a Kubernetes StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet.

Example Manifest

This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes.

If you copy the contents of the manifest into a file named etcd.yaml, it can be applied to a cluster with this command.

$ kubectl apply --filename etcd.yaml

Upon being applied, wait for the pods to become ready.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 24m
etcd-1 1/1 Running 0 24m
etcd-2 1/1 Running 0 24m

The container used in the example includes etcdctl and can be called directly inside the pods.

Run etcd clusters as a Kubernetes StatefulSet

Mon, 01 Jan 0001 00:00:00 +0000

Below demonstrates how to perform the static bootstrap process as a Kubernetes StatefulSet.

Example Manifest

This manifest contains a service and statefulset for deploying a static etcd cluster in kubernetes.

If you copy the contents of the manifest into a file named etcd.yaml, it can be applied to a cluster with this command.

$ kubectl apply --filename etcd.yaml

Upon being applied, wait for the pods to become ready.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
etcd-0 1/1 Running 0 24m
etcd-1 1/1 Running 0 24m
etcd-2 1/1 Running 0 24m

The container used in the example includes etcdctl and can be called directly inside the pods.

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with Docker using the static bootstrap process.

Docker

In order to expose the etcd API to clients outside of Docker host, use the host IP address of the container. Please see docker inspect for more detail on how to get the IP address. Alternatively, specify --net=host flag to docker run command to skip placing the container inside of a separate network stack.

Running a single node etcd

Use the host IP address when configuring etcd:

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with Docker using the static bootstrap process.

Docker

Running a single node etcd

Use the host IP address when configuring etcd:

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with Docker using the static bootstrap process.

Docker

Running a single node etcd

Use the host IP address when configuring etcd:

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with Docker using the static bootstrap process.

Docker

Running a single node etcd

Use the host IP address when configuring etcd:

Failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Failures are common in a large deployment of machines. A machine fails when its hardware or software malfunctions. Multiple machines fail together when there are power failures or network issues. Multiple kinds of failures can also happen at once; it is almost impossible to enumerate all possible failure cases.

In this section, we catalog kinds of failures and discuss how etcd is designed to tolerate these failures. Most users, if not all, can map a particular failure into one kind of failure. To prepare for rare or unrecoverable failures, always back up the etcd cluster.

Failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Failure modes

Mon, 01 Jan 0001 00:00:00 +0000

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd is designed to withstand machine failures. An etcd cluster automatically recovers from temporary failures (e.g., machine reboots) and tolerates up to (N-1)/2 permanent failures for a cluster of N members. When a member permanently fails, whether due to hardware failure or disk corruption, it loses access to the cluster. If the cluster permanently loses more than (N-1)/2 members then it disastrously fails, irrevocably losing quorum. Once quorum is lost, the cluster cannot reach consensus and therefore cannot continue accepting updates.

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses.

The gateway supports multiple etcd server endpoints and works on a simple round-robin policy. It only routes to available endpoints and hides failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses. It does not terminate TLS connections, do TLS handshakes on behalf of its clients, or verify if the connection is secured.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

The gRPC proxy is a stateless etcd reverse proxy operating at the gRPC layer (L7). The proxy is designed to reduce the total processing load on the core etcd cluster. For horizontal scalability, it coalesces watch and lease API requests. To protect the cluster against abusive clients, it caches key range requests.

The gRPC proxy supports multiple etcd server endpoints. When the proxy starts, it randomly picks one etcd server endpoint to use. This endpoint serves all requests until the proxy detects an endpoint failure. If the gRPC proxy detects an endpoint failure, it switches to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

etcd usually runs well with limited resources for development or testing purposes; it’s common to develop with etcd on a laptop or a cheap cloud machine. However, when running etcd clusters in production, some hardware guidelines are useful for proper administration. These suggestions are not hard rules; they serve as a good starting point for a robust production deployment. As always, deployments should be tested with simulated workloads before running in production.

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

An etcd cluster needs periodic maintenance to remain reliable. Depending on an etcd application’s needs, this maintenance can usually be automated and performed without downtime or significantly degraded performance.

All etcd maintenance manages storage resources consumed by the etcd keyspace. Failure to adequately control the keyspace size is guarded by storage space quotas; if an etcd member runs low on space, a quota will trigger cluster-wide alarms which will put the system into a limited-operation maintenance mode. To avoid running out of space for writes to the keyspace, the etcd keyspace history must be compacted. Storage space itself may be reclaimed by defragmenting etcd members. Finally, periodic snapshot backups of etcd member state makes it possible to recover any unintended logical data loss or corruption caused by operational error.

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

If --debug is set, the etcd server exports debugging information on its client port under the /debug path. Take care when setting --debug, since there will be degraded performance and verbose logging.

The /debug/pprof endpoint is the standard go runtime profiling endpoint. This can be used to profile CPU, heap, mutex, and goroutine utilization. For example, here go tool pprof gets the top 10 functions where etcd spends its time:

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

If --log-level=debug is set, the etcd server exports debugging information on its client port under the /debug path. Take care when setting --log-level=debug, since there will be degraded performance and verbose logging.

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

etcd provides stable, sustained high performance. Two factors define performance: latency and throughput. Latency is the time taken to complete an operation. Throughput is the total operations completed within some time period. Usually average latency increases as the overall throughput increases when etcd accepts concurrent client requests. In common cloud environments, like a standard n-4 on Google Compute Engine (GCE) or a comparable machine type on AWS, a three member etcd cluster finishes a request in less than one millisecond under light load, and can complete more than 30,000 requests per second under heavy load.

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform the cluster of the new configuration and then start the new member.

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform the cluster of the new configuration and then start the new member.

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform the cluster of the new configuration and then start the new member.

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform the cluster of the new configuration and then start the new member.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Reconfiguration requests can only be processed when a majority of cluster members are functioning. It is highly recommended to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not be able to make progress and need to restart from majority failure.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Current support

The following table lists etcd support status for common architectures and operating systems:

Architecture	Operating System	Status	Maintainers
amd64	Darwin	Experimental	etcd maintainers
amd64	Linux	Stable	etcd maintainers
amd64	Windows	Experimental
arm64	Linux	Experimental
arm	Linux	Unstable
386	Linux	Unstable
ppc64le	Linux	Stable	etcd maintainers

etcd-maintainers are listed in https://github.com/etcd-io/etcd/blob/main/OWNERS.

Experimental platforms appear to work in practice and have some platform specific code in etcd, but do not fully conform to the stable support policy. Unstable platforms have been lightly tested, but less than experimental. Unlisted architecture and operating system pairs are currently unsupported; caveat emptor.

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Support tiers

etcd runs on different platforms, but the guarantees it provides depends on a platform’s support tier:

Tier 1: fully supported by etcd maintainers; etcd is guaranteed to pass all tests including functional and robustness tests.
Tier 2: etcd is guaranteed to pass integration and end-to-end tests but not necessarily functional or robustness tests.
Tier 3: etcd is guaranteed to build, may be lightly tested (or not), and so it should be considered unstable.

Current support

The following table lists currently supported platforms and their corresponding etcd support tier:

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Support tiers

etcd runs on different platforms, but the guarantees it provides depends on a platform’s support tier:

Tier 1: fully supported by etcd maintainers; etcd is guaranteed to pass all tests including functional and robustness tests.
Tier 2: etcd is guaranteed to pass integration and end-to-end tests but not necessarily functional or robustness tests.
Tier 3: etcd is guaranteed to build, may be lightly tested (or not), and so it should be considered unstable.

Current support

The following table lists currently supported platforms and their corresponding etcd support tier:

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Support tiers

etcd runs on different platforms, but the guarantees it provides depends on a platform’s support tier:

Tier 1: fully supported by etcd maintainers; etcd is guaranteed to pass all tests including functional and robustness tests.
Tier 2: etcd is guaranteed to pass integration and end-to-end tests but not necessarily functional or robustness tests.
Tier 3: etcd is guaranteed to build, may be lightly tested (or not), and so it should be considered unstable.

Current support

The following table lists currently supported platforms and their corresponding etcd support tier:

Migrate applications from using API v2 to API v3

Mon, 01 Jan 0001 00:00:00 +0000

The data store v2 is still accessible from the API v2 after upgrading to etcd3. Thus, it will work as before and require no application changes. With etcd 3, applications use the new grpc API v3 to access the mvcc store, which provides more features and improved performance. The mvcc store and the old store v2 are separate and isolated; writes to the store v2 will not affect the mvcc store and, similarly, writes to the mvcc store will not affect the store v2.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the versions supported by the etcd project.

Service versioning and supported versions

etcd versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following Semantic Versioning terminology. New minor versions may add additional features to the API.

The etcd project maintains release branches for the current version and previous release. For example, when v3.5 is the current version, v3.4 is supported. When v3.6 is released, v3.4 goes out of support.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the versions supported by the etcd project.

Service versioning and supported versions

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the versions supported by the etcd project.

Service versioning and supported versions

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the versions supported by the etcd project.

Service versioning and supported versions

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the versions supported by the etcd project.

Service versioning and supported versions

Data Corruption

Mon, 01 Jan 0001 00:00:00 +0000

etcd has built in automated data corruption detection to prevent member state from diverging.

Enabling data corruption detection

Data corruption detection can be done in two ways:

Initial check, enabled with --experimental-initial-corrupt-check flag.
Periodic check, enabled with --experimental-corrupt-check-time flag.

Initial check will be executed during bootstrap of etcd member. Member will compare its persistent state vs other members and exit if there is a mismatch.

Periodic check will be executed by the cluster leader in a cluster that is already running. Leader will compare its persistent state vs other members and raise a CORRUPT ALARM if there is a mismatch. Period of checks is configured using format: 1m - every minute, 1h - evey hour. Recommended period is a couple of hours as there is a high performance cost. Running a check requires computing a checksum by scanning entire etcd content at given revision.

Data Corruption

Mon, 01 Jan 0001 00:00:00 +0000

etcd has built in automated data corruption detection to prevent member state from diverging.

Enabling data corruption detection

Data corruption detection can be done using:

Initial check, enabled with --experimental-initial-corrupt-check flag.
Periodic check of:
- Compacted revision hash, enabled with --experimental-compact-hash-check-enabled flag.
- Latest revision hash, enabled with --experimental-corrupt-check-time flag.

Initial check will be executed during bootstrap of etcd member. Member will compare its persistent state vs other members and exit if there is a mismatch.

Data Corruption

Mon, 01 Jan 0001 00:00:00 +0000

etcd has built in automated data corruption detection to prevent member state from diverging.

Enabling data corruption detection

Data corruption detection can be done using:

Initial check, enabled with --experimental-initial-corrupt-check flag.
Periodic check of:
- Compacted revision hash, enabled with --experimental-compact-hash-check-enabled flag.
- Latest revision hash, enabled with --experimental-corrupt-check-time flag.

Initial check will be executed during bootstrap of etcd member. Member will compare its persistent state vs other members and exit if there is a mismatch.

Data Corruption

Mon, 01 Jan 0001 00:00:00 +0000

etcd has built in automated data corruption detection to prevent member state from diverging.

Enabling data corruption detection

Data corruption detection can be done using:

Initial check, enabled with --experimental-initial-corrupt-check flag.
Periodic check of:
- Compacted revision hash, enabled with --experimental-compact-hash-check-enabled flag.
- Latest revision hash, enabled with --experimental-corrupt-check-time flag.

Initial check will be executed during bootstrap of etcd member. Member will compare its persistent state vs other members and exit if there is a mismatch.

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Two components of etcd storage consume physical memory. The etcd process allocates an in-memory index to speed key lookup. The process’s page cache, managed by the operating system, stores recently-accessed data from disk for quick re-use.

The in-memory index holds all the keys in a B-tree data structure, along with pointers to the on-disk data (the values). Each key in the B-tree may contain multiple pointers, pointing to different versions of its values. The theoretical memory consumption of the in-memory index can hence be approximated with the formula:

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: The watch features are under active development, and their memory usage may change as that development progresses. We do not expect it to significantly increase beyond the figures stated below.

A primary goal of etcd is supporting a very large number of watchers doing a massively large amount of watching. etcd aims to support O(10k) clients, O(100K) watch streams (O(10) streams per client) and O(10M) total watchings (O(100) watching per stream). The memory consumed by each individual watching accounts for the largest portion of etcd’s overall usage, and is therefore the focus of current and future optimizations.

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Note

The watch features are under active development, and their memory usage may change as that development progresses. We do not expect it to significantly increase beyond the figures stated below.

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

See etcd v3 performance benchmarking.

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the hey HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

Downgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to downgrading etcd clusters and applications.

Downgrading an etcd v3.x cluster

Downgrade etcd from 3.5 to 3.4

Downgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to downgrading etcd clusters and applications.

Downgrading an etcd v3.x cluster

Downgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to downgrading etcd clusters and applications.

Downgrading an etcd v3.x cluster

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Moving from etcd API v2 to API v3

Migrate applications from using API v2 to API v3

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Upgrade etcd from v3.5 to v3.6

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd v3.5 to v3.6 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.5 processes and replace them with etcd v3.6 processes
after running all v3.6 processes, new features in v3.6 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Update 3.5

Important

Before upgrading to 3.6, make sure that all of your 3.5 members are updated to 3.5.26 or later. Patch releases 3.5.24 through 3.5.26 fix several potential upgrade blockers.

Upgrade etcd from v3.5 to v3.6

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd v3.5 to v3.6 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.5 processes and replace them with etcd v3.6 processes
after running all v3.6 processes, new features in v3.6 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Update 3.5

Important

Before upgrading to 3.6, make sure that all of your 3.5 members are updated to 3.5.26 or later. Patch releases 3.5.24 through 3.5.26 fix several potential upgrade blockers.

Downgrade etcd from 3.5 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

content/en/docs/v3.5/op-guide/authentication/rbac.md

NOTE: If your cluster enables auth, rolling downgrade from 3.5 isn’t supported because 3.5 changes a format of WAL entries related to auth. You can follow the authentification instructions to disable auth, and delete all users first.

Downgrade etcd from 3.5 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

content/en/docs/v3.5/op-guide/authentication/rbac.md

Note

If your cluster enables auth, rolling downgrade from 3.5 isn’t supported because 3.5 changes a format of WAL entries related to auth. You can follow the authentification instructions to disable auth, and delete all users first.

Downgrade etcd from 3.5 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

content/en/docs/v3.5/op-guide/authentication/rbac.md

Downgrade etcd from v3.6 to v3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes
after enabling the downgrade, new features in v3.6 are no longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

Highlighted breaking changes from v3.6 to v3.5:

Difference in flags

If you are using any of the following flags in your v3.6 configurations, make sure to remove, rename, or change the default value when downgrading to v3.5.

Downgrade etcd from v3.6 to v3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, downgrading from etcd v3.6 to v3.5 can be a zero-downtime, rolling downgrade:

one by one, stop the etcd v3.6 processes and replace them with etcd v3.5 processes
after enabling the downgrade, new features in v3.6 are no longer available to the cluster

Before starting a downgrade, read through the rest of this guide to prepare.

Downgrade checklists

Highlighted breaking changes from v3.6 to v3.5:

Difference in flags

If you are using any of the following flags in your v3.6 configurations, make sure to remove, rename, or change the default value when downgrading to v3.5.

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

NOTE: When migrating from v2 with no v3 data, etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Note

When migrating from v2 with no v3 data, etcd server v3.2+ panics when etcd restores from existing snapshots but no v3 ETCD_DATA_DIR/member/snap/db file. This happens when the server had migrated from v2 with no previous v3 data. This also prevents accidental v3 data loss (e.g. db file might have been moved). etcd requires that post v3 migration can only happen with v3 data. Do not upgrade to newer v3 versions until v3.0 server contains v3 data.

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Note

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Note

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Note

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Note

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Note

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Amazon Web Services

Mon, 01 Jan 0001 00:00:00 +0000

This guide assumes operational knowledge of Amazon Web Services (AWS), specifically Amazon Elastic Compute Cloud (EC2). This guide provides an introduction to design considerations when designing an etcd deployment on AWS EC2 and how AWS specific features may be utilized in that context.

Capacity planning

As a critical building block for distributed systems it is crucial to perform adequate capacity planning in order to support the intended cluster workload. As a highly available and strongly consistent data store increasing the number of nodes in an etcd cluster will generally affect performance adversely. This makes sense intuitively, as more nodes means more members for the leader to coordinate state across. The most direct way to increase throughput and decrease latency of an etcd cluster is allocate more disk I/O, network I/O, CPU, and memory to cluster members. In the event it is impossible to temporarily divert incoming requests to the cluster, scaling the EC2 instances which comprise the etcd cluster members one at a time may improve performance. It is, however, best to avoid bottlenecks through capacity planning.

Container Linux with systemd

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster

Cluster bootstrapping in Container Linux is simplest with Ignition; coreos-metadata.service dynamically fetches the machine’s IP for discovery. Note that etcd’s discovery service protocol is only meant for bootstrapping, and cannot be used with runtime reconfiguration or cluster monitoring.

The Container Linux Config Transpiler compiles etcd configuration files into Ignition configuration files:

etcd:
 version: 3.2.0
 name: s1
 data_dir: /var/lib/etcd
 advertise_client_urls: http://{PUBLIC_IPV4}:2379
 initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
 listen_client_urls: http://0.0.0.0:2379
 listen_peer_urls: http://{PRIVATE_IPV4}:2380
 discovery: https://discovery.etcd.io/<token>

ct would produce the following Ignition Config:

FreeBSD

Mon, 01 Jan 0001 00:00:00 +0000

Starting with version 0.1.2 both etcd and etcdctl have been ported to FreeBSD and can be installed either via packages or ports system. Their versions have been recently updated to 0.2.0 so now etcd and etcdctl can be enjoyed on FreeBSD 10.0 (RC4 as of now) and 9.x, where they have been tested. They might also work when installed from ports on earlier versions of FreeBSD, but it is untested; caveat emptor.

Issue Triage Guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up issue management.

The etcd issues are listed at https://github.com/etcd-io/etcd/issues and are identified with labels. For example, an issue that is identified as a bug will eventually be set to label area/bug . New issues will start out without any labels, but typically etcd maintainers and active contributors add labels based on their findings. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels

Following are few predetermined searches on issues for convenience:

Avoiding Zombie Cluster Members When Upgrading to etcd v3.6

Wed, 17 Dec 2025 00:00:00 +0000

Summary: SIG-etcd has patched another potential issue blocking upgrades from v3.5 to v3.6. If you are upgrading, make sure to update to v3.5.26 or later first.

Issue Summary

Recently, the etcd community addressed an issue that may appear when users upgrade from v3.5 to v3.6. This bug can cause the cluster to report “zombie members”, which are etcd nodes that were removed from the database cluster some time ago, and are re-appearing and joining database consensus. The etcd cluster is then inoperable until these zombie members are removed.

Follow Up - Preventing Upgrade Failures from etcd v3.5 to v3.6

Tue, 21 Oct 2025 00:00:00 +0000

We have identified and fixed an additional scenario that may cause upgrade failures when moving from etcd v3.5 to v3.6. This post contains details, the fix, and additional workarounds. Please refer to issue 20793 to get detailed technical information.

Issue

In a previous post — How to Prevent a Common Failure when Upgrading etcd v3.5 to v3.6 — we described an upgrade issue affecting etcd versions in v3.5.1-v3.5.19. That issue was addressed in v3.5.20. However, a follow-up investigation revealed that the original fix did not cover all scenarios.

Autonomous Testing of etcd's Robustness

Fri, 03 Oct 2025 00:00:00 +0000

This is a post from the CNCF blog which we are sharing with our community as well.

As a critical component of many production systems, including Kubernetes, the etcd project’s first priority is reliability. Ensuring consistency and data safety requires our project contributors to continuously improve testing methodologies. In this article, we will describe how we used advanced simulation testing to uncover subtle bugs, validate the robustness of our releases, and increase our confidence in etcd’s stability. We’ll share our key findings and how they have improved etcd.

Announcing etcd v3.6.0

Thu, 15 May 2025 00:00:00 +0000

Introduction
Security
Features
Performance
- Memory
- Throughput
Breaking changes
Testing
Critical bug fixes
Upgrade issue
Platforms
Dependencies
Community
Extended Support for v3.4
Future Development

Introduction

Today, we are releasing etcd v3.6.0, the first minor release since etcd v3.5.0 on June 15, 2021. This release introduces several new features, makes significant progress on long-standing efforts like downgrade support and migration to v3store, and addresses numerous critical & major issues. It also includes major optimizations in memory usage, improving efficiency and performance.

How to Prevent a Common Failure when Upgrading etcd v3.5 to v3.6

Thu, 27 Mar 2025 00:00:00 +0000

Update (October 21, 2025)

We have identified and fixed two additional upgrade issues. Please see our new blog posts Follow Up - Preventing Upgrade Failures from etcd v3.5 to v3.6 and Avoiding Zombie Cluster Members When Upgrading for details.

There is a common issue 19557 in the etcd v3.5 to v3.6 upgrade that may cause the upgrade process to fail. You can find detailed information and related discussions in the issue.

TL; DR

Users are required to first upgrade to etcd v3.5.20 (or a higher patch version) before upgrading to etcd v3.6.0. Failure to do so may result in an unsuccessful upgrade.

Engage with the etcd project at KubeCon NA 2023

Sun, 05 Nov 2023 00:00:00 +0000

KubeCon NA 2023 in Chicago is just around the corner! This year, the etcd project has a diverse range of talks, tutorials, and even interactive contribfest sessions for you to get involved in . As a critical foundational pillar of the Kubernetes ecosystem, etcd’s presence at Kubecon underscores its importance in ensuring all our Kubernetes clusters continue to have robust and reliable distributed persistent state.

Here’s a detailed overview of what you can expect from the Etcd Project’s presence at KubeCon NA 2023:

Introducing sig-etcd

Wed, 01 Nov 2023 00:00:00 +0000

Special Interest Groups (SIGs) are a fundamental part of the Kubernetes project, with a substantial share of the community activity happening within them. When the need arises, new SIGs can be created, and that was precisely what happened recently.

SIG etcd is the most recent addition to the list of Kubernetes SIGs. In this article we will get to know it a bit better, understand its origins, scope, and plans.

The critical role of etcd

If we look inside the control plane of a Kubernetes cluster, we will find etcd, a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data -- this description alone highlights the critical role that etcd plays, and the importance of it within the Kubernetes ecosystem.

How to debug large db size issue?

Wed, 04 Jan 2023 00:00:00 +0000

Background

Users can configure the quota of the backend db size using flag --quota-backend-bytes. It’s the max number of bytes the etcd db file may consume, namely the ${etcd-data-dir}/member/snap/db file. Its default value is 2GB, and the suggested max value is 8GB.

2GB is usually sufficient for most use cases. If you run out of the db quota, you will see error message etcdserver: mvcc: database space exceeded when trying to write more data, and see alarm “NOSPACE” (see example below) when checking the endpoint status or health state. It would be better to figure out whether it’s expected. It’s exactly the reason why I provide this guide.

etcd Integrates Continuous Fuzzing

Fri, 11 Mar 2022 00:00:00 +0000

In the last few months, the team at Ada Logics has worked on integrating continuous fuzzing into the etcd project. This was an effort focused on improving the security posture of etcd and ensuring a continued good experience for etcds users. The fuzzing integration involved enrolling etcd in the OSS-Fuzz project and writing a set of fuzzers that would bring the test coverage of etcd up to a mature level. In total, 18 fuzzers were written, and eight bugs were found, demonstrating the work’s value for etcd both short term and long term. All fuzzers were implemented by way of go-fuzz and when running in OSS-Fuzz instrumented by way of libFuzzer, and as such, etcd uses state-of-the-art open source fuzzing capabilities. The full report of the engagement can be found here.

Announcing etcd 3.5

Tue, 15 Jun 2021 00:00:00 +0000

When we launched etcd 3.4 back in August 2019, our focus was on storage backend improvements, non-voting member and pre-vote features. Since then, etcd has become more widely used for various mission critical clustering and database applications and as a result, its feature set has grown more broad and complex. Thus, improving its stability and reliability has been top priority in recent development.

Today, we are releasing etcd 3.5. The past two years allowed for extensive iterations in fixing numerous bugs, identifying optimization opportunities at scale, and evolving its surrounding ecosystem. The etcd project also became a CNCF graduated project during this time frame. This release is the result of continuous evolution and grungy, thankless tasks done by the etcd community.

Latest Jepsen Results against etcd 3.4.3

Thu, 30 Jan 2020 00:00:00 +0000

Jepsen tested and analyzed etcd 3.4.3, and had both good results and useful feedback to share with us.

A key part of etcd’s design is strong consistency guarantees across the distributed key-value store. Kubernetes, Rook, OpenStack, and countless other critical software projects rely on etcd, in part, because of the etcd project’s focus on reliability and correctness.

Over the years, the etcd team has put tremendous effort on building testing and chaos engineering frameworks. We feel that we’ve improved our consistency, and have asked Jepsen for validation.

Container Linux with systemd

Tue, 23 May 2017 00:00:00 +0000

This is an adaptation of a page previously found in the Platforms section of the documentation which described etcd deployments on various platform services. The original page was authored by Gyuho Lee.

The following blog post shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster

Deployment of etcd on AWS EC2

Mon, 10 Apr 2017 00:00:00 +0000

This post provides an introduction to design considerations when designing an etcd deployment on AWS EC2 and how AWS specific features may be utilized in that context. Also, this post assumes operational knowledge of Amazon Web Services (AWS), specifically Amazon Elastic Compute Cloud (EC2).

FreeBSD

Thu, 23 Jan 2014 00:00:00 +0000

Mon, 01 Jan 0001 00:00:00 +0000

PR Management

Purpose

Speed up PR management.

The etcd PRs are listed at https://github.com/etcd-io/etcd/pulls A PR can have various labels, milestone, reviewer etc. The detailed list of labels can be found at https://github.com/kubernetes/kubernetes/labels

Following are few example searches on PR for convenience:

Scope

These guidelines serves as a primary document for managing PRs in etcd. Everyone is welcome to help manage PRs but the work and responsibilities discussed in this document is created with etcd maintainers and active contributors in mind.

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Data Directory

Lifecycle

When first started, etcd stores its configuration into a data directory specified by the data-dir configuration parameter. Configuration is stored in the write ahead log and includes: the local member ID, cluster ID, and initial cluster configuration. The write ahead log and snapshot files are used during member operation and to recover after a restart.

Having a dedicated disk to store wal files can improve the throughput and stabilize the cluster. It is highly recommended to dedicate a wal disk and set --wal-dir to point to a directory on that device for a production cluster deployment.

Amazon Web Services

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

Capacity planning

Amazon Web Services

Mon, 01 Jan 0001 00:00:00 +0000

Capacity planning

API reference

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Auth` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user. User name cannot be empty.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role. Role name cannot be empty.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.
MemberPromote	MemberPromoteRequest	MemberPromoteResponse	MemberPromote promotes a member from raft learner (non-voting) to raft voting member.

service `KV` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.

service `Watch` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (etcdserver/etcdserverpb/rpc.proto)

Empty field.

API reference

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Auth` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
AuthStatus	AuthStatusRequest	AuthStatusResponse	AuthStatus displays authentication status.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user. User name cannot be empty.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role. Role name cannot be empty.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.
MemberPromote	MemberPromoteRequest	MemberPromoteResponse	MemberPromote promotes a member from raft learner (non-voting) to raft voting member.

service `KV` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.
Downgrade	DowngradeRequest	DowngradeResponse	Downgrade requests downgrades, verifies feasibility or cancels downgrade on the cluster version. Supported since etcd 3.5.

service `Watch` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (api/etcdserverpb/rpc.proto)

Field	Description	Type
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (api/etcdserverpb/rpc.proto)

Empty field.

API reference

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Auth` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
AuthStatus	AuthStatusRequest	AuthStatusResponse	AuthStatus displays authentication status.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user. User name cannot be empty.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role. Role name cannot be empty.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.
MemberPromote	MemberPromoteRequest	MemberPromoteResponse	MemberPromote promotes a member from raft learner (non-voting) to raft voting member.

service `KV` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.
Downgrade	DowngradeRequest	DowngradeResponse	Downgrade requests downgrades, verifies feasibility or cancels downgrade on the cluster version. Supported since etcd 3.5.

service `Watch` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthDisableResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthEnableRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthEnableResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleAddRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the role to add to the authentication system.	string

message `AuthRoleAddResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleDeleteRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string

message `AuthRoleDeleteResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleGetRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string

message `AuthRoleGetResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
perm		(slice of) authpb.Permission

message `AuthRoleGrantPermissionRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the role which will be granted the permission.	string
perm	perm is the permission to grant to the role.	authpb.Permission

message `AuthRoleGrantPermissionResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleListRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthRoleListResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
roles		(slice of) string

message `AuthRoleRevokePermissionRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string
key		bytes
range_end		bytes

message `AuthRoleRevokePermissionResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthStatusRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthStatusResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
enabled		bool
authRevision	authRevision is the current revision of auth store	uint64

message `AuthUserAddRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
password		string
options		authpb.UserAddOptions
hashedPassword		string

message `AuthUserAddResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserChangePasswordRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the user whose password is being changed.	string
password	password is the new password for the user. Note that this field will be removed in the API layer.	string
hashedPassword	hashedPassword is the new password for the user. Note that this field will be initialized in the API layer.	string

message `AuthUserChangePasswordResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserDeleteRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the user to delete.	string

message `AuthUserDeleteResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserGetRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string

message `AuthUserGetResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
roles		(slice of) string

message `AuthUserGrantRoleRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
user	user is the name of the user which should be granted a given role.	string
role	role is the name of the role to grant to the user.	string

message `AuthUserGrantRoleResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserListRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthUserListResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
users		(slice of) string

message `AuthUserRevokeRoleRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
role		string

message `AuthUserRevokeRoleResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthenticateRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
password		string

message `AuthenticateResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
token	token is an authorized token that can be used in succeeding RPCs	string

message `CompactionRequest` (api/etcdserverpb/rpc.proto)

CompactionRequest compacts the key-value store up to a given revision. All superseded keys with a revision less than the compaction revision will be removed.

API reference

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Auth` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
AuthStatus	AuthStatusRequest	AuthStatusResponse	AuthStatus displays authentication status.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user. User name cannot be empty.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role. Role name cannot be empty.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.
MemberPromote	MemberPromoteRequest	MemberPromoteResponse	MemberPromote promotes a member from raft learner (non-voting) to raft voting member.

service `KV` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.
Downgrade	DowngradeRequest	DowngradeResponse	Downgrade requests downgrades, verifies feasibility or cancels downgrade on the cluster version. Supported since etcd 3.5.

service `Watch` (api/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthDisableResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthEnableRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthEnableResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleAddRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the role to add to the authentication system.	string

message `AuthRoleAddResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleDeleteRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string

message `AuthRoleDeleteResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleGetRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string

message `AuthRoleGetResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
perm		(slice of) authpb.Permission

message `AuthRoleGrantPermissionRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the role which will be granted the permission.	string
perm	perm is the permission to grant to the role.	authpb.Permission

message `AuthRoleGrantPermissionResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthRoleListRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthRoleListResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
roles		(slice of) string

message `AuthRoleRevokePermissionRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
role		string
key		bytes
range_end		bytes

message `AuthRoleRevokePermissionResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthStatusRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthStatusResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
enabled		bool
authRevision	authRevision is the current revision of auth store	uint64

message `AuthUserAddRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
password		string
options		authpb.UserAddOptions
hashedPassword		string

message `AuthUserAddResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserChangePasswordRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the user whose password is being changed.	string
password	password is the new password for the user. Note that this field will be removed in the API layer.	string
hashedPassword	hashedPassword is the new password for the user. Note that this field will be initialized in the API layer.	string

message `AuthUserChangePasswordResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserDeleteRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name	name is the name of the user to delete.	string

message `AuthUserDeleteResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserGetRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string

message `AuthUserGetResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
roles		(slice of) string

message `AuthUserGrantRoleRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
user	user is the name of the user which should be granted a given role.	string
role	role is the name of the role to grant to the user.	string

message `AuthUserGrantRoleResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthUserListRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option

message `AuthUserListResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
users		(slice of) string

message `AuthUserRevokeRoleRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
role		string

message `AuthUserRevokeRoleResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader

message `AuthenticateRequest` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
name		string
password		string

message `AuthenticateResponse` (api/etcdserverpb/rpc.proto)

Field	Description	Type
(versionpb.etcd_version_msg)		option
header		ResponseHeader
token	token is an authorized token that can be used in succeeding RPCs	string

message `CompactionRequest` (api/etcdserverpb/rpc.proto)

CompactionRequest compacts the key-value store up to a given revision. All superseded keys with a revision less than the compaction revision will be removed.

API reference: concurrency

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Lock` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

API reference: concurrency

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Lock` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

API reference: concurrency

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Lock` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

API reference: concurrency

Mon, 01 Jan 0001 00:00:00 +0000

This API reference is autogenerated from the named .proto files.

service `Lock` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

Authentication

Mon, 01 Jan 0001 00:00:00 +0000

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Note:

This is just a stub which needs to be filled and updated with more information on authentication. The text above is just a code example.

Authentication

Mon, 01 Jan 0001 00:00:00 +0000

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Note:

This is just a stub which needs to be filled and updated with more information on authentication. The text above is just a code example.

Authentication

Mon, 01 Jan 0001 00:00:00 +0000

auth,user,role for authentication:

export ETCDCTL_API=3
ENDPOINTS=localhost:2379

etcdctl --endpoints=${ENDPOINTS} role add root
etcdctl --endpoints=${ENDPOINTS} role get root

etcdctl --endpoints=${ENDPOINTS} user add root
etcdctl --endpoints=${ENDPOINTS} user grant-role root root
etcdctl --endpoints=${ENDPOINTS} user get root

etcdctl --endpoints=${ENDPOINTS} role add role0
etcdctl --endpoints=${ENDPOINTS} role grant-permission role0 readwrite foo
etcdctl --endpoints=${ENDPOINTS} user add user0
etcdctl --endpoints=${ENDPOINTS} user grant-role user0 role0

etcdctl --endpoints=${ENDPOINTS} auth enable
# now all client requests go through auth

etcdctl --endpoints=${ENDPOINTS} --user=user0:123 put foo bar
etcdctl --endpoints=${ENDPOINTS} get foo
# permission denied, user name is empty because the request does not issue an authentication request
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo
# user0 can read the key foo
etcdctl --endpoints=${ENDPOINTS} --user=user0:123 get foo1

Note:

This is just a stub which needs to be filled and updated with more information on authentication. The text above is just a code example.

Authentication Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Authentication -- having users and roles in etcd -- was added in etcd 2.1. This guide will help you set up basic authentication in etcd.

etcd before 2.1 was a completely open system; anyone with access to the API could change keys. In order to preserve backward compatibility and upgradability, this feature is off by default.

For a full discussion of the RESTful API, see the authentication API documentation

Special Users and Roles

There is one special user, root, and there are two special roles, root and guest.

Authentication Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Special users and roles

There is one special user, root, and one special role, root.

User `root`

Backward Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

The main goal of etcd 2.0 release is to improve cluster safety around bootstrapping and dynamic reconfiguration. To do this, we deprecated the old error-prone APIs and provide a new set of APIs.

The other main focus of this release was a more reliable Raft implementation, but as this change is internal it should not have any notable effects to users.

Command Line Flags Changes

The major flag changes are to mostly related to bootstrapping. The initial-* flags provide an improved way to specify the required criteria to start the cluster. The advertised URLs now support a list of values instead of a single value, which allows etcd users to gracefully migrate to the new set of IANA-assigned ports (2379/client and 2380/peers) while maintaining backward compatibility with the old ports.

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

Benchmarking etcd v2.1.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Bootstrap another machine, outside of the etcd cluster, and run the boom HTTP benchmark tool with a connection reuse patch to send requests to each etcd cluster member. See the benchmark instructions for the patch and the steps to reproduce our procedures.

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Benchmarking etcd v2.2.0

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0-rc

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

Benchmarking etcd v2.2.0-rc-memory

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
For most cases, the etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

Benchmarking etcd v3

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Benchmarking etcd v3-demo

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

Client feature matrix

Mon, 01 Jan 0001 00:00:00 +0000

Features

Feature	`clientv3-grpc1.14`	`jetcd v0.0.2`
Automatic retry	Yes	.
Retry backoff	Yes	.
Automatic failover	Yes	.
Load balancer	Round-Robin	·
`WithRequireLeader(context.Context)`	Yes	.
`TLS`	Yes	Yes
`SetEndpoints`	Yes	.
`Sync` endpoints	Yes	.
`AutoSyncInterval`	Yes	.
`KeepAlive` ping	Yes	.
`MaxCallSendMsgSize`	Yes	.
`MaxCallRecvMsgSize`	Yes	.
`RejectOldCluster`	Yes	.

KV

Feature	`clientv3-grpc1.14`	`jetcd v0.0.2`
`Put`	Yes	.
`Get`	Yes	.
`Delete`	Yes	.
`Compact`	Yes	.
`Do(Op)`	Yes	.
`Txn`	Yes	.

For details, see the KV API reference.

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Starting an etcd cluster statically requires that each member knows another in the cluster. In a number of cases, you might not know the IPs of your cluster members ahead of time. In these cases, you can bootstrap an etcd cluster with the help of a discovery service.

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Clustering Guide

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Configuration flags

Mon, 01 Jan 0001 00:00:00 +0000

etcd is configurable through command-line flags and environment variables. Options set on the command line take precedence over those from the environment.

The format of environment variable for flag --my-flag is ETCD_MY_FLAG. It applies to all flags.

The official etcd ports are 2379 for client requests and 2380 for peer communication. The etcd ports can be set to accept TLS traffic, non-TLS traffic, or both TLS and non-TLS traffic.

To start etcd automatically using custom settings at startup in Linux, using a systemd unit is highly recommended.

Configuration flags

Mon, 01 Jan 0001 00:00:00 +0000

etcd is configurable through command-line flags and environment variables. Options set on the command line take precedence over those from the environment.

The format of environment variable for flag --my-flag is ETCD_MY_FLAG. It applies to all flags.

The official etcd ports are 2379 for client requests and 2380 for peer communication. The etcd ports can be set to accept TLS traffic, non-TLS traffic, or both TLS and non-TLS traffic.

To start etcd automatically using custom settings at startup in Linux, using a systemd unit is highly recommended.

Configuration flags

Mon, 01 Jan 0001 00:00:00 +0000

etcd is configurable through a configuration file, various command-line flags, and environment variables.

A reusable configuration file is a YAML file made with name and value of one or more command-line flags described below. In order to use this file, specify the file path as a value to the --config-file flag. The sample configuration file can be used as a starting point to create a new configuration file as needed.

Options set on the command line take precedence over those from the environment. If a configuration file is provided, other command line flags and environment variables will be ignored. For example, etcd --config-file etcd.conf.yml.sample --data-dir /tmp will ignore the --data-dir flag.

Configuration Flags

Mon, 01 Jan 0001 00:00:00 +0000

etcd is configurable through command-line flags and environment variables. Options set on the command line take precedence over those from the environment.

The format of environment variable for flag --my-flag is ETCD_MY_FLAG. It applies to all flags.

The official etcd ports are 2379 for client requests, and 2380 for peer communication. Some legacy code and documentation still references ports 4001 and 7001, but all new etcd use and discussion should adopt the assigned ports.

Container Linux with systemd

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster

The Container Linux Config Transpiler compiles etcd configuration files into Ignition configuration files:

etcd:
 version: 3.2.0
 name: s1
 data_dir: /var/lib/etcd
 advertise_client_urls: http://{PUBLIC_IPV4}:2379
 initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
 listen_client_urls: http://0.0.0.0:2379
 listen_peer_urls: http://{PRIVATE_IPV4}:2380
 discovery: https://discovery.etcd.io/<token>

ct would produce the following Ignition Config:

Data model

Mon, 01 Jan 0001 00:00:00 +0000

etcd stores data in a multiversion persistent key-value store. The persistent key-value store preserves the previous version of a key-value pair when its value is superseded with new data. The key-value store is effectively immutable; its operations do not update the structure in-place, but instead always generates a new updated structure. All past versions of keys are still accessible and watchable after modification. To prevent the data store from growing indefinitely over time from maintaining old versions, the store may be compacted to shed the oldest versions of superseded data.

Data model

Mon, 01 Jan 0001 00:00:00 +0000

etcd stores data in a multiversion persistent key-value store. The persistent key-value store preserves the previous version of a key-value pair when its value is superseded with new data. The key-value store is effectively immutable; its operations do not update the structure in-place, but instead always generates a new updated structure. All past versions of keys are still accessible and watchable after modification. To prevent the data store from growing indefinitely over time from maintaining old versions, the store may be compacted to shed the oldest versions of superseded data.

Data model

Mon, 01 Jan 0001 00:00:00 +0000

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform cluster of new configuration and then start the new member.

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform cluster of new configuration and then start the new member.

Design of runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two phase config changes keep the cluster safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member, first inform the cluster of the new configuration and then start the new member.

Design of Runtime Reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

Runtime reconfiguration is one of the hardest and most error prone features in a distributed system, especially in a consensus based system like etcd.

Read on to learn about the design of etcd’s runtime reconfiguration commands and how we tackled these problems.

Two Phase Config Changes Keep you Safe

In etcd, every runtime reconfiguration has to go through two phases for safety reasons. For example, to add a member you need to first inform cluster of new configuration and then start the new member.

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

Disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery Service Protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

Error Code

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the error code used in key space ‘/v2/keys’. Feel free to import ‘https://github.com/etcd-io/etcd/tree/v2.3.8/error' to use.

It’s categorized into four groups:

Command Related Error

name	code	strerror
EcodeKeyNotFound	100	“Key not found”
EcodeTestFailed	101	“Compare failed”
EcodeNotFile	102	“Not a file”
EcodeNotDir	104	“Not a directory”
EcodeNodeExist	105	“Key already exists”
EcodeRootROnly	107	“Root is read only”
EcodeDirNotEmpty	108	“Directory not empty”

Post Form Related Error

name	code	strerror
EcodePrevValueRequired	201	“PrevValue is Required in POST form”
EcodeTTLNaN	202	“The given TTL in POST form is not a number”
EcodeIndexNaN	203	“The given index in POST form is not a number”
EcodeInvalidField	209	“Invalid field”
EcodeInvalidForm	210	“Invalid POST form”

Raft Related Error

name	code	strerror
EcodeRaftInternal	300	“Raft Internal Error”
EcodeLeaderElect	301	“During Leader Election”

etcd Related Error

name	code	strerror
EcodeWatcherCleared	400	“watcher is cleared due to etcd recovery”
EcodeEventIndexCleared	401	“The event in requested index is outdated and cleared”

etcd 2.1.0-alpha benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the boom HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

etcd 2.2.0-rc benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

etcd 2.2.0-rc memory benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
At most cases, etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, you need to decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

etcd 3 demo benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

etcd API

Mon, 01 Jan 0001 00:00:00 +0000

Running a Single Machine Cluster

These examples will use a single member cluster to show you the basics of the etcd REST API. Let’s start etcd:

./bin/etcd

This will bring up etcd listening on the IANA assigned ports and listening on localhost. The IANA assigned ports for etcd are 2379 for client communication and 2380 for server-to-server communication.

Getting the etcd version

The etcd version of a specific instance can be obtained from the /version endpoint.

etcd API reference

Mon, 01 Jan 0001 00:00:00 +0000

etcd API Reference

This is a generated documentation. Please read the proto files for more.

service `Auth` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.

service `KV` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.

service `Maintenance` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash returns the hash of the local KV state for consistency checking purpose. This is designed for testing; do not use this in production when there are ongoing transactions.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.

service `Watch` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (etcdserver/etcdserverpb/rpc.proto)

Empty field.

etcd API Reference

Mon, 01 Jan 0001 00:00:00 +0000

This is a generated documentation. Please read the proto files for more.

service `Auth` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.

service `KV` (etcdserver/etcdserverpb/rpc.proto)

for grpc-gateway

etcd API Reference

Mon, 01 Jan 0001 00:00:00 +0000

This is a generated documentation. Please read the proto files for more.

service `Auth` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.

service `KV` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.

service `Watch` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (etcdserver/etcdserverpb/rpc.proto)

Empty field.

etcd concurrency API Reference

Mon, 01 Jan 0001 00:00:00 +0000

etcd concurrency API Reference

This is a generated documentation. Please read the proto files for more.

service `Lock` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquistion; locking twice with the same lease is a no-op.	int64

message `LockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

etcd concurrency API Reference

Mon, 01 Jan 0001 00:00:00 +0000

This is a generated documentation. Please read the proto files for more.

service `Lock` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses.

The gateway supports multiple etcd server endpoints. When the gateway starts, it randomly picks one etcd server endpoint and forwards all requests to that endpoint. This endpoint serves all requests until the gateway detects a network failure. If the gateway detects an endpoint failure, it will switch to a different endpoint, if available, to hide failures from its clients. Other retry policies, such as weighted round-robin, may be supported in the future.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses.

etcd gateway

Mon, 01 Jan 0001 00:00:00 +0000

What is etcd gateway

etcd gateway is a simple TCP proxy that forwards network data to the etcd cluster. The gateway is stateless and transparent; it neither inspects client requests nor interferes with cluster responses.

etcd v2.1.0-alpha benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.1.0 alpha

etcd Cluster

3 etcd members, each runs on a single machine

Testing

Bootstrap another machine and use the hey HTTP benchmark tool to send requests to each etcd member. Check the benchmark hacking guide for detailed instructions.

Performance

reading one single key

key size in bytes	number of clients	target etcd server	read QPS	90th Percentile Latency (ms)
64	1	leader only	1534	0.7
64	64	leader only	10125	9.1
64	256	leader only	13892	27.1
256	1	leader only	1530	0.8
256	64	leader only	10106	10.1
256	256	leader only	14667	27.0
64	64	all servers	24200	3.9
64	256	all servers	33300	11.8
256	64	all servers	24800	3.9
256	256	all servers	33000	11.5

writing one single key

key size in bytes	number of clients	target etcd server	write QPS	90th Percentile Latency (ms)
64	1	leader only	60	21.4
64	64	leader only	1742	46.8
64	256	leader only	3982	90.5
256	1	leader only	58	20.3
256	64	leader only	1770	47.8
256	256	leader only	4157	105.3
64	64	all servers	1028	123.4
64	256	all servers	3260	123.8
256	64	all servers	1033	121.5
256	256	all servers	3061	119.3

etcd v2.2.0 benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical Machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted as etcd data directory
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0 members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0
Git SHA: e4561dd
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

etcd v2.2.0-rc benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs

etcd Cluster

3 etcd 2.2.0-rc members, each runs on a single machine.

Detailed versions:

etcd Version: 2.2.0-alpha.1+git
Git SHA: 59a5a7e
Go Version: go1.4.2
Go OS/Arch: linux/amd64

Also, we use 3 etcd 2.1.0 alpha-stage members to form cluster to get base performance. etcd’s commit head is at c7146bd5, which is the same as the one that we use in etcd 2.1 benchmark.

etcd v2.2.0-rc-memory benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machine

GCE n1-standard-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
7.5 GB memory
2x CPUs

etcd

etcd Version: 2.2.0-rc.0+git
Git SHA: 103cb5c
Go Version: go1.5
Go OS/Arch: linux/amd64

Testing

Start 3-member etcd cluster, each of which uses 2 cores.

The length of key name is always 64 bytes, which is a reasonable length of average key bytes.

Memory Maximal Usage

etcd may use maximal memory if one follower is dead and the leader keeps sending snapshots.
max RSS is the maximal memory usage recorded in 3 runs.

value bytes	key number	data size(MB)	max RSS(MB)	max RSS/data rate on leader
128	50000	6	433	72x
128	100000	12	659	54x
128	200000	24	1466	61x
1024	50000	48	1253	26x
1024	100000	96	2344	24x
1024	200000	192	4361	22x

Data Size Threshold

When etcd reaches data size threshold, it may trigger leader election easily and drop part of proposals.
At most cases, etcd cluster should work smoothly if it doesn’t hit the threshold. If it doesn’t work well due to insufficient resources, you need to decrease its data size.

value bytes	key number limitation	suggested data size threshold(MB)	consumed RSS(MB)
128	400K	48	2400
1024	300K	292	6500

etcd v3 API

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The etcd v3 API is designed to give users a more efficient and cleaner abstraction compared to etcd v2. There are a number of semantic and protocol changes in this new API.

To prove out the design of the v3 API the team has also built a number of example recipes, there is a video discussing these recipes too.

Design

Flatten binary key-value space
Keep the event history until compaction
- access to old version of keys
- user controlled history compaction
Support range query

etcd v3 API

Mon, 01 Jan 0001 00:00:00 +0000

The etcd v3 API is designed to give users a more efficient and cleaner abstraction compared to etcd v2. There are a number of semantic and protocol changes in this new API.

To prove out the design of the v3 API the team has also built a number of example recipes, there is a video discussing these recipes too.

Design

Flatten binary key-value space
Keep the event history until compaction
- access to old version of keys
- user controlled history compaction
Support range query

etcd v3 API

Mon, 01 Jan 0001 00:00:00 +0000

The etcd v3 API is designed to give users a more efficient and cleaner abstraction compared to etcd v2. There are a number of semantic and protocol changes in this new API.

To prove out the design of the v3 API the team has also built a number of example recipes, there is a video discussing these recipes too.

Design

Flatten binary key-value space
Keep the event history until compaction
- access to old version of keys
- user controlled history compaction
Support range query

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

etcd v3 authentication design

Mon, 01 Jan 0001 00:00:00 +0000

Why not reuse the v2 auth system?

etcd v3-demo benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Physical machines

GCE n1-highcpu-2 machine type

1x dedicated local SSD mounted under /var/lib/etcd
1x dedicated slow disk for the OS
1.8 GB memory
2x CPUs
etcd version 2.2.0

etcd Cluster

1 etcd member running in v3 demo mode

Testing

Use etcd v3 benchmark tool.

Performance

reading one single key

key size in bytes	number of clients	read QPS	90th Percentile Latency (ms)
256	1	2716	0.4
256	64	16623	6.1
256	256	16622	21.7

The performance is nearly the same as the one with empty server handler.

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

The name “etcd” originated from two ideas, the unix “/etc” folder and “d"istibuted systems. The “/etc” folder is a place to store configuration data for a single system whereas etcd stores configuration information for large scale distributed systems. Hence, a “d"istributed “/etc” is “etcd”.

etcd versus other key-value stores

Mon, 01 Jan 0001 00:00:00 +0000

etcd3 API

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: this doc is not finished!

Response header

All Responses from etcd API have a response header attached. The response header includes the metadata of the response.

message ResponseHeader {
 uint64 cluster_id = 1;
 uint64 member_id = 2;
 int64 revision = 3;
 uint64 raft_term = 4;
}

Cluster_ID - the ID of the cluster that generates the response
Member_ID - the ID of the member that generates the response
Revision - the revision of the key-value store when the response is generated
Raft_Term - the Raft term of the member when the response is generated

An application may read the Cluster_ID (Member_ID) field to ensure it is communicating with the intended cluster (member).

etcd3 API

Mon, 01 Jan 0001 00:00:00 +0000

etcd3 API

Mon, 01 Jan 0001 00:00:00 +0000

Experimental APIs and features

Mon, 01 Jan 0001 00:00:00 +0000

For the most part, the etcd project is stable, but we are still moving fast! We believe in the release fast philosophy. We want to get early feedback on features still in development and stabilizing. Thus, there are, and will be more, experimental features and APIs. We plan to improve these features based on the early feedback from the community, or abandon them if there is little interest, in the next few releases. Please do not rely on any experimental features or APIs in production environment.

Experimental APIs and features

Mon, 01 Jan 0001 00:00:00 +0000

Experimental APIs and features

Mon, 01 Jan 0001 00:00:00 +0000

Failure modes

Mon, 01 Jan 0001 00:00:00 +0000

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

1) Why can an etcd client read an old version of data when a majority of the etcd cluster members are down?

In situations where a client connects to a minority, etcd favors by default availability over consistency. This means that even though data might be “out of date”, it is still better to return something versus nothing.

In order to confirm that a read is up to date with a majority of the cluster, the client can use the quorum=true parameter on reads of keys. This means that a majority of the cluster is checked on reads before returning the data, otherwise the read will timeout and fail.

FreeBS

Mon, 01 Jan 0001 00:00:00 +0000

Starting with version 0.1.2 both etcd and etcdctl have been ported to FreeBSD and can be installed either via packages or ports system. Their versions have been recently updated to 0.2.0 so now you can enjoy using etcd and etcdctl on FreeBSD 10.0 (RC4 as of now) and 9.x where they have been tested. They might also work when installed from ports on earlier versions of FreeBSD, but your mileage may vary.

FreeBSD

Mon, 01 Jan 0001 00:00:00 +0000

Starting with version 0.1.2 both etcd and etcdctl have been ported to FreeBSD and can be installed either via packages or ports system. Their versions have been recently updated to 0.2.0 so now you can enjoy using etcd and etcdctl on FreeBSD 10.0 (RC4 as of now) and 9.x where they have been tested. They might also work when installed from ports on earlier versions of FreeBSD, but your mileage may vary.

FreeBSD

Mon, 01 Jan 0001 00:00:00 +0000

FreeBSD

Mon, 01 Jan 0001 00:00:00 +0000

Frequently Asked Questions (FAQ)

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

Do clients have to send requests to the etcd leader?

Configuration

What is the difference between advertise-urls and listen-urls?

listen-urls specifies the local addresses etcd server binds to for accepting incoming connections. To listen on a port for all interfaces, specify 0.0.0.0 as the listen IP address.

Frequently Asked Questions (FAQ)

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

Do clients have to send requests to the etcd leader?

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

Frequently Asked Questions (FAQ)

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

Do clients have to send requests to the etcd leader?

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Node

Node is an instance of raft state machine.

It has a unique identification, and records other nodes’ progress internally when it is the leader.

Member

Member is an instance of etcd. It hosts a node, and provides service to clients.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

Glossary

Mon, 01 Jan 0001 00:00:00 +0000

This document defines the various terms used in etcd documentation, command line and source code.

Alarm

The etcd server raises an alarm whenever the cluster needs operator intervention to remain reliable.

Authentication

Authentication manages user access permissions for etcd resources.

Client

A client connects to the etcd cluster to issue service requests such as fetching key-value pairs, writing data, or watching for updates.

Cluster

Cluster consists of several members.

The node in each member follows raft consensus protocol to replicate logs. Cluster receives proposals from members, commits them and apply to local store.

gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

etcd v3 uses gRPC for its messaging protocol. The etcd project includes a gRPC-based Go client and a command line utility, etcdctl, for communicating with an etcd cluster through gRPC. For languages with no gRPC support, etcd provides a JSON grpc-gateway. This gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Using grpc-gateway

The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON.

gRPC Gateway

Mon, 01 Jan 0001 00:00:00 +0000

etcd v3 uses gRPC for its messaging protocol. The etcd project includes a gRPC-based Go client and a command line utility, etcdctl, for communicating with an etcd cluster through gRPC. For languages with no gRPC support, etcd provides a JSON grpc-gateway. This gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Using grpc-gateway

The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON.

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client and given a target for resolution:

import (
	"github.com/coreos/etcd/clientv3"
	etcdnaming "github.com/coreos/etcd/clientv3/naming"

	"google.golang.org/grpc"
)

...

cli, cerr := clientv3.NewFromURL("http://localhost:2379")
r := &etcdnaming.GRPCResolver{Client: cli}
b := grpc.RoundRobin(r)
conn, gerr := grpc.Dial("my-service", grpc.WithBalancer(b))

Managing service endpoints

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client and given a target for resolution:

import (
	"github.com/coreos/etcd/clientv3"
	etcdnaming "github.com/coreos/etcd/clientv3/naming"

	"google.golang.org/grpc"
)

...

cli, cerr := clientv3.NewFromURL("http://localhost:2379")
r := &etcdnaming.GRPCResolver{Client: cli}
b := grpc.RoundRobin(r)
conn, gerr := grpc.Dial("my-service", grpc.WithBalancer(b))

Managing service endpoints

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client and given a target for resolution:

import (
	"go.etcd.io/etcd/clientv3"
	etcdnaming "go.etcd.io/etcd/clientv3/naming"

	"google.golang.org/grpc"
)

...

cli, cerr := clientv3.NewFromURL("http://localhost:2379")
r := &etcdnaming.GRPCResolver{Client: cli}
b := grpc.RoundRobin(r)
conn, gerr := grpc.Dial("my-service", grpc.WithBalancer(b), grpc.WithBlock(), ...)

Managing service endpoints

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

This is an alpha feature, we are looking for early feedback.

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

gRPC proxy

Mon, 01 Jan 0001 00:00:00 +0000

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

Hardware recommendations

Mon, 01 Jan 0001 00:00:00 +0000

Install

Mon, 01 Jan 0001 00:00:00 +0000

System requirements

The etcd performance benchmarks run etcd on 8 vCPU, 16GB RAM, 50GB SSD GCE instances, but any relatively modern machine with low latency storage and a few gigabytes of memory should suffice for most use cases. Applications with large v2 data stores will require more memory than a large v3 data store since data is kept in anonymous memory instead of memory mapped from a file. than For running etcd on a cloud provider, we suggest at least a medium instance on AWS or a standard-1 instance on GCE.

Install

Mon, 01 Jan 0001 00:00:00 +0000

System requirements

The etcd performance benchmarks run etcd on 8 vCPU, 16GB RAM, 50GB SSD GCE instances, but any relatively modern machine with low latency storage and a few gigabytes of memory should suffice for most use cases. Applications with large v2 data stores will require more memory than a large v3 data store since data is kept in anonymous memory instead of memory mapped from a file. For running etcd on a cloud provider, we suggest at least a medium instance on AWS or a standard-1 instance on GCE.

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

By default, etcdctl talks to the etcd server with the v2 API for backward compatibility. For etcdctl to speak to etcd using the v3 API, the API version must be set to version 3 via the ETCDCTL_API environment variable.

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Issue triage guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up issue management.

Following are few predetermined searches on issues for convenience:

Issue triage guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up issue management.

Following are few predetermined searches on issues for convenience:

Issue triage guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up issue management.

Following are few predetermined searches on issues for convenience:

KV API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

etcd is a consistent and durable key value store with mini-transaction(TODO: link to txn doc when we have it) support. The key value store is exposed through the KV APIs. etcd tries to ensure the strongest consistency and durability guarantees for a distributed system. This specification enumerates the KV API guarantees made by etcd.

APIs to consider

Read APIs
- range
- watch
Write APIs
- put
- delete
Combination (read-modify-write) APIs
- txn

etcd specific definitions

Operation completed

An etcd operation is considered complete when it is committed through consensus, and therefore “executed” -- permanently stored -- by the etcd storage engine. The client knows an operation is completed when it receives a response from the etcd server. Note that the client may be uncertain about the status of an operation if it times out, or there is a network disruption between the client and the etcd member. etcd may also abort operations when there is a leader election. etcd does not send abort responses to clients’ outstanding requests in this event.

KV API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

etcd is a consistent and durable key value store with mini-transaction support. The key value store is exposed through the KV APIs. etcd tries to ensure the strongest consistency and durability guarantees for a distributed system. This specification enumerates the KV API guarantees made by etcd.

APIs to consider

Read APIs
- range
- watch
Write APIs
- put
- delete
Combination (read-modify-write) APIs
- txn

etcd specific definitions

Operation completed

KV API guarantees

Mon, 01 Jan 0001 00:00:00 +0000

APIs to consider

Read APIs
- range
- watch
Write APIs
- put
- delete
Combination (read-modify-write) APIs
- txn

etcd specific definitions

Operation completed

Learner

Mon, 01 Jan 0001 00:00:00 +0000

Background

Membership reconfiguration has been one of the biggest operational challenges. Let’s review common challenges.

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2
etcdloadtest - A command line load test client for etcd version 3.0 and above.

Go libraries

Libraries and Tools

Mon, 01 Jan 0001 00:00:00 +0000

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2

Go libraries

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the capnslog library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
- Examples:
  - A failure to allocate disk space for WAL
Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the capnslog library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
- Examples:
  - A failure to allocate disk space for WAL
Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the capnslog library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
- Examples:
  - A failure to allocate disk space for WAL
Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Members API

Mon, 01 Jan 0001 00:00:00 +0000

List members

Return an HTTP 200 OK response code and a representation of all members in the etcd cluster.

Request

GET /v2/members HTTP/1.1

Example

curl http://10.0.0.10:2379/v2/members

{
 "members": [
 {
 "id": "272e204152",
 "name": "infra1",
 "peerURLs": [
 "http://10.0.0.10:2380"
 ],
 "clientURLs": [
 "http://10.0.0.10:2379"
 ]
 },
 {
 "id": "2225373f43",
 "name": "infra2",
 "peerURLs": [
 "http://10.0.0.11:2380"
 ],
 "clientURLs": [
 "http://10.0.0.11:2379"
 ]
 },
 ]
}

Add a member

Returns an HTTP 201 response code and the representation of added member with a newly generated a memberID when successful. Returns a string describing the failure condition when unsuccessful.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

Migrate applications from using API v2 to API v3

Mon, 01 Jan 0001 00:00:00 +0000

Migrate applications from using API v2 to API v3

Mon, 01 Jan 0001 00:00:00 +0000

Migrate applications from using API v2 to API v3

Mon, 01 Jan 0001 00:00:00 +0000

Miscellaneous APIs

Mon, 01 Jan 0001 00:00:00 +0000

Getting the etcd version

The etcd version of a specific instance can be obtained from the /version endpoint.

curl -L http://127.0.0.1:2379/version

etcd 2.0.12

Checking health of an etcd member node

etcd provides a /health endpoint to verify the health of a particular member.

curl http://10.0.0.10:2379/health

{"health":"true"}

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server exports metrics under the /metrics path on its client port.

The metrics can be fetched with curl:

$ curl -L http://localhost:2379/metrics

# HELP etcd_debugging_mvcc_keys_total Total number of keys.
# TYPE etcd_debugging_mvcc_keys_total gauge
etcd_debugging_mvcc_keys_total 0
# HELP etcd_debugging_mvcc_pending_events_total Total number of pending events to be sent.
# TYPE etcd_debugging_mvcc_pending_events_total gauge
etcd_debugging_mvcc_pending_events_total 0
...

Prometheus

Running a Prometheus monitoring service is the easiest way to ingest and record etcd’s metrics.

First, install Prometheus:

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

Monitoring etcd

Mon, 01 Jan 0001 00:00:00 +0000

Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.

Debug endpoint

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

Performance

Mon, 01 Jan 0001 00:00:00 +0000

Understanding performance

PR management

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up PR management.

Following are few example searches on PR for convenience:

Scope

PR management

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up PR management.

Following are few example searches on PR for convenience:

Scope

PR management

Mon, 01 Jan 0001 00:00:00 +0000

Purpose

Speed up PR management.

Following are few example searches on PR for convenience:

Scope

Production users

Mon, 01 Jan 0001 00:00:00 +0000

This document tracks people and use cases for etcd in production. By creating a list of production use cases we hope to build a community of advisors that we can reach out to with experience using various etcd applications, operation environments, and cluster sizes. The etcd development team may reach out periodically to check-in on how etcd is working in the field and update this list.

discovery.etcd.io

Application: https://github.com/etcd-io/discovery.etcd.io
Launched: Feb. 2014
Cluster Size: 5 members, 5 discovery proxies
Order of Data Size: 100s of Megabytes
Operator: CoreOS, brandon.philips@coreos.com
Environment: AWS
Backups: Periodic async to S3

discovery.etcd.io is the longest continuously running etcd backed service that we know about. It is the basis of automatic cluster bootstrap and was launched in Feb. 2014: https://coreos.com/blog/etcd-0.3.0-released/.

Production users

Mon, 01 Jan 0001 00:00:00 +0000

All Kubernetes Users

Application: https://kubernetes.io/
Environments: AWS, OpenStack, Azure, Google Cloud, Huawei Cloud, Bare Metal, etc

This is a meta user; please feel free to document specific Kubernetes clusters!

Production users

Mon, 01 Jan 0001 00:00:00 +0000

All Kubernetes Users

Application: https://kubernetes.io/
Environments: AWS, OpenStack, Azure, Google Cloud, Huawei Cloud, Bare Metal, etc

This is a meta user; please feel free to document specific Kubernetes clusters!

Production Users

Mon, 01 Jan 0001 00:00:00 +0000

discovery.etcd.io

Application: https://github.com/etcd-io/discovery.etcd.io
Launched: Feb. 2014
Cluster Size: 5 members, 5 discovery proxies
Order of Data Size: 100s of Megabytes
Operator: CoreOS, brandon.philips@coreos.com
Environment: AWS
Backups: Periodic async to S3

Proxy

Mon, 01 Jan 0001 00:00:00 +0000

etcd can run as a transparent proxy. Doing so allows for easy discovery of etcd within your infrastructure, since it can run on each machine as a local service. In this mode, etcd acts as a reverse proxy and forwards client requests to an active etcd cluster. The etcd proxy does not participate in the consensus replication of the etcd cluster, thus it neither increases the resilience nor decreases the write performance of the etcd cluster.

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Reporting Bugs

Mon, 01 Jan 0001 00:00:00 +0000

If you find bugs or documentation mistakes in the etcd project, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make your bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. You can also attach etcd log (the starting log with etcd configuration is especially important).

Role-based access control

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Special users and roles

There is one special user, root, and one special role, root.

User `root`

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with rkt and Docker using the static bootstrap process.

rkt

Running a single node etcd

The following rkt run command will expose the etcd client API on port 2379 and expose the peer API on port 2380.

Use the host IP address when configuring etcd.

export NODE1=192.168.1.21

Trust the CoreOS App Signing Key.

sudo rkt trust --prefix coreos.com/etcd
# gpg key fingerprint is: 18AD 5014 C99E F7E3 BA5F 6CE9 50BD D3E0 FC8A 365E

Run the v3.0.6 version of etcd or specify another release version.

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with rkt and Docker using the static bootstrap process.

rkt

Running a single node etcd

The following rkt run command will expose the etcd client API on port 2379 and expose the peer API on port 2380.

Use the host IP address when configuring etcd.

export NODE1=192.168.1.21

Trust the CoreOS App Signing Key.

sudo rkt trust --prefix coreos.com/etcd
# gpg key fingerprint is: 18AD 5014 C99E F7E3 BA5F 6CE9 50BD D3E0 FC8A 365E

Run the v3.1.2 version of etcd or specify another release version.

Run etcd clusters inside containers

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with rkt and Docker using the static bootstrap process.

rkt

Running a single node etcd

The following rkt run command will expose the etcd client API on port 2379 and expose the peer API on port 2380.

Use the host IP address when configuring etcd.

export NODE1=192.168.1.21

Trust the CoreOS App Signing Key.

sudo rkt trust --prefix quay.io/coreos/etcd
# gpg key fingerprint is: 18AD 5014 C99E F7E3 BA5F 6CE9 50BD D3E0 FC8A 365E

Run the v3.2 version of etcd or specify another release version.

Run etcd on Container Linux with systemd

Mon, 01 Jan 0001 00:00:00 +0000

The following guide shows how to run etcd with systemd under Container Linux.

Provisioning an etcd cluster

The Container Linux Config Transpiler compiles etcd configuration files into Ignition configuration files:

etcd:
 version: 3.2.0
 name: s1
 data_dir: /var/lib/etcd
 advertise_client_urls: http://{PUBLIC_IPV4}:2379
 initial_advertise_peer_urls: http://{PRIVATE_IPV4}:2380
 listen_client_urls: http://0.0.0.0:2379
 listen_peer_urls: http://{PRIVATE_IPV4}:2380
 discovery: https://discovery.etcd.io/<token>

ct would produce the following Ignition Config:

Running etcd under Docker

Mon, 01 Jan 0001 00:00:00 +0000

The following guide will show you how to run etcd under Docker using the static bootstrap process.

Running etcd in standalone mode

In order to expose the etcd API to clients outside of the Docker host you’ll need use the host IP address when configuring etcd.

export HostIP="192.168.12.50"

The following docker run command will expose the etcd client API over ports 4001 and 2379, and expose the peer port over 2380.

This will run the latest release version of etcd. You can specify version if needed (e.g. quay.io/coreos/etcd:v2.2.0).

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Reconfiguration requests can only be processed when the majority of the cluster members are functioning. It is highly recommended to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not able to make progress and need to restart from majority failure.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Reconfiguration requests can only be processed when a majority of cluster members are functioning. It is highly recommended to always have a cluster size greater than two in production. It is unsafe to remove a member from a two member cluster. The majority of a two member cluster is also two. If there is a failure during the removal process, the cluster might not able to make progress and need to restart from majority failure.

Runtime reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Runtime Reconfiguration

Mon, 01 Jan 0001 00:00:00 +0000

etcd comes with support for incremental runtime reconfiguration, which allows users to update the membership of the cluster at run time.

Security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication.

To get up and running, first have a CA certificate and a signed key pair for one member. It is recommended to create and sign a new key pair for every member in a cluster.

For convenience, the cfssl tool provides an easy interface to certificate generation, and we provide an example using the tool here. Alternatively, try this guide to generating self-signed key pairs.

Security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication.

To get up and running, first have a CA certificate and a signed key pair for one member. It is recommended to create and sign a new key pair for every member in a cluster.

For convenience, the cfssl tool provides an easy interface to certificate generation, and we provide an example using the tool here. Alternatively, try this guide to generating self-signed key pairs.

Security Model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports SSL/TLS as well as authentication through client certificates, both for clients to server as well as peer (server to server / cluster) communication.

To get up and running you first need to have a CA certificate and a signed key pair for one member. It is recommended to create and sign a new key pair for every member in a cluster.

For convenience, the cfssl tool provides an easy interface to certificate generation, and we provide an example using the tool here. You can also examine this alternative guide to generating self-signed key pairs.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to set up a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Deploying an etcd cluster as a standalone cluster is straightforward. Start it with just one command:

$ ./etcd
...

The started etcd member listens on localhost:2379 for client requests.

To interact with the started cluster by using etcdctl:

# use API version 3
$ export ETCDCTL_API=3

$ ./etcdctl put foo bar
OK

$ ./etcdctl get foo
bar

Local multi-member cluster

A Procfile at the base of this git repo is provided to easily set up a local multi-member cluster. To start a multi-member cluster go to the root of an etcd source tree and run:

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

Setup a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to set up a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Deploying an etcd cluster as a standalone cluster is straightforward. Start it with just one command:

$ ./etcd
...

The started etcd member listens on localhost:2379 for client requests.

To interact with the started cluster by using etcdctl:

# use API version 3
$ export ETCDCTL_API=3

$ ./etcdctl put foo bar
OK

$ ./etcdctl get foo
bar

Local multi-member cluster

A Procfile at the base of this git repo is provided to easily set up a local multi-member cluster. To start a multi-member cluster go to the root of an etcd source tree and run:

Snapshot Migration

Mon, 01 Jan 0001 00:00:00 +0000

You can migrate a snapshot of your data from a v0.4.9+ cluster into a new etcd 2.2 cluster using a snapshot migration. After snapshot migration, the etcd indexes of your data will change. Many etcd applications rely on these indexes to behave correctly. This operation should only be done while all etcd applications are stopped.

To get started get the newest data snapshot from the 0.4.9+ cluster:

curl http://cluster.example.com:4001/v2/migration/snapshot > backup.snap

Now, import the snapshot into your new cluster:

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Storage Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Current support

The following table lists etcd support status for common architectures and operating systems,

Architecture	Operating System	Status	Maintainers
amd64	Darwin	Experimental	etcd maintainers
amd64	Linux	Stable	etcd maintainers
amd64	Windows	Experimental
arm64	Linux	Experimental
arm	Linux	Unstable
386	Linux	Unstable

etcd-maintainers are listed in https://github.com/etcd-io/etcd/blob/main/OWNERS.

Supported platforms

Mon, 01 Jan 0001 00:00:00 +0000

Current support

The following table lists etcd support status for common architectures and operating systems,

Architecture	Operating System	Status	Maintainers
amd64	Darwin	Experimental	etcd maintainers
amd64	Linux	Stable	etcd maintainers
amd64	Windows	Experimental
arm64	Linux	Experimental
arm	Linux	Unstable
386	Linux	Unstable
ppc64le	Linux	Stable	etcd maintainers

etcd-maintainers are listed in https://github.com/etcd-io/etcd/blob/main/OWNERS.

Supported systems

Mon, 01 Jan 0001 00:00:00 +0000

Current support

The following table lists etcd support status for common architectures and operating systems:

Architecture	Operating System	Status	Maintainers
amd64	Darwin	Experimental	etcd maintainers
amd64	Linux	Stable	etcd maintainers
amd64	Windows	Experimental
arm64	Linux	Experimental
arm	Linux	Unstable
386	Linux	Unstable
ppc64le	Linux	Stable	etcd maintainers

etcd-maintainers are listed in https://github.com/etcd-io/etcd/blob/main/OWNERS.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. For the time being, etcd guarantees to support RPC requests with up to 1MB of data. In the future, the size limit may be loosened or made it configurable.

Storage size limit

The default storage size limit is 2GB, configurable with --quota-backend-bytes flag; supports up to 8GB.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

Storage size limit

The default storage size limit is 2GB, configurable with --quota-backend-bytes flag; supports up to 8GB.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

Storage size limit

The default storage size limit is 2GB, configurable with --quota-backend-bytes flag. 8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

Transport security model

Mon, 01 Jan 0001 00:00:00 +0000

etcd supports automatic TLS as well as authentication through client certificates for both clients to server as well as peer (server to server / cluster) communication.

To get up and running, first have a CA certificate and a signed key pair for one member. It is recommended to create and sign a new key pair for every member in a cluster.

For convenience, the cfssl tool provides an easy interface to certificate generation, and we provide an example using the tool here. Alternatively, try this guide to generating self-signed key pairs.

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency you may need to tweak the heartbeat interval and election timeout settings.

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Understand failures

Mon, 01 Jan 0001 00:00:00 +0000

Understand failures

Mon, 01 Jan 0001 00:00:00 +0000

Upgrade etcd from 2.1 to 2.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.0 to 2.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.0 processes and replace them with etcd v2.1 processes
after you are running all v2.1 processes, new features in v2.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Checklists

Upgrade Requirements

To upgrade an existing etcd deployment to 2.1, you must be running 2.0. If you’re running a version of etcd before 2.0, you must upgrade to 2.0 before upgrading to 2.1.

Upgrade etcd from 2.1 to 2.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.1 to 2.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.1 processes and replace them with etcd v2.2 processes
after you are running all v2.2 processes, new features in v2.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Checklists

Upgrade Requirement

To upgrade an existing etcd deployment to 2.2, you must be running 2.1. If you’re running a version of etcd before 2.1, you must upgrade to 2.1 before upgrading to 2.2.

Upgrade etcd from 2.2 to 2.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.2 to 2.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.2 processes and replace them with etcd v2.3 processes
after running all v2.3 processes, new features in v2.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade Checklists

Upgrade Requirements

To upgrade an existing etcd deployment to 2.3, the running cluster must be 2.2 or greater. If it’s before 2.2, please upgrade to 2.2 before upgrading to 2.3.

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 2.3 to 3.0

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 2.3 to 3.0 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v2.3 processes and replace them with etcd v3.0 processes
after running all v3.0 processes, new features in v3.0 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.0 to 3.1

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.0 to 3.1 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.0 processes and replace them with etcd v3.1 processes
after running all v3.1 processes, new features in v3.1 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.1 to 3.2

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.1 to 3.2 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.1 processes and replace them with etcd v3.2 processes
after running all v3.2 processes, new features in v3.2 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.2 to 3.3

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.2 to 3.3 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.2 processes and replace them with etcd v3.3 processes
after running all v3.3 processes, new features in v3.3 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.3 to 3.4

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.3 to 3.4 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.3 processes and replace them with etcd v3.4 processes
after running all v3.4 processes, new features in v3.4 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrade etcd from 3.4 to 3.5

Mon, 01 Jan 0001 00:00:00 +0000

In the general case, upgrading from etcd 3.4 to 3.5 can be a zero-downtime, rolling upgrade:

one by one, stop the etcd v3.4 processes and replace them with etcd v3.5 processes
after running all v3.5 processes, new features in v3.5 are available to the cluster

Before starting an upgrade, read through the rest of this guide to prepare.

Upgrade checklists

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Moving from etcd API v2 to API v3

Migrate applications from using API v2 to API v3

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

Upgrading etcd clusters and applications

Mon, 01 Jan 0001 00:00:00 +0000

This section contains documents specific to upgrading etcd clusters and applications.

Moving from etcd API v2 to API v3

Migrate applications from using API v2 to API v3

Upgrading an etcd v3.x cluster

Upgrading from etcd v2.3

Upgrade a v2.3 cluster to v3.0

v2 Auth and Security

Mon, 01 Jan 0001 00:00:00 +0000

etcd Resources

There are three types of resources in etcd

permission resources: users and roles in the user store
key-value resources: key-value pairs in the key-value store
settings resources: security settings, auth settings, and dynamic etcd cluster settings (election/heartbeat)

Permission Resources

Users

A user is an identity to be authenticated. Each user can have multiple roles. The user has a capability (such as reading or writing) on the resource if one of the roles has that capability.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

Goal: We want to be able to upgrade an individual peer in an etcd cluster to a newer version of etcd. The process will take the form of individual followers upgrading to the latest version until the entire cluster is on the new version.

Immediate need: etcd is moving too fast to version the internal API right now. But, we need to keep mixed version clusters from being started by a rolling upgrade process (e.g. the CoreOS developer alpha).

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

Service versioning

etcd uses semantic versioning New minor versions may add additional features to the API.

Get the running etcd cluster version with etcdctl:

ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 endpoint status

API versioning

The v3 API responses should not change after the 3.0.0 release but new features will be added over time.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

Service versioning

etcd uses semantic versioning New minor versions may add additional features to the API.

Get the running etcd cluster version with etcdctl:

ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 endpoint status

API versioning

The v3 API responses should not change after the 3.0.0 release but new features will be added over time.

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Watch Memory Usage Benchmark

Mon, 01 Jan 0001 00:00:00 +0000

Why etcd

Mon, 01 Jan 0001 00:00:00 +0000

etcd stores metadata in a consistent and fault-tolerant way. Distributed systems use etcd as a consistent key-value store for configuration management, service discovery, and coordinating distributed work. Common distributed patterns using etcd include leader election, distributed locks, and monitoring machine liveness.

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000