v3.3 docs on etcd

Install

Mon, 01 Jan 0001 00:00:00 +0000

System requirements

The etcd performance benchmarks run etcd on 8 vCPU, 16GB RAM, 50GB SSD GCE instances, but any relatively modern machine with low latency storage and a few gigabytes of memory should suffice for most use cases. Applications with large v2 data stores will require more memory than a large v3 data store since data is kept in anonymous memory instead of memory mapped from a file. For running etcd on a cloud provider, see the Example hardware configuration documentation.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses Prometheus for metrics reporting. The metrics can be used for real-time monitoring and debugging. etcd does not persist its metrics; if a member restarts, the metrics will be reset.

The simplest way to see the available metrics is to cURL the metrics endpoint /metrics. The format is described here.

Follow the Prometheus getting started doc to spin up a Prometheus server to collect etcd metrics.

The naming of metrics follows the suggested Prometheus best practices. A metric name has an etcd or etcd_debugging prefix as its namespace and a subsystem prefix (for example wal and etcdserver).

Libraries and tools

Mon, 01 Jan 0001 00:00:00 +0000

Note that third-party libraries and tools (not hosted on etcd-io main repository) mentioned below are not tested or maintained by the etcd team. Before using them, users are recommended to read and investigate them.

Tools

etcdctl - A command line client for etcd
etcd-backup - A powerful command line utility for dumping/restoring etcd - Supports v2
etcd-dump - Command line utility for dumping/restoring etcd.
etcd-fs - FUSE filesystem for etcd
etcddir - Realtime sync etcd and local directory. Work with windows and linux.
etcd-browser - A web-based key/value editor for etcd using AngularJS
etcd-lock - Master election & distributed r/w lock implementation using etcd - Supports v2
etcd-console - A web-base key/value editor for etcd using PHP
etcd-viewer - An etcd key-value store editor/viewer written in Java
etcdtool - Export/Import/Edit etcd directory as JSON/YAML/TOML and Validate directory using JSON schema
etcd-rest - Create generic REST API in Go using etcd as a backend with validation using JSON schema
etcdsh - A command line client with support of command history and tab completion. Supports v2
etcdloadtest - A command line load test client for etcd version 3.0 and above.
lucas - A web-based key-value viewer for kubernetes etcd3.0+ cluster.
etcd-workbench - A free and powerful ui client for etcd v3. Provides desktop application and web packages.

Libraries

The sections below list etcd client libraries by language.

Demo

Mon, 01 Jan 0001 00:00:00 +0000

This series of examples shows the basic procedures for working with an etcd cluster.

Set up a cluster

On each etcd node, specify the cluster members:

TOKEN=token-01
CLUSTER_STATE=new
NAME_1=machine-1
NAME_2=machine-2
NAME_3=machine-3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380

Run this on each machine:

# For machine 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For machine 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
etcd --data-dir=data.etcd --name ${THIS_NAME} \
	--initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
	--advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

Or use our public discovery service:

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

The protocol uses a new discovery token to bootstrap one unique etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if it fails halfway, it must not be used to bootstrap another etcd cluster.

Frequently Asked Questions (FAQ)

Mon, 01 Jan 0001 00:00:00 +0000

etcd, general

Do clients have to send requests to the etcd leader?

Raft is leader-based; the leader handles all client requests which need cluster consensus. However, the client does not need to know which node is the leader. Any request that requires consensus sent to a follower is automatically forwarded to the leader. Requests that do not require consensus (e.g., serialized reads) can be processed by any cluster member.

Configuration

What is the difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?

listen-client-urls and listen-peer-urls specify the local addresses etcd server binds to for accepting incoming connections. To listen on a port for all interfaces, specify 0.0.0.0 as the listen IP address.

Logging conventions

Mon, 01 Jan 0001 00:00:00 +0000

etcd uses the capnslog library for logging application output categorized into levels. A log message’s level is determined according to these conventions:

Error: Data has been lost, a request has failed for a bad reason, or a required resource has been lost
- Examples:
  - A failure to allocate disk space for WAL
Warning: (Hopefully) Temporary conditions that may cause errors, but may work fine. A replica disappearing (that may reconnect) is a warning.

Production users

Mon, 01 Jan 0001 00:00:00 +0000

This document tracks people and use cases for etcd in production. By creating a list of production use cases we hope to build a community of advisors that we can reach out to with experience using various etcd applications, operation environments, and cluster sizes. The etcd development team may reach out periodically to check-in on how etcd is working in the field and update this list.

All Kubernetes Users

Application: https://kubernetes.io/
Environments: AWS, OpenStack, Azure, Google Cloud, Huawei Cloud, Bare Metal, etc

This is a meta user; please feel free to document specific Kubernetes clusters!

Reporting bugs

Mon, 01 Jan 0001 00:00:00 +0000

If any part of the etcd project has bugs or documentation mistakes, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.

To make the bug report accurate and easy to understand, please try to create bug reports that are:

Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the etcd server, please attach the etcd log (the starting log with etcd configuration is especially important).

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

The default settings in etcd should work well for installations on a local network where the average network latency is low. However, when using etcd across multiple data centers or over networks with high latency, the heartbeat interval and election timeout settings may need tuning.

The network isn’t the only source of latency. Each request and response may be impacted by slow disks on both the leader and follower. Each of these timeouts represents the total time from request to successful response from the other machine.