Developer guide on etcd

Discovery service protocol

Mon, 01 Jan 0001 00:00:00 +0000

Discovery service protocol helps new etcd member to discover all other members in cluster bootstrap phase using a shared discovery URL.

Discovery service protocol is only used in cluster bootstrap phase, and cannot be used for runtime reconfiguration or cluster monitoring.

The protocol uses a new discovery token to bootstrap one unique etcd cluster. Remember that one discovery token can represent only one etcd cluster. As long as discovery protocol on this token starts, even if it fails halfway, it must not be used to bootstrap another etcd cluster.

etcd API Reference

Mon, 01 Jan 0001 00:00:00 +0000

This is a generated documentation. Please read the proto files for more.

service `Auth` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
AuthEnable	AuthEnableRequest	AuthEnableResponse	AuthEnable enables authentication.
AuthDisable	AuthDisableRequest	AuthDisableResponse	AuthDisable disables authentication.
Authenticate	AuthenticateRequest	AuthenticateResponse	Authenticate processes an authenticate request.
UserAdd	AuthUserAddRequest	AuthUserAddResponse	UserAdd adds a new user.
UserGet	AuthUserGetRequest	AuthUserGetResponse	UserGet gets detailed user information.
UserList	AuthUserListRequest	AuthUserListResponse	UserList gets a list of all users.
UserDelete	AuthUserDeleteRequest	AuthUserDeleteResponse	UserDelete deletes a specified user.
UserChangePassword	AuthUserChangePasswordRequest	AuthUserChangePasswordResponse	UserChangePassword changes the password of a specified user.
UserGrantRole	AuthUserGrantRoleRequest	AuthUserGrantRoleResponse	UserGrant grants a role to a specified user.
UserRevokeRole	AuthUserRevokeRoleRequest	AuthUserRevokeRoleResponse	UserRevokeRole revokes a role of specified user.
RoleAdd	AuthRoleAddRequest	AuthRoleAddResponse	RoleAdd adds a new role.
RoleGet	AuthRoleGetRequest	AuthRoleGetResponse	RoleGet gets detailed role information.
RoleList	AuthRoleListRequest	AuthRoleListResponse	RoleList gets lists of all roles.
RoleDelete	AuthRoleDeleteRequest	AuthRoleDeleteResponse	RoleDelete deletes a specified role.
RoleGrantPermission	AuthRoleGrantPermissionRequest	AuthRoleGrantPermissionResponse	RoleGrantPermission grants a permission of a specified key or range to a specified role.
RoleRevokePermission	AuthRoleRevokePermissionRequest	AuthRoleRevokePermissionResponse	RoleRevokePermission revokes a key or range permission of a specified role.

service `Cluster` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
MemberAdd	MemberAddRequest	MemberAddResponse	MemberAdd adds a member into the cluster.
MemberRemove	MemberRemoveRequest	MemberRemoveResponse	MemberRemove removes an existing member from the cluster.
MemberUpdate	MemberUpdateRequest	MemberUpdateResponse	MemberUpdate updates the member configuration.
MemberList	MemberListRequest	MemberListResponse	MemberList lists all the members in the cluster.

service `KV` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Range	RangeRequest	RangeResponse	Range gets the keys in the range from the key-value store.
Put	PutRequest	PutResponse	Put puts the given key into the key-value store. A put request increments the revision of the key-value store and generates one event in the event history.
DeleteRange	DeleteRangeRequest	DeleteRangeResponse	DeleteRange deletes the given range from the key-value store. A delete request increments the revision of the key-value store and generates a delete event in the event history for every deleted key.
Txn	TxnRequest	TxnResponse	Txn processes multiple requests in a single transaction. A txn request increments the revision of the key-value store and generates events with the same revision for every completed request. It is not allowed to modify the same key several times within one txn.
Compact	CompactionRequest	CompactionResponse	Compact compacts the event history in the etcd key-value store. The key-value store should be periodically compacted or the event history will continue to grow indefinitely.

service `Lease` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
LeaseGrant	LeaseGrantRequest	LeaseGrantResponse	LeaseGrant creates a lease which expires if the server does not receive a keepAlive within a given time to live period. All keys attached to the lease will be expired and deleted if the lease expires. Each expired key generates a delete event in the event history.
LeaseRevoke	LeaseRevokeRequest	LeaseRevokeResponse	LeaseRevoke revokes a lease. All keys attached to the lease will expire and be deleted.
LeaseKeepAlive	LeaseKeepAliveRequest	LeaseKeepAliveResponse	LeaseKeepAlive keeps the lease alive by streaming keep alive requests from the client to the server and streaming keep alive responses from the server to the client.
LeaseTimeToLive	LeaseTimeToLiveRequest	LeaseTimeToLiveResponse	LeaseTimeToLive retrieves lease information.
LeaseLeases	LeaseLeasesRequest	LeaseLeasesResponse	LeaseLeases lists all existing leases.

service `Maintenance` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Alarm	AlarmRequest	AlarmResponse	Alarm activates, deactivates, and queries alarms regarding cluster health.
Status	StatusRequest	StatusResponse	Status gets the status of the member.
Defragment	DefragmentRequest	DefragmentResponse	Defragment defragments a member’s backend database to recover storage space.
Hash	HashRequest	HashResponse	Hash computes the hash of whole backend keyspace, including key, lease, and other buckets in storage. This is designed for testing ONLY! Do not rely on this in production with ongoing transactions, since Hash operation does not hold MVCC locks. Use “HashKV” API instead for “key” bucket consistency checks.
HashKV	HashKVRequest	HashKVResponse	HashKV computes the hash of all MVCC keys up to a given revision. It only iterates “key” bucket in backend storage.
Snapshot	SnapshotRequest	SnapshotResponse	Snapshot sends a snapshot of the entire backend from a member over a stream to a client.
MoveLeader	MoveLeaderRequest	MoveLeaderResponse	MoveLeader requests current leader node to transfer its leadership to transferee.

service `Watch` (etcdserver/etcdserverpb/rpc.proto)

Method	Request Type	Response Type	Description
Watch	WatchRequest	WatchResponse	Watch watches for events happening or that have happened. Both input and output are streams; the input stream is for creating and canceling watchers and the output stream sends events. One watch RPC can watch on multiple key ranges, streaming events for several watches at once. The entire event history can be watched starting from the last compaction revision.

message `AlarmMember` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
memberID	memberID is the ID of the member associated with the raised alarm.	uint64
alarm	alarm is the type of alarm which has been raised.	AlarmType

message `AlarmRequest` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
action	action is the kind of alarm request to issue. The action may GET alarm statuses, ACTIVATE an alarm, or DEACTIVATE a raised alarm.	AlarmAction
memberID	memberID is the ID of the member associated with the alarm. If memberID is 0, the alarm request covers all members.	uint64
alarm	alarm is the type of alarm to consider for this request.	AlarmType

message `AlarmResponse` (etcdserver/etcdserverpb/rpc.proto)

Field	Description	Type
header		ResponseHeader
alarms	alarms is a list of alarms associated with the alarm request.	(slice of) AlarmMember

message `AuthDisableRequest` (etcdserver/etcdserverpb/rpc.proto)

Empty field.

etcd concurrency API Reference

Mon, 01 Jan 0001 00:00:00 +0000

This is a generated documentation. Please read the proto files for more.

service `Lock` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

The lock service exposes client-side locking facilities as a gRPC interface.

Method	Request Type	Response Type	Description
Lock	LockRequest	LockResponse	Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires.
Unlock	UnlockRequest	UnlockResponse	Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock.

message `LockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
name	name is the identifier for the distributed shared lock to be acquired.	bytes
lease	lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op.	int64

message `LockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader
key	key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior.	bytes

message `UnlockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
key	key is the lock ownership key granted by Lock.	bytes

message `UnlockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

Field	Description	Type
header		etcdserverpb.ResponseHeader

service `Election` (etcdserver/api/v3election/v3electionpb/v3election.proto)

The election service exposes client-side election facilities as a gRPC interface.

Experimental APIs and features

Mon, 01 Jan 0001 00:00:00 +0000

For the most part, the etcd project is stable, but we are still moving fast! We believe in the release fast philosophy. We want to get early feedback on features still in development and stabilizing. Thus, there are, and will be more, experimental features and APIs. We plan to improve these features based on the early feedback from the community, or abandon them if there is little interest, in the next few releases. Please do not rely on any experimental features or APIs in production environment.

gRPC naming and discovery

Mon, 01 Jan 0001 00:00:00 +0000

etcd provides a gRPC resolver to support an alternative name system that fetches endpoints from etcd for discovering gRPC services. The underlying mechanism is based on watching updates to keys prefixed with the service name.

Using etcd discovery with go-grpc

The etcd client provides a gRPC resolver for resolving gRPC endpoints with an etcd backend. The resolver is initialized with an etcd client and given a target for resolution:

import (
	"go.etcd.io/etcd/clientv3"
	etcdnaming "go.etcd.io/etcd/clientv3/naming"

	"google.golang.org/grpc"
)

...

cli, cerr := clientv3.NewFromURL("http://localhost:2379")
r := &etcdnaming.GRPCResolver{Client: cli}
b := grpc.RoundRobin(r)
conn, gerr := grpc.Dial("my-service", grpc.WithBalancer(b), grpc.WithBlock(), ...)

Managing service endpoints

The etcd resolver treats all keys under the prefix of the resolution target following a “/” (e.g., “my-service/”) with JSON-encoded go-grpc naming.Update values as potential service endpoints. Endpoints are added to the service by creating new keys and removed from the service by deleting keys.

Interacting with etcd

Mon, 01 Jan 0001 00:00:00 +0000

Users mostly interact with etcd by putting or getting the value of a key. This section describes how to do that by using etcdctl, a command line tool for interacting with etcd server. The concepts described here should apply to the gRPC APIs or client library APIs.

The API version used by etcdctl to speak to etcd may be set to version 2 or 3 via the ETCDCTL_API environment variable. By default, etcdctl on master (3.4) uses the v3 API and earlier versions (3.3 and earlier) default to the v2 API.

Set up a local cluster

Mon, 01 Jan 0001 00:00:00 +0000

For testing and development deployments, the quickest and easiest way is to configure a local cluster. For a production deployment, refer to the clustering section.

Local standalone cluster

Starting a cluster

Run the following to deploy an etcd cluster as a standalone cluster:

$ ./etcd
...

If the etcd binary is not present in the current working directory, it might be located either at $GOPATH/bin/etcd or at /usr/local/bin/etcd. Run the command appropriately.

System limits

Mon, 01 Jan 0001 00:00:00 +0000

Request size limit

etcd is designed to handle small key value pairs typical for metadata. Larger requests will work, but may increase the latency of other requests. By default, the maximum size of any request is 1.5 MiB. This limit is configurable through --max-request-bytes flag for etcd server.

Storage size limit

The default storage size limit is 2GB, configurable with --quota-backend-bytes flag. 8GB is a suggested maximum size for normal environments and etcd warns at startup if the configured value exceeds it.

Why gRPC gateway

Mon, 01 Jan 0001 00:00:00 +0000

etcd v3 uses gRPC for its messaging protocol. The etcd project includes a gRPC-based Go client and a command line utility, etcdctl, for communicating with an etcd cluster through gRPC. For languages with no gRPC support, etcd provides a JSON gRPC gateway. This gateway serves a RESTful proxy that translates HTTP/JSON requests into gRPC messages.

Using gRPC gateway

The gateway accepts a JSON mapping for etcd’s protocol buffer message definitions. Note that key and value fields are defined as byte arrays and therefore must be base64 encoded in JSON. The following examples use curl, but any HTTP/JSON client should work all the same.

Developer guide on etcd

Discovery service protocol

etcd API Reference

service Auth (etcdserver/etcdserverpb/rpc.proto)

service Cluster (etcdserver/etcdserverpb/rpc.proto)

service KV (etcdserver/etcdserverpb/rpc.proto)

service Lease (etcdserver/etcdserverpb/rpc.proto)

service Maintenance (etcdserver/etcdserverpb/rpc.proto)

service Watch (etcdserver/etcdserverpb/rpc.proto)

message AlarmMember (etcdserver/etcdserverpb/rpc.proto)

message AlarmRequest (etcdserver/etcdserverpb/rpc.proto)

message AlarmResponse (etcdserver/etcdserverpb/rpc.proto)

message AuthDisableRequest (etcdserver/etcdserverpb/rpc.proto)

etcd concurrency API Reference

service Lock (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message LockRequest (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message LockResponse (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message UnlockRequest (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message UnlockResponse (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

service Election (etcdserver/api/v3election/v3electionpb/v3election.proto)

Experimental APIs and features

gRPC naming and discovery

Using etcd discovery with go-grpc

Managing service endpoints

Interacting with etcd

Set up a local cluster

Local standalone cluster

Starting a cluster

System limits

Request size limit

Storage size limit

Why gRPC gateway

Using gRPC gateway

service `Auth` (etcdserver/etcdserverpb/rpc.proto)

service `Cluster` (etcdserver/etcdserverpb/rpc.proto)

service `KV` (etcdserver/etcdserverpb/rpc.proto)

service `Lease` (etcdserver/etcdserverpb/rpc.proto)

service `Maintenance` (etcdserver/etcdserverpb/rpc.proto)

service `Watch` (etcdserver/etcdserverpb/rpc.proto)

message `AlarmMember` (etcdserver/etcdserverpb/rpc.proto)

message `AlarmRequest` (etcdserver/etcdserverpb/rpc.proto)

message `AlarmResponse` (etcdserver/etcdserverpb/rpc.proto)

message `AuthDisableRequest` (etcdserver/etcdserverpb/rpc.proto)

service `Lock` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message `LockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message `LockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message `UnlockRequest` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

message `UnlockResponse` (etcdserver/api/v3lock/v3lockpb/v3lock.proto)

service `Election` (etcdserver/api/v3election/v3electionpb/v3election.proto)