使用 StatefulSet 部署 etcd 集群

上文我們簡(jiǎn)單介紹了 etcd 的基本概念和使用場(chǎng)景，本文就來(lái)介紹如何搭建 etcd 集群。在生產(chǎn)環(huán)境中，為了整個(gè)集群的高可用，etcd 正常都會(huì)以集群方式部署，避免單點(diǎn)故障。引導(dǎo) etcd 集群的啟動(dòng)有以下三種機(jī)制：

• 靜態(tài)
• etcd 動(dòng)態(tài)發(fā)現(xiàn)
• DNS 發(fā)現(xiàn)

靜態(tài)啟動(dòng) etcd 集群要求每個(gè)成員都知道集群中的另一個(gè)成員。在許多情況下，群集成員的 IP 可能未知，在這些情況下，可以在發(fā)現(xiàn)服務(wù)的幫助下引導(dǎo) etcd 集群。

“

可以使用官方提供的工具來(lái)生成 etcd 集群的配置：http://play.etcd.io/install

”

這里我們將主要介紹靜態(tài)方式啟動(dòng) etcd 集群。

安裝

由于 etcd 是基于 raft 分布式協(xié)議的集群，所以如果要組成一個(gè)集群，集群的數(shù)量需要奇數(shù)臺(tái)。如果只有一個(gè)節(jié)點(diǎn)的話我們也可以通過(guò)不同的端口來(lái)進(jìn)行模擬，比如這里我們?cè)谝慌_(tái)機(jī)器上來(lái)安裝一個(gè)3節(jié)點(diǎn)的 etcd 偽集群，對(duì)應(yīng)的配置如下所示：

這里我們?cè)谝慌_(tái) CentOS7 系統(tǒng)的節(jié)點(diǎn)上來(lái)進(jìn)行演示，首先下載 etcd 二進(jìn)制包，直接從 github release 頁(yè)面下載對(duì)應(yīng)的包即可：

$?wget?https://github.com/etcd-io/etcd/releases/download/v3.4.13/etcd-v3.4.13-linux-amd64.tar.gz
$?tar?-xvf?etcd-v3.4.13-linux-amd64.tar.gz
$?mkdir?/tmp/etcd?
$?mv?etcd-v3.4.13-linux-amd64/etcd?/tmp/etcd/
$?mv?etcd-v3.4.13-linux-amd64/etcdctl?/tmp/etcd

etcd 安裝完成后，啟動(dòng)3個(gè)不同的終端分別執(zhí)行下面的3個(gè)命令來(lái)啟動(dòng) etcd 集群：

#?確保?etcd?進(jìn)程對(duì)數(shù)據(jù)目錄具有寫(xiě)訪問(wèn)權(quán)
#?如果集群是新集群，則刪除該目錄；如果重新啟動(dòng)則保留該目錄

#?啟動(dòng)第一個(gè)節(jié)點(diǎn)
$?/tmp/etcd/etcd?--name?s1?\??#?etcd?節(jié)點(diǎn)名稱
??--data-dir?/tmp/etcd/s1?\??#?數(shù)據(jù)存儲(chǔ)目錄?
??--listen-client-urls?http://localhost:2379?\??#?本節(jié)點(diǎn)訪問(wèn)地址
??--advertise-client-urls?http://localhost:2379?\??#?用于通知其他?ETCD?節(jié)點(diǎn)，客戶端接入本節(jié)點(diǎn)的監(jiān)聽(tīng)地址
??--listen-peer-urls?http://localhost:2380?\??#?本節(jié)點(diǎn)與其他節(jié)點(diǎn)進(jìn)行數(shù)據(jù)交換的監(jiān)聽(tīng)地址
??--initial-advertise-peer-urls?http://localhost:2380?\??#?通知其他節(jié)點(diǎn)與本節(jié)點(diǎn)進(jìn)行數(shù)據(jù)交換的地址
??--initial-cluster?s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380?\??#?集群所有節(jié)點(diǎn)配置
??--initial-cluster-token?tkn?\??#?集群唯一標(biāo)識(shí)
??--initial-cluster-state?new??#?節(jié)點(diǎn)初始化方式

#?啟動(dòng)第二個(gè)節(jié)點(diǎn)
$?/tmp/etcd/etcd?--name?s2?\
??--data-dir?/tmp/etcd/s2?\
??--listen-client-urls?http://localhost:22379?\
??--advertise-client-urls?http://localhost:22379?\
??--listen-peer-urls?http://localhost:22380?\
??--initial-advertise-peer-urls?http://localhost:22380?\
??--initial-cluster?s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380?\
??--initial-cluster-token?tkn?\
??--initial-cluster-state?new

#?啟動(dòng)第三個(gè)節(jié)點(diǎn)
$?/tmp/etcd/etcd?--name?s3?\
??--data-dir?/tmp/etcd/s3?\
??--listen-client-urls?http://localhost:32379?\
??--advertise-client-urls?http://localhost:32379?\
??--listen-peer-urls?http://localhost:32380?\
??--initial-advertise-peer-urls?http://localhost:32380?\
??--initial-cluster?s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380?\
??--initial-cluster-token?tkn?\
??--initial-cluster-state?new

正常啟動(dòng)完成后，我們可以使用 etcdctl 命令來(lái)查看集群的狀態(tài)：

$?ETCDCTL_API=3?/tmp/etcd/etcdctl?\
??--endpoints?localhost:2379,localhost:22379,localhost:32379?\
??endpoint?health
localhost:2379?is?healthy:?successfully?committed?proposal:?took?=?14.22105ms
localhost:22379?is?healthy:?successfully?committed?proposal:?took?=?13.058173ms
localhost:32379?is?healthy:?successfully?committed?proposal:?took?=?16.497453ms
$?ETCDCTL_API=3?/tmp/etcd/etcdctl?\
??--endpoints?localhost:2379,localhost:22379,localhost:32379?\?
??endpoint?status?--write-out=table
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|????ENDPOINT?????|????????ID????????|?VERSION?|?DB?SIZE?|?IS?LEADER?|?IS?LEARNER?|?RAFT?TERM?|?RAFT?INDEX?|?RAFT?APPLIED?INDEX?|?ERRORS?|
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|??localhost:2379?|?7339c4e5e833c029?|??3.4.13?|???20?kB?|??????true?|??????false?|????????43?|??????????9?|??????????????????9?|????????|
|?localhost:22379?|?729934363faa4a24?|??3.4.13?|???20?kB?|?????false?|??????false?|????????43?|??????????9?|??????????????????9?|????????|
|?localhost:32379?|??b548c2511513015?|??3.4.13?|???20?kB?|?????false?|??????false?|????????43?|??????????9?|??????????????????9?|????????|
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

正?？梢钥吹?個(gè) etcd 節(jié)點(diǎn)都運(yùn)行成功了，也可以查看當(dāng)前集群的 LEADER 節(jié)點(diǎn)。如果這個(gè)時(shí)候我們把 2379 端口的進(jìn)程殺掉，再來(lái)查看集群的狀態(tài)：

$?ETCDCTL_API=3?/tmp/etcd/etcdctl???--endpoints?localhost:2379,localhost:22379,localhost:32379??endpoint?status?--write-out=table
{"level":"warn","ts":"2020-11-16T14:39:25.024+0800","caller":"clientv3/retry_interceptor.go:62","msg":"retrying?of?unary?invoker?failed","target":"passthrough:///localhost:2379","attempt":0,"error":"rpc?error:?code?=?DeadlineExceeded?desc?=?latest?balancer?error:?connection?error:?desc?=?\"transport:?Error?while?dialing?dial?tcp?[::1]:2379:?connect:?connection?refused\""}
Failed?to?get?the?status?of?endpoint?localhost:2379?(context?deadline?exceeded)
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|????ENDPOINT?????|????????ID????????|?VERSION?|?DB?SIZE?|?IS?LEADER?|?IS?LEARNER?|?RAFT?TERM?|?RAFT?INDEX?|?RAFT?APPLIED?INDEX?|?ERRORS?|
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?localhost:22379?|?729934363faa4a24?|??3.4.13?|???20?kB?|??????true?|??????false?|????????44?|?????????10?|?????????????????10?|????????|
|?localhost:32379?|??b548c2511513015?|??3.4.13?|???20?kB?|?????false?|??????false?|????????44?|?????????10?|?????????????????10?|????????|
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

可以看到現(xiàn)在集群中只有兩個(gè)節(jié)點(diǎn)，但是現(xiàn)在集群還是可以正常使用的，因?yàn)槲覀円还?個(gè)節(jié)點(diǎn)，現(xiàn)在還有超過(guò)一一半的節(jié)點(diǎn)正常，那么集群就是正常的，當(dāng)然如果我們?cè)訇P(guān)閉一個(gè)節(jié)點(diǎn)那么我們集群就將處于不健康的狀態(tài)下了。

參數(shù)說(shuō)明

上面我們成功搭建了 etcd 的集群，etcd 在啟動(dòng)過(guò)程中有很多參數(shù)可以配置，這里我們就先來(lái)對(duì)這些參數(shù)進(jìn)行一個(gè)簡(jiǎn)單的說(shuō)明。

• --name ：節(jié)點(diǎn)名稱

  默認(rèn)值：default

  環(huán)境變量：ETCD_NAME

  說(shuō)明：這個(gè)值和 --initial-cluster 參數(shù)中的 key 值一一對(duì)應(yīng)，如果在集群環(huán)境中，name 必須是唯一的，建議用主機(jī)名稱或者機(jī)器 ID。

• --data-dir：數(shù)據(jù)存儲(chǔ)目錄

  默認(rèn)值：${name}.etcd

  環(huán)境變量：ETCD_DATA_DIR

• --wal-dir：存放預(yù)寫(xiě)日志目錄

  默認(rèn)值：""

  環(huán)境變量：ETCD_WAL_DIR

  說(shuō)明：最大的作用是記錄了整個(gè)數(shù)據(jù)變化的全部歷程，如果未配置則共用 --data-dir 文件所在目錄。

• --snapshot-count：數(shù)據(jù)快照觸發(fā)數(shù)量

  默認(rèn)值：100000

  環(huán)境變量：ETCD_SNAPSHOT_COUNT

  說(shuō)明：etcd 處理指定次數(shù)的事務(wù)提交后，生產(chǎn)數(shù)據(jù)快照

• --heartbeat-interval：客戶端連接后的心跳間隔（毫秒）

  默認(rèn)值：100

  環(huán)境變量：ETCD_HEARTBEAT_INTERVAL

• --election-timeout**：**集群選舉的超時(shí)時(shí)間

  默認(rèn)值：1000

  環(huán)境變量：ETCD_ELECTION_TIMEOUT

• --max-snapshots：最大快照數(shù)量，0表示不限制

  默認(rèn)值：5

  環(huán)境變量：ETCD_MAX_SNAPSHOTS

• --max-wals：最大預(yù)寫(xiě)日志數(shù)量，0表示不限制

  默認(rèn)值：5

  環(huán)境變量：ETCD_MAX_WAL

• --listen-peer-urls：本節(jié)點(diǎn)使用，用于監(jiān)聽(tīng)其他節(jié)點(diǎn)發(fā)送信息的地址，地址寫(xiě)法是 scheme://IP:port

  默認(rèn)值：http://localhost:2380

  環(huán)境變量：ETCD_LISTEN_PEER_URL

  示例：http://10.0.0.1:2380

  無(wú)效的示例：http://example.com:2380 (域名對(duì)綁定無(wú)效)

• --listen-client-urls：本節(jié)點(diǎn)使用，用于 etcd 客戶端通信的 url，寫(xiě)法是 scheme://IP:port，可以多個(gè)并用逗號(hào)隔開(kāi)

  默認(rèn)值：http://localhost:2379

  示例：http://10.0.0.1:2379

  無(wú)效的示例：http://example.com:2379 (域名對(duì)綁定無(wú)效)

• --initial-advertise-peer-urls：其他節(jié)點(diǎn)通過(guò)該地址與本節(jié)點(diǎn)進(jìn)行數(shù)據(jù)交換（選舉，同步）的地址，URL 可以使用域名地址

  默認(rèn)值：http://localhost:2380

  環(huán)境變量：ETCD_INITIAL_ADVERTISE_PEER_URL

  說(shuō)明：與 --listener-peer-urls 不同在于 listener-peer-urls 用于請(qǐng)求客戶端的接入控制，initial-advertise-peer-urls 是告知其他集群節(jié)點(diǎn)訪問(wèn)哪個(gè) URL，一般來(lái)說(shuō)，initial-advertise-peer-urlsl 將是 listene-peer-urls 的子集。靜態(tài)配置方式下，該參數(shù)值一定要同時(shí)在 --initial-cluster 參數(shù)中存在

• --advertise-client-urls ：用于通知其他 ETCD 節(jié)點(diǎn)，客戶端接入本節(jié)點(diǎn)的監(jiān)聽(tīng)地址，一般來(lái)說(shuō) advertise-client-urls 是 listen-client-urls 子集

  默認(rèn)值：http://localhost:2379

  環(huán)境變量：ETCD_ADVERTISE_CLIENT_URL

• --initial-cluster ：集群所有節(jié)點(diǎn)配置，多個(gè)用逗號(hào)隔開(kāi)

  默認(rèn)值：http://localhost:2380

  環(huán)境變量：ETCD_INITIAL_CLUSTER

  說(shuō)明：key 是所提供的每個(gè)節(jié)點(diǎn)的 --name 標(biāo)志的值

• --initial-cluster-state ：節(jié)點(diǎn)初始化方式

  默認(rèn)值：new

  環(huán)境變量：ETCD_INITIAL_CLUSTER_STATE

  說(shuō)明：new 表示如果沒(méi)有集群不存在，創(chuàng)建新集群，existing 表示如果集群不存在，節(jié)點(diǎn)將處于加入集群失敗狀態(tài)。

• --initial-cluster-token ：集群唯一標(biāo)識(shí)

  默認(rèn)值：etcd-cluster

  環(huán)境變量：ETCD_INITIAL_CLUSTER_TOKEN

  說(shuō)明：相同標(biāo)識(shí)的節(jié)點(diǎn)將視為在一個(gè)集群內(nèi)

在 Kubernetes 集群中部署

我們現(xiàn)在了解了 etcd 集群的基本搭建方式，那么我們應(yīng)該如何將其運(yùn)行到 Kubernetes 集群中呢？畢竟我們是要來(lái)編寫(xiě) Operator 的，所以我們肯定需要先知道怎么能夠在 Kubernetes 集群中運(yùn)行起來(lái)。

這里我們可以使用 StatefulSet 這個(gè)控制器來(lái)運(yùn)行 etcd 集群，etcd 集群的編排的資源清單文件我們可以使用 Kubernetes 源碼中提供的，位于目錄：test/e2e/testing-manifests/statefulset/etcd 下面。

$?ls?-la?test/e2e/testing-manifests/statefulset/etcd??
total?40
drwxr-xr-x???6?ych??staff???192?Jun?18??2019?.
drwxr-xr-x??10?ych??staff???320?Oct?10??2018?..
-rw-r--r--???1?ych??staff???173?Oct?10??2018?pdb.yaml
-rw-r--r--???1?ych??staff???242?Oct?10??2018?service.yaml
-rw-r--r--???1?ych??staff??6441?Jun?18??2019?statefulset.yaml
-rw-r--r--???1?ych??staff???550?Oct?10??2018?tester.yaml

其中 service.yaml 文件中就是一個(gè)用戶 StatefulSet 使用的 headless service：

apiVersion:?v1
kind:?Service
metadata:
??name:?etcd
??labels:
????app:?etcd
spec:
??ports:
??-?port:?2380
????name:?etcd-server
??-?port:?2379
????name:?etcd-client
??clusterIP:?None
??selector:
????app:?etcd
??publishNotReadyAddresses:?true

而 pdb.yaml 文件是用來(lái)保證 etcd 的高可用的一個(gè) PodDisruptionBudget 資源對(duì)象：

apiVersion:?policy/v1beta1
kind:?PodDisruptionBudget
metadata:
??name:?etcd-pdb
??labels:
????pdb:?etcd
spec:
??minAvailable:?2
??selector:
????matchLabels:
??????app:?etcd

最重要的當(dāng)然是 statefulset.yaml 文件，但是這個(gè)文件有很多 bug，比如在上面參數(shù)配置的時(shí)候我們就提到過(guò) --listen-peer-urls 和 ?--listen-client-urls 這兩個(gè)參數(shù)的值是不支持域名的綁定形式的，而這里使用的 http://${HOSTNAME}.${SET_NAME} 這個(gè) FQDN 形式的域名，所以啟動(dòng)失敗了，我們需要將其修改為 IP 的形式，要獲取 Pod 的 IP 地址也很簡(jiǎn)單，我們可以通過(guò) Kubernetes 提供的 Downward API 來(lái)注入一個(gè) POD_IP 的環(huán)境變量來(lái)獲取，然后將這兩個(gè)參數(shù)值改成 http://${POD_IP}:PORT 即可。另外這個(gè)資源清單文件的鏡像版本太老了，而且沒(méi)有支持 V3 版本的 API，所以這里我們升級(jí)下鏡像，讓其支持 V3 版本的接口。修改過(guò)后的完整資源清單文件如下所示：

apiVersion:?apps/v1
kind:?StatefulSet
metadata:
??labels:
????app:?etcd
??name:?etcd
spec:
??replicas:?3
??selector:
????matchLabels:
??????app:?etcd
??serviceName:?etcd
??template:
????metadata:
??????labels:
????????app:?etcd
????spec:
??????containers:
????????-?name:?etcd
??????????image:?cnych/etcd:v3.4.13
??????????imagePullPolicy:?IfNotPresent
??????????ports:
??????????-?containerPort:?2380
????????????name:?peer
????????????protocol:?TCP
??????????-?containerPort:?2379
????????????name:?client
????????????protocol:?TCP
??????????env:
??????????-?name:?INITIAL_CLUSTER_SIZE
????????????value:?"3"
??????????-?name:?MY_NAMESPACE
????????????valueFrom:
??????????????fieldRef:
????????????????fieldPath:?metadata.namespace
??????????-?name:?POD_IP
????????????valueFrom:
??????????????fieldRef:
????????????????fieldPath:?status.podIP
??????????-?name:?SET_NAME
????????????value:?"etcd"
??????????command:
????????????-?/bin/sh
????????????-?-ec
????????????-?|
??????????????HOSTNAME=$(hostname)

??????????????ETCDCTL_API=3

??????????????eps()?{
??????????????????EPS=""
??????????????????for?i?in?$(seq?0?$((${INITIAL_CLUSTER_SIZE}?-?1)));?do
??????????????????????EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379"
??????????????????done
??????????????????echo?${EPS}
??????????????}

??????????????member_hash()?{
??????????????????etcdctl?member?list?|?grep?-w?"$HOSTNAME"?|?awk?'{?print?$1}'?|?awk?-F?","?'{?print?$1}'
??????????????}

??????????????initial_peers()?{
??????????????????PEERS=""
??????????????????for?i?in?$(seq?0?$((${INITIAL_CLUSTER_SIZE}?-?1)));?do
????????????????????PEERS="${PEERS}${PEERS:+,}${SET_NAME}-${i}=http://${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380"
??????????????????done
??????????????????echo?${PEERS}
??????????????}

??????????????#?etcd-SET_ID
??????????????SET_ID=${HOSTNAME##*-}

??????????????#?adding?a?new?member?to?existing?cluster?(assuming?all?initial?pods?are?available)
??????????????if?[?"${SET_ID}"?-ge?${INITIAL_CLUSTER_SIZE}?];?then
??????????????????#?export?ETCDCTL_ENDPOINTS=$(eps)
??????????????????#?member?already?added?

??????????????????MEMBER_HASH=$(member_hash)
??????????????????if?[?-n?"${MEMBER_HASH}"?];?then
??????????????????????#?the?member?hash?exists?but?for?some?reason?etcd?failed
??????????????????????#?as?the?datadir?has?not?be?created,?we?can?remove?the?member
??????????????????????#?and?retrieve?new?hash
??????????????????????echo?"Remove?member?${MEMBER_HASH}"
??????????????????????etcdctl?--endpoints=$(eps)?member?remove?${MEMBER_HASH}
??????????????????fi

??????????????????echo?"Adding?new?member"

??????????????????echo?"etcdctl?--endpoints=$(eps)?member?add?${HOSTNAME}?--peer-urls=http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380"
??????????????????etcdctl?member?--endpoints=$(eps)?add?${HOSTNAME}?--peer-urls=http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380?|?grep?"^ETCD_"?>?/var/run/etcd/new_member_envs

??????????????????if?[?$??-ne?0?];?then
??????????????????????echo?"member?add?${HOSTNAME}?error."
??????????????????????rm?-f?/var/run/etcd/new_member_envs
??????????????????????exit?1
??????????????????fi

??????????????????echo?"==>?Loading?env?vars?of?existing?cluster..."
??????????????????sed?-ie?"s/^/export?/"?/var/run/etcd/new_member_envs
??????????????????cat?/var/run/etcd/new_member_envs
??????????????????.?/var/run/etcd/new_member_envs

??????????????????echo?"etcd?--name?${HOSTNAME}?--initial-advertise-peer-urls?${ETCD_INITIAL_ADVERTISE_PEER_URLS}?--listen-peer-urls?http://${POD_IP}:2380?--listen-client-urls?http://${POD_IP}:2379,http://127.0.0.1:2379?--advertise-client-urls?http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379?--data-dir?/var/run/etcd/default.etcd?--initial-cluster?${ETCD_INITIAL_CLUSTER}?--initial-cluster-state?${ETCD_INITIAL_CLUSTER_STATE}"

??????????????????exec?etcd?--listen-peer-urls?http://${POD_IP}:2380?\
??????????????????????--listen-client-urls?http://${POD_IP}:2379,http://127.0.0.1:2379?\
??????????????????????--advertise-client-urls?http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379?\
??????????????????????--data-dir?/var/run/etcd/default.etcd
??????????????fi

??????????????for?i?in?$(seq?0?$((${INITIAL_CLUSTER_SIZE}?-?1)));?do
??????????????????while?true;?do
??????????????????????echo?"Waiting?for?${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local?to?come?up"
??????????????????????ping?-W?1?-c?1?${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local?>?/dev/null?&&?break
??????????????????????sleep?1s
??????????????????done
??????????????done

??????????????echo?"join?member?${HOSTNAME}"
??????????????#?join?member
??????????????exec?etcd?--name?${HOSTNAME}?\
??????????????????--initial-advertise-peer-urls?http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2380?\
??????????????????--listen-peer-urls?http://${POD_IP}:2380?\
??????????????????--listen-client-urls?http://${POD_IP}:2379,http://127.0.0.1:2379?\
??????????????????--advertise-client-urls?http://${HOSTNAME}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379?\
??????????????????--initial-cluster-token?etcd-cluster-1?\
??????????????????--data-dir?/var/run/etcd/default.etcd?\
??????????????????--initial-cluster?$(initial_peers)?\
??????????????????--initial-cluster-state?new
??????????lifecycle:
????????????preStop:
??????????????exec:
????????????????command:
??????????????????-?/bin/sh
??????????????????-?-ec
??????????????????-?|
????????????????????HOSTNAME=$(hostname)

????????????????????member_hash()?{
????????????????????????etcdctl?member?list?|?grep?-w?"$HOSTNAME"?|?awk?'{?print?$1}'?|?awk?-F?","?'{?print?$1}'
????????????????????}

????????????????????eps()?{
????????????????????????EPS=""
????????????????????????for?i?in?$(seq?0?$((${INITIAL_CLUSTER_SIZE}?-?1)));?do
????????????????????????????EPS="${EPS}${EPS:+,}http://${SET_NAME}-${i}.${SET_NAME}.${MY_NAMESPACE}.svc.cluster.local:2379"
????????????????????????done
????????????????????????echo?${EPS}
????????????????????}

????????????????????export?ETCDCTL_ENDPOINTS=$(eps)
????????????????????SET_ID=${HOSTNAME##*-}

????????????????????#?Removing?member?from?cluster
????????????????????if?[?"${SET_ID}"?-ge?${INITIAL_CLUSTER_SIZE}?];?then
????????????????????????echo?"Removing?${HOSTNAME}?from?etcd?cluster"
????????????????????????etcdctl?member?remove?$(member_hash)
????????????????????????if?[?$??-eq?0?];?then
????????????????????????????#?Remove?everything?otherwise?the?cluster?will?no?longer?scale-up
????????????????????????????rm?-rf?/var/run/etcd/*
????????????????????????fi
????????????????????fi
??????????volumeMounts:
??????????-?mountPath:?/var/run/etcd
????????????name:?datadir
??volumeClaimTemplates:
??-?metadata:
??????name:?datadir
????spec:
??????accessModes:
??????-?"ReadWriteOnce"
??????resources:
????????requests:
??????????#?upstream?recommended?max?is?700M
??????????storage:?1Gi

修改完成后，我們?cè)?Kubernetes 集群中創(chuàng)建上面的幾個(gè)資源對(duì)象：

???etcd?git:(1e11e4a2108)???kubectl?apply?-f?pdb.yaml?????????????????????????????
poddisruptionbudget.policy/etcd-pdb?created
???etcd?git:(1e11e4a2108)???kubectl?apply?-f?service.yaml?
service/etcd?created
???etcd?git:(1e11e4a2108)???kubectl?apply?-f?statefulset.yaml?
statefulset.apps/etcd?created
???etcd?git:(1e11e4a2108)???kubectl?get?pods?-l?app=etcd?????
NAME?????READY???STATUS????RESTARTS???AGE
etcd-0???1/1?????Running???0??????????5m35s
etcd-1???1/1?????Running???0??????????5m12s
etcd-2???1/1?????Running???0??????????4m51s

“

需要注意上面的 StatefulSet 需要3個(gè)可用的 1G 容量 ReadWriteOnce 模式的 PV

”

現(xiàn)在我們可以看到 etcd 集群就啟動(dòng)成功了，同樣我們可以查看集群現(xiàn)在的狀態(tài)：

???etcd?git:(1e11e4a2108)???kubectl?exec?-it?etcd-0?/bin/sh??????????????????????????????????????????????????????????????????????????
#?etcdctl?--endpoints?etcd-0.etcd:2379,etcd-1.etcd:2379,etcd-2.etcd:2379?endpoint?status?--write-out=table
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?????ENDPOINT?????|????????ID????????|?VERSION?|?DB?SIZE?|?IS?LEADER?|?IS?LEARNER?|?RAFT?TERM?|?RAFT?INDEX?|?RAFT?APPLIED?INDEX?|?ERRORS?|
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?etcd-0.etcd:2379?|?2e80f96756a54ca9?|??3.4.13?|???20?kB?|??????true?|??????false?|?????????2?|??????????9?|??????????????????9?|????????|
|?etcd-1.etcd:2379?|?7fd61f3f79d97779?|??3.4.13?|???20?kB?|?????false?|??????false?|?????????2?|??????????9?|??????????????????9?|????????|
|?etcd-2.etcd:2379?|?b429c86e3cd4e077?|??3.4.13?|???20?kB?|?????false?|??????false?|?????????2?|??????????9?|??????????????????9?|????????|
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

我們可以嘗試去刪除其中一個(gè) Pod 來(lái)驗(yàn)證集群是否正常，Pod 重建后是否還可以加入到集群中去。比如我們執(zhí)行下面的命令將集群擴(kuò)容到5個(gè)副本：

???etcd?git:(1e11e4a2108)???kubectl?scale?--replicas=5?statefulset?etcd
???etcd?git:(1e11e4a2108)???kubectl?get?pods?-l?app=etcd???????????????
NAME?????READY???STATUS????RESTARTS???AGE
etcd-0???1/1?????Running???0??????????5m59s
etcd-1???1/1?????Running???0??????????5m52s
etcd-2???1/1?????Running???0??????????5m47s
etcd-3???1/1?????Running???0??????????4m
etcd-4???1/1?????Running???1??????????3m55s

此時(shí)我們?cè)偃ゲ榭醇旱臓顟B(tài)：

???etcd?git:(1e11e4a2108)???kubectl?exec?-it?etcd-0?/bin/sh????????????
#?etcdctl?--endpoints?etcd-0.etcd:2379,etcd-1.etcd:2379,etcd-2.etcd:2379,etcd-3.etcd:2379,etcd-4.etcd:2379?endpoint?status?--write-out=table
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?????ENDPOINT?????|????????ID????????|?VERSION?|?DB?SIZE?|?IS?LEADER?|?IS?LEARNER?|?RAFT?TERM?|?RAFT?INDEX?|?RAFT?APPLIED?INDEX?|?ERRORS?|
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?etcd-0.etcd:2379?|?c799a6ef06bc8c14?|??3.4.13?|???20?kB?|?????false?|??????false?|????????16?|?????????13?|?????????????????13?|????????|
|?etcd-1.etcd:2379?|?9869f0647883a00d?|??3.4.13?|???20?kB?|??????true?|??????false?|????????16?|?????????13?|?????????????????13?|????????|
|?etcd-2.etcd:2379?|?42c8b94265b9b79a?|??3.4.13?|???20?kB?|?????false?|??????false?|????????16?|?????????13?|?????????????????13?|????????|
|?etcd-3.etcd:2379?|?41eec5480dc0d9ec?|??3.4.13?|???20?kB?|?????false?|??????false?|????????16?|?????????13?|?????????????????13?|????????|
|?etcd-4.etcd:2379?|?ebbc833cba01ecad?|??3.4.13?|???20?kB?|?????false?|??????false?|????????16?|?????????13?|?????????????????13?|????????|
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

當(dāng)然同樣也可以去對(duì)其進(jìn)行縮容操作：

???etcd?git:(1e11e4a2108)???kubectl?scale?--replicas=3?statefulset?etcd
statefulset.apps/etcd?scaled
???etcd?git:(1e11e4a2108)???kubectl?get?pods?-l?app=etcd???????????????
NAME?????READY???STATUS????RESTARTS???AGE
etcd-0???1/1?????Running???0??????????11m
etcd-1???1/1?????Running???0??????????28s
etcd-2???1/1?????Running???0??????????23s
???etcd?git:(1e11e4a2108)???kubectl?exec?-it?etcd-0?/bin/sh????????????
#?etcdctl?--endpoints?etcd-0.etcd:2379,etcd-1.etcd:2379,etcd-2.etcd:2379?endpoint?status?--write-out=table
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?????ENDPOINT?????|????????ID????????|?VERSION?|?DB?SIZE?|?IS?LEADER?|?IS?LEARNER?|?RAFT?TERM?|?RAFT?INDEX?|?RAFT?APPLIED?INDEX?|?ERRORS?|
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|?etcd-0.etcd:2379?|?2e80f96756a54ca9?|??3.4.13?|???20?kB?|??????true?|??????false?|???????139?|?????????23?|?????????????????23?|????????|
|?etcd-1.etcd:2379?|?7fd61f3f79d97779?|??3.4.13?|???20?kB?|?????false?|??????false?|???????139?|?????????23?|?????????????????23?|????????|
|?etcd-2.etcd:2379?|?b429c86e3cd4e077?|??3.4.13?|???20?kB?|?????false?|??????false?|???????139?|?????????23?|?????????????????23?|????????|
+------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

現(xiàn)在又變回我們的集群模式了。