Thanos Receive 模塊介紹

起初（大概2020上半年之前）使用 thanos 作 prometheus分 布式監(jiān)控方案的時候，thanos receive 模塊還在試驗階段，當時是使用的 thanos sidecar 方式?，F(xiàn)在此功能模塊已經(jīng)被社區(qū)正式接受，功能會相對穩(wěn)定了，因此，考慮部分場景使用 receive 模塊替換 sidecar。

目標

通過使用receive模塊，期望做到：

• 收攏分散的prometheus采集數(shù)據(jù)，減少sidecar數(shù)量（如果thanos query后面掛過多sidecar會影響性能）
• 盡可能減少采集prometheus實例本地存儲數(shù)據(jù)量，使重啟、故障恢復時間更短

架構(gòu)介紹

架構(gòu)圖

                 +
Tenant's Premise | Provider Premise
                 |
                 |            +------------------------+
                 |            |                        |
                 |  +-------->+     Object Storage     |
                 |  |         |                        |
                 |  |         +-----------+------------+
                 |  |                     ^
                 |  | S3 API              | S3 API
                 |  |                     |
                 |  |         +-----------+------------+
                 |  |         |                        |       Store API
                 |  |         |  Thanos Store Gateway  +<-----------------------+
                 |  |         |                        |                        |
                 |  |         +------------------------+                        |
                 |  |                                                           |
                 |  +---------------------+                                     |
                 |                        |                                     |
+--------------+ |            +-----------+------------+              +---------+--------+
|              | | Remote     |                        |  Store API   |                  |
|  Prometheus  +------------->+     Thanos Receiver    +<-------------+  Thanos Querier  |
|              | | Write      |                        |              |                  |
+--------------+ |            +------------------------+              +---------+--------+
                 |                                                              ^
                 |                                                              |
+--------------+ |                                                              |
|              | |                PromQL                                        |
|    User      +----------------------------------------------------------------+
|              | |
+--------------+ |
                 +

功能說明

負載分發(fā)

為了支持多臺機器的擴展，時間序列將被分發(fā)到不同的receiver上。receiver是通過hash的方式來實現(xiàn)時間序列的持續(xù)分發(fā)的。每個receiver在hashring中的位置決定了哪些時間序列被哪個receiver接收和存儲。

remote write api過來的請求，經(jīng)過receiver前面的負載均衡，然后請求隨機落到receiver節(jié)點上。這時，會計算時間序列標簽的哈希值。另外，考慮到對于量很大的時間序列，不希望全都分發(fā)到某一個receiver節(jié)點上，通過參數(shù)--receive.tenant-header指定的tenant ID（如果沒有指定，默認為空字符串）也會用在hash值的計算中。大致的hash計算方式如下：

hash(string(tenant_id) + sort(timeseries.labelset).join())

如果receiver節(jié)點接收到的請求計算出來的hash值不是分發(fā)到當前節(jié)點，當前receiver會把他分發(fā)到該去的那個receiver節(jié)點。這個路由的功能本可以做成一個獨立的模塊，但是考慮到為了便于部署，這個功能是直接集成在receiver中了。

                                  Soft tenant hashring
                                 +-----------------------+
                                 |                       |
+-----------------+              |  +-----------------+  |
|                 |              |  |                 |  |
|  Load Balancer  +-------+      |  | Thanos receiver |  |
|                 |       |      |  |                 |  |
+-----------------+       |      |  +-----------------+  |
                          |      |                       |
                          |      |                       |
                          |      |  +-----------------+  |
                          |      |  |                 |  |
                          +-------->+ Thanos receiver +-----------+
                                 |  |                 |  |        |
                                 |  +-----------------+  |        |
                                 |                       |        |
                                 +-----------------------+        |
                                                                  |
                                   Hard Tenant A hashring         |
                                 +-----------------------+        |
                                 |                       |        |
                                 |  +-----------------+  |        |
                                 |  |                 |  |        |
                                 |  | Thanos receiver +<----------+
                                 |  |                 |  |        |
                                 |  +-----------------+  |        |
                                 |                       |        |
                                 |                       |        |
                                 |  +-----------------+  |        |
                                 |  |                 |  |        |
                                 |  | Thanos receiver +<----------+
                                 |  |                 |  |
                                 |  +-----------------+  |
                                 |                       |
                                 +-----------------------+

多副本

thanos receiver支持復制接受到的時間序列到hashring中的其他receiver。副本數(shù)由參數(shù)--receive.replication-factor來指定。如果被接收的時間序列，沒有寫入到(副本數(shù)+1)/2節(jié)點數(shù)（即要達到副本數(shù)半數(shù)以上），receiver將返回錯誤。

例如，要存儲3份時間序列的副本，則hashring中至少要有2個目標thanos receiver才行；并且，需要receiver配置此參數(shù)：--receive.replication-factor=3。

receiver節(jié)點的縮容/擴容/失敗場景

當發(fā)生擴縮容的時候，由于hashring發(fā)生變化，所有的節(jié)點需要將write-ahead-log的數(shù)據(jù)flush到TSDB塊并上傳到OSS中（如果配置了的話），因為這些節(jié)點之后將有一個新的分配。之前已存在節(jié)點上的時間序列不需要作調(diào)整，只是后面過來的請求按新的分發(fā)來尋找該去的receiver節(jié)點。注意，這種情況發(fā)生的flush可能會產(chǎn)生較小的TSDB塊，但thanos compactor模塊可以將它們優(yōu)化合并，因此不會有什么問題。

當有receiver節(jié)點發(fā)生故障時，prometheus的遠程寫會在后端目標無響應或503時進行重試，因此，receiver一定時間的服務掛掉是可以容忍的。如果這種掛機時間是不可接受的話，可以將副本數(shù)配置為3或以上，這樣即使有一個receiver節(jié)點掛掉，還有其他receiver節(jié)點來接收寫請求。

原文鏈接：https://blog.csdn.net/felix_yujing/article/details/108302167