開源工具集Carvel在CI/CD流水線中的集成

CI/CD流水線中使用VMware開源工具集Carvel

Carvel項目簡介

Carvel項目的主頁: https://carvel.dev/

簡介

• 一套適用于開發(fā)者和平臺運維者構建;分發(fā);安裝;管理K8s上容器化應用的工具集.
• 遵循Unix哲學^[1] Make each program do one thing well.

工具集中包含但不限于下圖幾款軟件:

它們分別在應用發(fā)布的生命周期中發(fā)揮著不同的作用:????

? ? Carvel每個工具執(zhí)行一項特定的功能, 這樣使用者就可以決定在哪個階段用哪個工具做哪個任務. 而不是做成一個單體式多功能的工具(如Helm). 這里對工具的使用并沒有好壞之分, 只要適用于用戶的環(huán)境和用戶的DevSecOps文化的就是好的工具.

????下文中我會對Carvel中的ytt; kbld以及kapp做進一步的介紹, 分別展示其功能及特性. 最后在一個Gitlab CI中將這些工具串聯(lián)起來, 打包一個Java Spring程序源碼成容器鏡像, 部署進Tanzu K8s集群.

YTT

????簡介 這工具名字取得...emmmm, 我心想就工程師就這么直男嗎, 八成是什么Yaml Template Tool啥的. 結果在一次Tanzu總工的視頻上聽到這ytt原意來自釔^[2]元素(自以為不直男的一次知識點的炫技).

????官方文檔對ytt^[3]有詳細的介紹, 我在這收斂一下重點, 然后再通過幾個列子演示一下.

• ytt識別近乎所有主流的Yaml配置(K8s Configuration, Concourse Pipeline, Docker Compose, GitHub Action workflow...)
• Yaml進, Yaml出
• 類Python語法

????一張ytt如何工作的示意圖, 感覺不是很直觀, 上用例.

????

????用例 我們用一個例子來展示ytt的多種功能, 內置變量, 外置變量, 簡單數(shù)學公式, 覆蓋變量. 首先我們制作一個K8s Deployment和Service的Yaml模版, 起名Deployment.yml:

#@?load("@ytt:data",?"data")

#@?def?labels():
app:?"spring-demo"
team:?"dev"
#@?end
---
apiVersion:?apps/v1
kind:?Deployment
metadata:
??name:?spring-sample-demo
??labels:?#@?labels()
spec:
??replicas:?#@?data.values.replicas?/?2
??selector:
????matchLabels:?#@?data.values.labels1
??template:
????metadata:
??????labels:?#@?data.values.labels1
????spec:
??????containers:
????????-?image:?#@?data.values.image1
??????????name:?spring-sample-app
??????????ports:
????????????-?containerPort:?#@?data.values.app_port
??????????????name:?http
---
apiVersion:?v1
kind:?Service
metadata:
??name:?spring-sample-demo-svc
spec:
??selector:
????app:?spring-demo
??type:?NodePort
??ports:
??-?port:?#@?data.values.svc_port
????targetPort:?#@?data.values.app_port
????nodePort:?30028

#@ load("@ytt:data", "data") 是告訴ytt加載外置變量來自values.yml.#@ def labels(): 是Yaml模版內變量, 定義了兩個標簽app: "spring-demo", team: "dev"

????在有效K8s字段中,我們讓Deployment的標簽來自內置變量labels(), 而ReplicaSet和Selector標簽從外置變量values.yml中讀取labels1. 副本數(shù)量replicas: #@ data.values.replicas / 2 做一個簡單的除法. 其他的變量的Patch原則如出一轍.

????再看values.yml: 以#@data/values, 告訴ytt渲染成values. 類似Helm中的values, 作為運維者或者流水線制定者在該文件中通過注釋讓開發(fā)者填寫與應用相關的參數(shù). 真正部署到K8s里的配置文件通過ytt渲染生成.

#@data/values
---
replicas:?8?#需要部署幾個實例
svc_port:?80?#集群內暴露的服務端口
app_port:?8080?#應用監(jiān)聽的端口
image1:?""?#實例的容器鏡像名字
#?一些需要的標簽
labels1:
??app:?"spring-demo"
??cluster:?"tce-mc"

????這里我故意把values中image1的值設為空, 因為我希望通過覆蓋方式填寫進去, 比如這個鏡像名字是來自CI流水線中的內置變量之類的.

執(zhí)行 接下來我們在命令行中執(zhí)行:

#?演示用途,?設置一個環(huán)境變量IMAGE1
export?IMAGE1=docker.io/rock981119/spring-sample:ci-main

#?執(zhí)行ytt
ytt?-f?Deployment.yml?-f?values.yml?-v?image1=$IMAGE1

????通過-v image1=$IMAGE1覆蓋掉values.yml里的image1的值, 那么總體的輸出結果為:

apiVersion:?apps/v1
kind:?Deployment
metadata:
??name:?spring-sample-demo
??labels:
????app:?spring-demo
????team:?dev
spec:
??replicas:?4
??selector:
????matchLabels:
??????app:?spring-demo
??????cluster:?tce-mc
??template:
????metadata:
??????labels:
????????app:?spring-demo
????????cluster:?tce-mc
????spec:
??????containers:
??????-?image:?docker.io/rock981119/spring-sample:ci-main
????????name:?spring-sample-app
????????ports:
????????-?containerPort:?8080
??????????name:?http
---
apiVersion:?v1
kind:?Service
metadata:
??name:?spring-sample-demo-svc
spec:
??selector:
????app:?spring-demo
??type:?NodePort
??ports:
??-?port:?80
????targetPort:?8080
????nodePort:?30028

????這樣不管是輸出成文件還是通過管道符交給Kubectl apply -f-執(zhí)行都是可以的. 通過這個例子我們簡單小結一下, ytt懂得Yaml格式, 不同于Shell中使用sed或其他模版語言只能替換固定位置的變量. 語法也比較精簡, 有Python基礎即可.

????Python語法渲染這個點是ytt重點優(yōu)點里我最喜歡的, 讓我這種只會點Python的人很快就能用起來. 其次就是現(xiàn)在的工具實在是太多了, 考慮學習成本迫使我學習心態(tài)變得更功利. 例如之前學習的Terraform的時候, 我就很反感HashiCorp非要再搞出個HCL的語法. 學OPA的時候要學習Rego語法... 而這些語法除了這些獨立的場景外別的地方都用不著. 官網(wǎng)有ytt對比其他模版工具的觀點^[4], 歡迎查閱.

????如果是自定義的K8s CRD, ytt也可以通過寫Schema來渲染你要的Yaml層級, 篇幅問題就不演示更多的例子了, 更多更詳細的說明在官網(wǎng)的文檔中都有用例.

KBLD

簡介kbld^[5] (pronounced: kei·bild):

• 鏡像構建(委派 Docker, pack, kubectl-buildkit等工具)和推送;搬遷的編排工具
• 解析Yaml中鏡像的摘要, 渲染出新的Yaml將鏡像的值替換成該鏡像的摘要(而不是某個tag或latest), 確保調用的鏡像是不可變的

????用例1 在Kubernetes安全原則中經(jīng)常提及到: 永遠不要使用image:latest. 其實就算你使用的是image:tag, 你也不能保證相同的tag的鏡像已經(jīng)被更改. 主流鏡像倉庫(如DockerHub; Harbor)支持引用鏡像時采用哈希值摘要.

????

????即便鏡像的便簽不變, 只要構建時任何一層發(fā)生變化, 其哈希摘要都會發(fā)生變化. 剛才ytt渲染后的Yaml已經(jīng)是我在Docker Hub上真實上傳的鏡像名+Tag. 我們看看ytt渲染后的Yaml再交給kbld渲染一次會如何. 執(zhí)行: ytt -f Deployment.yml -f values.yml -v image1=$IMAGE1 | kbld -f -

resolve?|?final:?docker.io/rock981119/spring-sample:ci-main?->?index.docker.io/rock981119/spring-sample@sha256:e901302da31edf61cbaec68df230e8b0f5cc33932d43337f9dac8a548ff23b7e
---
apiVersion:?apps/v1
kind:?Deployment
metadata:
??annotations:
????kbld.k14s.io/images:?|
??????-?origins:
????????-?resolved:
????????????tag:?ci-main
????????????url:?docker.io/rock981119/spring-sample:ci-main
????????url:?index.docker.io/rock981119/spring-sample@sha256:e901302da31edf61cbaec68df230e8b0f5cc33932d43337f9dac8a548ff23b7e
#?省略
????spec:
??????containers:
??????-?image:?index.docker.io/rock981119/spring-sample@sha256:e901302da31edf61cbaec68df230e8b0f5cc33932d43337f9dac8a548ff23b7e
#省略

---
apiVersion:?v1
kind:?Service
#省略

Succeeded

????kbld只能解析到鏡像倉庫可達并正確的鏡像名, 比如你自己瞎寫一個或者只是本地的鏡像它則不能解析了. 除了解析鏡像的哈希摘要外, kbld可以同時委派Docker或者Pack來構建鏡像, 你也可以一步把構建的鏡像再推到倉庫里.

????用例2 構建鏡像并推送至倉庫, 文件名kbld.yml

---
apiVersion:?kbld.k14s.io/v1alpha1
kind:?Config
sources:
-?image:?docker.io/rock981119/spring-sample:ci-main
??path:?apps/java-maven/
??pack:
????build:
??????builder:?paketobuildpacks/builder:tiny

---
apiVersion:?kbld.k14s.io/v1alpha1
kind:?Config
destinations:
-?image:?docker.io/rock981119/spring-sample:ci-main
??newImage:?docker.io/rock981119/spring-sample
??tags:?[latest,?ci-main]

sources配置就是告訴kbld委派Pack構建本地鏡像docker.io/rock981119/spring-sample:ci-main, path告訴Pack源碼的位置, 你可以單獨指定builder. 如果你的Pack設置了默認的builder也可以不指定. 關于kbld委派其他的構建工具可以參考官網(wǎng)文檔^[6].

destinations告訴kbld將哪個本地鏡像重命名并推送至倉庫, 推送時可以加若干標簽, 他們的哈希摘要都是一致的.

????kbld不能直接對kbld.yml生效, 因為它的目的本身并不在于構建鏡像和推送, 而是攝取正確的鏡像哈希摘要填入到需要部署的Yaml中去.

????承接上一個ytt的用例, 我們繼續(xù)聯(lián)合兩個工具一起工作, ytt渲染配置, kbld委派構建鏡像并推送至倉庫, 最終渲染出部署Yaml

ytt?-f?Deployment.yml?-f?values.yml?-v?image1=$IMAGE1?|?kbld?-f?kbld.yml?-f?-

????最終輸出的Yaml跟kbld的第一個用例是類似的, 但在你的Console中它已經(jīng)完成了委派鏡像構建和推送.

KAPP

????ytt和kbld的實質任務都是渲染出最終可用于部署的Yaml配置, 是時候該找個工具部署了. 事實上渲染后的Yaml文件可以直接用kubectl或者helm來部署了. 那我們再看看Carvel中的kapp又有什么不同.

這個官網(wǎng)^[7]的一句話定義:

Deploy and view groups of Kubernetes resources as "applications". Apply changes safely and predictably, watching resources as they converge.

????簡單來說當你部署一個Yaml Bundle的時候, 內容里包含了若干CRD, Deployment, Service, Ingress等, 其實它們是同一個應用. 通過kapp部署這個Yaml Bundle, kapp會將它實例化為一個app(自定義名字).

• apply階段識別Yaml中資源的部署先后順序
• apply階段告知創(chuàng)建了何種配置
• diff階段以git格式告知產(chǎn)生了何種變化
• 樹狀展示資源

????用例?我們用kapp將ytt和kbld渲染后的Yaml部署到已建好的Tanzu Kubernetes集群當中去:

ytt?-f?Deployment.yml?-f?values.yml?-v?image1=$IMAGE1?|?kbld?-f?kbld.yml?-f?-?|?kapp?deploy?-a?spring-demo?-c?-y?-f?-
#輸出省略
@@?update?deployment/spring-sample-demo?(apps/v1)?namespace:?default?@@
??...
?16,?16???????????????-?ci-main
?17?????-?????????url:?index.docker.io/rock981119/spring-sample@sha256:feb86d80c50f95d269c6cb331ae3a36f6e3585b04ded0ff9aff1ad65edc974f7
?????17?+?????????url:?index.docker.io/rock981119/spring-sample@sha256:f8fc4021d58af607f5ef15a31091cdc759a2b958981a8fe9792654376e3e39c8
?18,?18?????creationTimestamp:?"2022-01-29T06:28:57Z"
?19,?19?????generation:?4
??...
138,138?????????containers:
139?????-???????-?image:?index.docker.io/rock981119/spring-sample@sha256:feb86d80c50f95d269c6cb331ae3a36f6e3585b04ded0ff9aff1ad65edc974f7
????139?+???????-?image:?index.docker.io/rock981119/spring-sample@sha256:f8fc4021d58af607f5ef15a31091cdc759a2b958981a8fe9792654376e3e39c8
140,140???????????name:?spring-sample-app
141,141???????????ports:

Changes

Namespace??Name????????????????Kind????????Conds.??Age??Op??????Op?st.??Wait?to????Rs??Ri??
default????spring-sample-demo??Deployment??2/2?t???14m??update??-???????reconcile??ok??-??

Op:??????0?create,?0?delete,?1?update,?0?noop,?0?exists
Wait?to:?1?reconcile,?0?delete,?0?noop
#輸出省略

????因為演示, kapp部署時, -a為應用實例起名spring-demo, 我apply了多次, 用例中kapp輸出了與上次變更了的內容. kapp ls羅列當前通過kapp部署的實例. kapp inspect -a spring-demo樹狀展示spring-demo實例的資源關系以及同步狀態(tài).

kapp?inspect?-a?spring-demo
Target?cluster?'https://192.168.31.100:6443'?(nodes:?tce-management-control-plane-86792,?1+)
#省略部分告警
Resources?in?app?'spring-demo'

Namespace??Name?????????????????????????????????Kind???????????Owner????Conds.??Rs??Ri??Age??
default????spring-sample-demo???????????????????Deployment?????kapp?????2/2?t???ok??-???19m??
^??????????spring-sample-demo-65b8d65957????????ReplicaSet?????cluster??-???????ok??-???4m??
^??????????spring-sample-demo-65b8d65957-brmpg??Pod????????????cluster??4/4?t???ok??-???4m??
^??????????spring-sample-demo-65b8d65957-jx2gw??Pod????????????cluster??4/4?t???ok??-???4m??
^??????????spring-sample-demo-65b8d65957-ls2nt??Pod????????????cluster??4/4?t???ok??-???4m??
^??????????spring-sample-demo-65b8d65957-vzn6b??Pod????????????cluster??4/4?t???ok??-???4m??
^??????????spring-sample-demo-667859679?????????ReplicaSet?????cluster??-???????ok??-???19m??
^??????????spring-sample-demo-6774567466????????ReplicaSet?????cluster??-???????ok??-???15m??
^??????????spring-sample-demo-svc???????????????Endpoints??????cluster??-???????ok??-???19m??
^??????????spring-sample-demo-svc???????????????Service????????kapp?????-???????ok??-???19m??
^??????????spring-sample-demo-svc-ncrlx?????????EndpointSlice??cluster??-???????ok??-???19m??

Rs:?Reconcile?state
Ri:?Reconcile?information

11?resources

????kapp刪除實例, 在Op列中可以看到, 實際刪除的就是Deployment和Service.

kapp?delete?-a?spring-demo
Changes

Namespace??Name?????????????????????????????????Kind???????????Conds.??Age??Op??????Op?st.??Wait?to??Rs??Ri??
default????spring-sample-demo???????????????????Deployment?????2/2?t???21m??delete??-???????delete???ok??-??
^??????????spring-sample-demo-65b8d65957????????ReplicaSet?????-???????7m???-???????-???????delete???ok??-??
^??????????spring-sample-demo-65b8d65957-brmpg??Pod????????????4/4?t???7m???-???????-???????delete???ok??-??
^??????????spring-sample-demo-65b8d65957-jx2gw??Pod????????????4/4?t???7m???-???????-???????delete???ok??-??
^??????????spring-sample-demo-65b8d65957-ls2nt??Pod????????????4/4?t???7m???-???????-???????delete???ok??-??
^??????????spring-sample-demo-65b8d65957-vzn6b??Pod????????????4/4?t???7m???-???????-???????delete???ok??-??
^??????????spring-sample-demo-667859679?????????ReplicaSet?????-???????21m??-???????-???????delete???ok??-??
^??????????spring-sample-demo-6774567466????????ReplicaSet?????-???????17m??-???????-???????delete???ok??-??
^??????????spring-sample-demo-svc???????????????Endpoints??????-???????21m??-???????-???????delete???ok??-??
^??????????spring-sample-demo-svc???????????????Service????????-???????21m??delete??-???????delete???ok??-??
^??????????spring-sample-demo-svc-ncrlx?????????EndpointSlice??-???????21m??-???????-???????delete???ok??-??

Op:??????0?create,?2?delete,?0?update,?9?noop,?0?exists
Wait?to:?0?reconcile,?11?delete,?0?noop

Carvel in CI/CD

????簡介完這三個Carvel的工具后, 我們基本就可以用它們完成一個源碼到鏡像, 生成配置再到實例化部署的一個過程了. 那么我們把這個過程在CI/CD流水線中實踐一下.

源碼用的是Buildpack官方的例子Buildpack官方的例子源碼

1. 上傳本地Gitlab Server. 如果你想安裝Gitlab CE私有環(huán)境可以考慮我司在Bitnami上打包好的虛擬機鏡像Bitnami Gitlab CE OVA^[8], 還附有設置文檔.
2. 制作Deplpoyment.yml模版, values.yml, kbld.yml
3. 為該項目注冊一個Runner, 為了方便選用了Photon Linux, Shell執(zhí)行模式
4. 構建Gitlab Pipeline文件
5. 驗證??
6. 拓撲與流程如上圖, 下方是我的Gitlab Pipeline配置文件, .gitlab-ci.yml:

#?This?file?is?a?template,?and?might?need?editing?before?it?works?on?your?project.
#?To?contribute?improvements?to?CI/CD?templates,?please?follow?the?Development?guide?at:
#?https://docs.gitlab.com/ee/development/cicd/templates.html
#?This?specific?template?is?located?at:
#?https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Getting-Started.gitlab-ci.yml

#?This?is?a?sample?GitLab?CI/CD?configuration?file?that?should?run?without?any?modifications.
#?It?demonstrates?a?basic?3?stage?CI/CD?pipeline.?Instead?of?real?tests?or?scripts,
#?it?uses?echo?commands?to?simulate?the?pipeline?execution.
#
#?A?pipeline?is?composed?of?independent?jobs?that?run?scripts,?grouped?into?stages.
#?Stages?run?in?sequential?order,?but?jobs?within?stages?run?in?parallel.
#
#?For?more?information,?see:?https://docs.gitlab.com/ee/ci/yaml/index.html#stages

stages:??????????#?List?of?stages?for?jobs,?and?their?order?of?execution
??-?test
??-?build
??-?deploy

test:
??stage:?test
??tags:
????-?linux
??script:
????-?echo?"測了,?但沒完全測"

build:
??stage:?build
??needs:?["test"]
??variables:
????IMAGE1:?"docker.io/rock981119/spring-sample:ci-main"
??tags:
????-?linux
??before_script:
????-?echo?"Before?Script?For?Docker?Hub?Login"
????-?echo?$DOCKER_PWD?|?docker?login?--username?$DOCKER_USER?--password-stdin
??script:
????-?echo?"?---->?使用ytt和kbld渲染Yaml配置"
????-?echo?"?kbld委派Pack構建鏡像并推送至倉庫"
????-?ytt?-f?Deployment.yml?-f?values.yml?-v?image1=$IMAGE1?|?kbld?-f?kbld.yml?-f?-?>>?app.yml
??artifacts:
????paths:
??????-?app.yml

deploy2tanzu:
??stage:?deploy
??needs:?["build"]
??tags:
????-?linux
??when:?manual
??script:
????-?echo?"使用kapp部署應用實例"
????-?kapp?deploy?-a?$CI_PROJECT_NAME?-c?-y?-f?app.yml

????查看一下結果吧

ci-with-carvel?[main]?kapp?ls
Target?cluster?'https://192.168.31.100:6443'?(nodes:?tce-management-control-plane-86792,?1+)

Apps?in?namespace?'default'

Name????????????Namespaces??Lcs???Lca??
ci-with-carvel??default?????true??1m??
tce-repo-ctrl???default?????true??1m??

Lcs:?Last?Change?Successful
Lca:?Last?Change?Age

2?apps

Succeeded
ci-with-carvel?[main]?kapp?inspect?-a?ci-with-carvel

Resources?in?app?'ci-with-carvel'

Namespace??Name?????????????????????????????????Kind???????????Owner????Conds.??Rs??Ri??Age??
default????spring-sample-demo???????????????????Deployment?????kapp?????2/2?t???ok??-???34m??
^??????????spring-sample-demo-6798f54d9c????????ReplicaSet?????cluster??-???????ok??-???2m??
^??????????spring-sample-demo-6798f54d9c-grm5k??Pod????????????cluster??4/4?t???ok??-???1m??
^??????????spring-sample-demo-6798f54d9c-q88tn??Pod????????????cluster??4/4?t???ok??-???1m??
^??????????spring-sample-demo-6798f54d9c-rxr5w??Pod????????????cluster??4/4?t???ok??-???2m??
^??????????spring-sample-demo-6798f54d9c-smf6j??Pod????????????cluster??4/4?t???ok??-???2m??
^??????????spring-sample-demo-746b764476????????ReplicaSet?????cluster??-???????ok??-???34m??
^??????????spring-sample-demo-svc???????????????Endpoints??????cluster??-???????ok??-???34m??
^??????????spring-sample-demo-svc???????????????Service????????kapp?????-???????ok??-???34m??
^??????????spring-sample-demo-svc-8md44?????????EndpointSlice??cluster??-???????ok??-???34m??

Rs:?Reconcile?state
Ri:?Reconcile?information

10?resources

kubectl?get?all
NAME??????????????????????????????????????READY???STATUS????RESTARTS???AGE
pod/spring-sample-demo-6798f54d9c-grm5k???1/1?????Running???0??????????2m26s
pod/spring-sample-demo-6798f54d9c-q88tn???1/1?????Running???0??????????2m28s
pod/spring-sample-demo-6798f54d9c-rxr5w???1/1?????Running???0??????????2m42s
pod/spring-sample-demo-6798f54d9c-smf6j???1/1?????Running???0??????????2m43s

NAME?????????????????????????????TYPE????????CLUSTER-IP???????EXTERNAL-IP???PORT(S)????????AGE
service/kubernetes???????????????ClusterIP???100.64.0.1???????????????443/TCP????????4d2h
service/spring-sample-demo-svc???NodePort????100.65.232.184???????????80:30028/TCP???35m

NAME?????????????????????????????????READY???UP-TO-DATE???AVAILABLE???AGE
deployment.apps/spring-sample-demo???4/4?????4????????????4???????????35m

NAME????????????????????????????????????????????DESIRED???CURRENT???READY???AGE
replicaset.apps/spring-sample-demo-6798f54d9c???4?????????4?????????4???????2m43s
replicaset.apps/spring-sample-demo-746b764476???0?????????0?????????0???????35m

目標達到了，看起來不錯.

才疏學淺,只把玩了一下Carvel Toolset的小功能, 希望對大家DevSecOps的日常有幫助, 多多支持與關注我們VMware的開源項目.

結語

????結語就不上什么價值觀了, 這一年比較懶, 沒輸出啥內容, 希望明年能更努力. 最后祝大家虎年行大運, 走出一個虎虎生風.

參考資料