一、升级最稳妥的 LTS 逐级升级（推荐） 先补齐本线最后一个补丁：2.516.3（把 2.516 线能拿到的修复先吃满） 升到下一条 LTS 基线：2.528.3（过渡 LTS 线） 升到当前在维护的 LTS：2.541.1（截至 2026-02-05 仍是主流 LTS） 这么做的好处：每一步跨度小，遇到插件/鉴权问题更容易定位；并且升级指南也建议如果跳 LTS，要把中间每段的升级注意事项都看一遍。root@k8s-01:/woke/jenkins# cat deployment.yaml # jenkins-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: jenkins spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins spec: securityContext: fsGroup: 1000 serviceAccountName: jenkins-admin containers: - name: jenkins image: registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17:lts-jdk17 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 - containerPort: 50000 # ★ JVM 参数 env: - name: JENKINS_JAVA_OPTIONS value: "-Djava.net.preferIPv4Stack=true" - name: JAVA_OPTS value: "-Djava.net.preferIPv4Stack=true" volumeMounts: - name: jenkins-data mountPath: /var/jenkins_home # ★ 新增挂载，把每个节点上的 /root/cicd 挂进 Jenkins 家目录 - name: cicd-tools mountPath: /var/jenkins_home/cicd resources: limits: cpu: "1" memory: "4Gi" requests: cpu: "0.5" memory: "1Gi" livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 90 periodSeconds: 10 volumes: - name: jenkins-data persistentVolumeClaim: claimName: jenkins-pvc # ★ 新增 volume，使用 hostPath 指向每个节点的 /root/cicd - name: cicd-tools hostPath: path: /root/cicd type: Directory # 如果想让 k8s 自动创建目录，也可以用： # type: DirectoryOrCreate
一、Jenkins动态slave介绍 1.1为什么需要动态slave1. 配置管理困难：不同项目可能使用不同的编程语言、框架或库，这导致了每个Slave的配置环境各不相同。因此，需要动态Slave能够根据不同的项目需求，灵活配置不同的运行环境，从而简化配置管理和维护工作。 2. 资源分配不均衡：在使用静态Slave时，可能会出现某些Slave处于空闲状态，而其他Slave却处于繁忙状态，导致资源分配不均衡。动态Slave可以根据当前任务的需求自动调配资源，使得任务能够在空闲的Slave上尽快执行，从而提高资源利用率和任务执行效率。 3. 资源浪费：静态Slave在没有任务执行时仍然占用着资源，这导致了资源的浪费。而动态Slave能够根据实际需要自动扩容或缩减，当没有任务执行时会释放资源，从而避免了资源的浪费。1.2动态slave工作流程正因为上面的这些种种痛点，我们渴望一种更高效更可靠的方式来完成这个 CI/CD 流程，而 Docker虚拟化容器技术能很好的解决这个痛点，又特别是在 Kubernetes 集群环境下面能够更好来解决上面的问题，下图是基于 Kubernetes 搭建 Jenkins 集群的简单示意图：从图上可以看到 Jenkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。 这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。二、服务部署本项目所有服务均运行在k8s集群上，使用nfs共享存储 nfs共享存储部署 container部署 harbor部署 gitlab部署 jenkins部署 SonarQube部署三、项目与权限配置 3.1Harbor配置Harbor的项目分为公开和私有的: 公开项目:所有用户都可以访问，通常存放公共的镜像，默认有一个library公开项目。 私有项目:只有授权用户才可以访问，通常存放项目本身的镜像。 我们可以为微服务项目创建一个新的项目创建用户 创建一个普通用户xing。配置项目用户权限 在spring_boot_demo项目中添加普通用户xing，并设置角色为开发者。权限说明角色权限访客对项目有只读权限开发人员对项目有读写权限维护人员对项目有读写权限、创建webhook权限项目管理员出上述外，还有用户管理等权限3.2gitlab项目权限配置创建组 管理员用户登录，创建群组，组名称为develop，组权限为私有创建项目 创建sprint boot demo项目，并指定develop，项目类型为私有创建用户 创建一个普通用户xing用户添加到组中 将xing添加到群组develop中，cuiliang角色为Developer配置分支权限用户权限验证 使用任意一台机器模拟开发人员拉取代码，完成开发后推送至代码仓库。 拉取仓库代码#拉取代码 root@k8s-03:~/work# git clone https://gitee.com/cuiliang0302/sprint_boot_demo.git Cloning into 'sprint_boot_demo'... remote: Enumerating objects: 261, done. remote: Total 261 (delta 0), reused 0 (delta 0), pack-reused 261 (from 1) Receiving objects: 100% (261/261), 105.79 KiB | 162.00 KiB/s, done. Resolving deltas: 100% (116/116), done. root@k8s-03:~/work# git remote set-url origin http://192.168.30.181/develop/sprint-boot-demo.git fatal: not a git repository (or any of the parent directories): .git推送至gitlab仓库root@k8s-03:~/work/sprint_boot_demo# git remote set-url origin http://192.168.30.181/develop/sprint-boot-demo.git root@k8s-03:~/work/sprint_boot_demo# git remote -v origin http://192.168.30.181/develop/sprint-boot-demo.git (fetch) origin http://192.168.30.181/develop/sprint-boot-demo.git (push) root@k8s-03:~/work/sprint_boot_demo# git push --set-upstream origin --all Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': Enumerating objects: 254, done. Counting objects: 100% (254/254), done. Delta compression using up to 8 threads Compressing objects: 100% (119/119), done. Writing objects: 100% (254/254), 105.20 KiB | 105.20 MiB/s, done. Total 254 (delta 111), reused 253 (delta 110), pack-reused 0 remote: Resolving deltas: 100% (111/111), done. remote: remote: To create a merge request for master, visit: remote: http://192.168.30.181/develop/sprint-boot-demo/-/merge_requests/new?merge_request%5Bsource_branch%5D=master remote: To http://192.168.30.181/develop/sprint-boot-demo.git * [new branch] master -> master Branch 'master' set up to track remote branch 'master' from 'origin'. 查看验证 四、jenkins配置 4.1插件安装与配置GitLab插件安装与配置：https://www.cuiliangblog.cn/detail/section/127410630 SonarQube Scanner插件安装与配置：https://www.cuiliangblog.cn/detail/section/165534414 Kubernetes插件安装与配置：https://www.cuiliangblog.cn/detail/section/127230452 Email Extension邮件推送插件安装与配置：https://www.cuiliangblog.cn/detail/section/133029974 Version Number版本号插件安装与配置：https://plugins.jenkins.io/versionnumber/ Content Replace文件内容替换插件安装与配置：https://plugins.jenkins.io/content-replace/4.2jenkins slave镜像制作安装完Kubernetes插件后，默认的slave镜像仅包含一些基础功能和软件包，如果需要构建镜像，执行kubectl命令，则需要引入其他container或者自定义slave镜像。 关于镜像构建问题，如果k8s容器运行时为docker，可以直接使用docker in docker方案，启动一个docker:dind容器，通过Docker pipeline插件执行镜像构建与推送操作，具体内容可参考https://www.cuiliangblog.cn/detail/section/166573065。 如果k8s容器运行时为container，则使用nerdctl+buildkitd方案，启动一个buildkit容器，通过nerdctl命令执行镜像构建与推送操作，具体内容可参考:https://axzys.cn/index.php/archives/521/ 本次实验以container环境为例，通过nerdctl+buildkitd方案演示如何构建并推送镜像。构建jenkins-slave镜像root@k8s-01:~/jenkins/work# cp /usr/bin/kubectl . root@k8s-01:~/jenkins/work# cp /usr/bin/nerdctl . root@k8s-01:~/jenkins/work# cp /usr/local/bin/buildctl . root@k8s-01:~/jenkins/work# ls buildctl Dockerfile kubectl nerdctl测试jenkins-slave镜像构建容器与操作k8s 以下操作在k8s集群master机器，容器运行时为container节点执行测试# 启动buildkit镜像构建服务 # 挂载/run/containerd/containerd.sock方便container调用buildkitd # 挂载/var/lib/buildkit，以便于将构建过程中下载的镜像持久化存储，方便下次构建时使用缓存 # 挂载/run/buildkit/目录方便nerctl调用buildkitd root@k8s-03:~/bin# nerdctl run --name buildkit -d --privileged=true \ -v /run/buildkit/:/run/buildkit/ \ -v /var/lib/buildkit:/var/lib/buildkit \ -v /run/containerd/containerd.sock:/run/containerd/containerd.sock \ registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2 registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2: resolved |++++++++++++++++++++++++++++++++++++++| manifest-sha256:ff1ed58245d6871cc4bbe07a838603720a90ca33f124484fff2af61b344c3b2f: done |++++++++++++++++++++++++++++++++++++++| config-sha256:b67e88949be2f8ee8844bbe205c8b0054533a2c8cf35a6bc39ebc1d7cd7ce8f1: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:1479f6e1da81ab38384f704f742a35030364df4c9f9ed65e812a9c921bc20d25: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:77e3817cafb06118d96cfbd8af2fb7834a03e14d83acbcd9b7ff2a298c292d4d: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:fabeb51fadd96525499e8ce7171956d81bf70c836d4943ff8be9714528da736a: done |++++++++++++++++++++++++++++++++++++++| elapsed: 19.4s total: 87.2 M (4.5 MiB/s) 661d91a4ade1379768948ea962541ce5876e4dc51c14d3673fedfbdb7d142af5 root@k8s-03:~/bin# nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4778573cadf3 registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2 "buildkitd" About a minute ago Up buildkit root@k8s-03:~/bin# # 启动jenkins-slave容器 # 挂载/run/containerd/containerd.sock方便netdctl操作container # 挂载/run/buildkit/目录方便nerctl调用buildkitd构建镜像 # 挂载/root/.kube/目录方便kubectl工具操作k8s root@k8s-03:~/bin# nerdctl run --name jenkins-slave -it --privileged=true -v /run/buildkit/:/run/buildkit/ -v /root/.kube/:/root/.kube/ -v /run/containerd/containerd.sock:/run/containerd/containerd.sock registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 bash registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1: resolved |++++++++++++++++++++++++++++++++++++++| manifest-sha256:b3e519ae85d0f05ff170778c8ffae494879397d9881ca8bc905bc889da82fc07: done |++++++++++++++++++++++++++++++++++++++| config-sha256:ad852c7e884a5f9f6e87fcb6112fbe0c616b601a69ae5cf74ba09f2456d4e578: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:9ac051fdbd99f7d8c9e496724860b8ae3373f24d6f8a54f1d9096526df425d3c: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:85a4e35755fb1aa44b91602297dc8d9f10eb8ad3f32baab32094cebc0eda41a4: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:0bdf2c4d1714b5962675435789b4e83edb5aa4d94ec0d7643737940b0e73c4ed: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:6b189407e830325d112140133ef8a0adcae1a94a9231cfa56872c78f21886b66: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:28f95146d6851ca39a2ce18612cb5e5b19845ef85e4bb95bfa3095193fdf5777: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:68a03bb16ee6a2c356d2e354bab5ec566dd7ef1e4e6daee52c1f87aa9d0cd139: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:1c55318e78a1d3f438c6ca3cf6532d365a86fabbf7e3ecd14140e455f1489991: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:65375761a96d26587431452e982c657d479c41c41027a7f0acf37a6a21fd1112: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:f26c0daf8d2d5ff437f0eb85dad34fe122657585c3e8e5589fdf4cd007705fbe: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:cef14de45bb7cc343e593f80531764848fe724db9084ae4a3cabacc7a7e24083: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:7db9f9afd5f7e18ef5e410566fe6342de266960271c5ba712b8027e593617498: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:392ad068a4827c711e9af3956bd5d3bcac7e8d66b3dd35bb0a9fae60de47f80d: done |++++++++++++++++++++++++++++++++++++++| elapsed: 4.9 s total: 170.8 (34.9 MiB/s) root@4da1c7c8f6f4:/home/jenkins# # 测试container管理 root@4da1c7c8f6f4:/home/jenkins# nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4da1c7c8f6f4 registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 "/usr/local/bin/jenk…" 47 seconds ago Up jenkins-slave 4778573cadf3 registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2 "buildkitd" 6 minutes ago Up buildkit root@4da1c7c8f6f4:/home/jenkins# #测试k8s管理 root@4da1c7c8f6f4:/home/jenkins# kubectl get node NAME STATUS ROLES AGE VERSION k8s-01 Ready control-plane 5d v1.27.6 k8s-02 Ready <none> 5d v1.27.6 k8s-03 Ready <none> 5d v1.27.6 # 测试镜像构建解释一下下面的操作，如果是宿主机已经开启了buildkitd服务必须先关闭，不然/var/lib/buildkit/buildkitd.lock会生成，这个生成后Docker启动不了buildkitd容器。#停止buildkitd服务 root@k8s-03:/var/lib/buildkit# sudo systemctl stop buildkitd sudo systemctl disable buildkitd # 2. 清理残留文件 sudo rm -f /run/buildkit/buildkitd.sock sudo rm -f /run/buildkit/otel-grpc.sock sudo rm -f /var/lib/buildkit/buildkitd.lock # 3. 创建专用数据目录（避免冲突） sudo mkdir -p /var/lib/buildkit-container sudo chmod 700 /var/lib/buildkit-container Removed /etc/systemd/system/multi-user.target.wants/buildkitd.service. # 启动buildkit镜像构建服务 # 挂载/run/containerd/containerd.sock方便container调用buildkitd # 挂载/var/lib/buildkit，以便于将构建过程中下载的镜像持久化存储，方便下次构建时使用缓存 # 挂载/run/buildkit/目录方便nerctl调用buildkitd root@k8s-03:/var/lib/buildkit# nerdctl run -d --name buildkit \ --privileged \ -v /run/buildkit/:/run/buildkit/ \ -v /var/lib/buildkit-container:/var/lib/buildkit \ -v /run/containerd/containerd.sock:/run/containerd/containerd.sock \ registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2 a9d371968d62cb654e50677bb7433b6c58ac25cf177c76dcc81dfb3a311b8289 root@k8s-03:/var/lib/buildkit# nerdctl ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a9d371968d62 registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2 "buildkitd" 7 seconds ago Up buildkit root@k8s-03:/var/lib/buildkit# root@k8s-03:/var/lib/buildkit# root@k8s-03:/var/lib/buildkit# buildctl --addr unix:///run/buildkit/buildkitd.sock debug workers ID PLATFORMS 37647lrzalh7j20qy1hskzedp linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/386 qi74wmika7i8tvdej2ob0ooj5 linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/386 root@k8s-03:/var/lib/buildkit# root@k8s-03:/var/lib/buildkit# # 启动jenkins-slave容器 # 挂载/run/containerd/containerd.sock方便netdctl操作container # 挂载/run/buildkit/目录方便nerctl调用buildkitd构建镜像 # 挂载/root/.kube/目录方便kubectl工具操作k8s root@k8s-03:/var/lib/buildkit# nerdctl run --name jenkins-slave -it --privileged=true \ -v /run/buildkit/:/run/buildkit/ \ -v /root/.kube/:/root/.kube/ \ -v /run/containerd/containerd.sock:/run/containerd/containerd.sock \ registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 bash # 测试镜像构建 root@d2f81acc6e0b:/home/jenkins# echo 'FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest' >> Dockerfile echo 'CMD ["echo","hello","container"]' >> Dockerfile cat Dockerfile FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest CMD ["echo","hello","container"] root@d2f81acc6e0b:/home/jenkins# root@d2f81acc6e0b:/home/jenkins# root@d2f81acc6e0b:/home/jenkins# nerdctl build -t test-test:v1 . [+] Building 12.4s (3/5) [+] Building 12.5s (5/5) FINISHED => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 143B 0.0s => [internal] load metadata for registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest 11.6s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [1/1] FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest@sha256:b41a05bd7a4a32e4c48c284cc2178abe8c11 0.8s => => resolve registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest@sha256:b41a05bd7a4a32e4c48c284cc2178abe8c11 0.0s => => sha256:90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c 2.15MB / 2.15MB 0.8s => exporting to docker image format 0.8s => => exporting layers 0.0s => => exporting manifest sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348 0.0s => => exporting config sha256:0b44030dca1d1504de8aa100696d5c86f19b06cec660cf55c2ba6c5c36d1fb89 0.0s => => sending tarball 0.0s Loaded image: docker.io/library/test-test:v1 root@d2f81acc6e0b:/home/jenkins# 4.3job任务创建与配置配置webhook构建触发器，当分支代码提交时触发构建，具体配置如下：流水线选择SCM从代码仓库中获取jenkinsfile，脚本路径填写Jenkinsfile-k8s.groov手动触发完会生成一个pod4.4部署总结1. jenkinsfile中如果涉及yaml的代码需要注意权限。 2. 还需要查看ServiceAccount是否跟之前设置的正确 3. 镜像尽量自己做成国内的地址 4. 如果是自定义的Harbor仓库需要提前创建Docker Registry Secret就是Harbor仓库的账号密码，然后在yaml中添加上配置imagePullSecrets。 5. 提前把名称空间创建出来五、效果演示 5.1开发测试阶段模拟开发人员完成功能开发后提交代码至test分支，推送以后会自动触发gitlab的webhook然后自动调用拉取jenkins你设置好的job流水线。root@k8s-03:~/work/sprint-boot-demo# git branch -a main * test remotes/origin/HEAD -> origin/main remotes/origin/main root@k8s-03:~/work/sprint-boot-demo# cat src/main/java/com/example/springbootdemo/HelloWorldController.java package com.example.springbootdemo; import org.springframework.stereotype.Controller; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.ResponseBody; @Controller public class HelloWorldController { @RequestMapping("/") @ResponseBody public String hello() { // 获取环境变量 ENV_NAME，如果不存在则使用默认值 "default" String envName = System.getenv().getOrDefault("ENV_NAME", "default"); return String.format(" Hello SpringBoot Version:v1 Env:%s ", envName); } @RequestMapping("/health") @ResponseBody public String healthy() { return "ok"; } } root@k8s-03:~/work/sprint-boot-demo# vi src/main/java/com/example/springbootdemo/HelloWorldController.java root@k8s-03:~/work/sprint-boot-demo# git add . root@k8s-03:~/work/sprint-boot-demo# git commit -m "test环境更新版本至v2" [test 368ff3d] test环境更新版本至v2 1 file changed, 1 insertion(+), 1 deletion(-) root@k8s-03:~/work/sprint-boot-demo# git push fatal: The current branch test has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream origin test root@k8s-03:~/work/sprint-boot-demo# root@k8s-03:~/work/sprint-boot-demo# git push --set-upstream origin test Username for 'http://192.168.30.181': root Password for 'http://root@192.168.30.181': Enumerating objects: 17, done. Counting objects: 100% (17/17), done. Delta compression using up to 8 threads Compressing objects: 100% (6/6), done. Writing objects: 100% (9/9), 697 bytes | 697.00 KiB/s, done. Total 9 (delta 2), reused 0 (delta 0), pack-reused 0 remote: remote: To create a merge request for test, visit: remote: http://192.168.30.181/develop/sprint-boot-demo/-/merge_requests/new?merge_request%5Bsource_branch%5D=test remote: To http://192.168.30.181/develop/sprint-boot-demo.git * [new branch] test -> test Branch 'test' set up to track remote branch 'test' from 'origin'.此时查看cicd名称空间下的pod信息，发现已经创建一个名为springbootdemo-275-rf832-h6jkq-630x8的pod，包含3个container，分别是jnlp、maven、buildkitd。root@k8s-02:~# kubectl get pods -n cicd NAME READY STATUS RESTARTS AGE jenkins-7d65887794-tsqs5 1/1 Running 5 (104m ago) 2d8h springbootdemo-40-p9smx-4mw1j-z0v30 3/3 Running 0 7m30s 查看jenkins任务信息，已顺利完成了集成部署工作。并且收到了jenkins自动发出的邮件，内容如下：查看SonarQube代码扫描信息，未发现异常代码。查看k8s，已成功创建相关资源root@k8s-01:~/gitlab# kubectl get all -n test NAME READY STATUS RESTARTS AGE pod/demo-7d4975d656-cdpk7 1/1 Running 0 4m28s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/demo ClusterIP 10.108.234.81 <none> 8888/TCP 14m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/demo 1/1 1 1 14m NAME DESIRED CURRENT READY AGE replicaset.apps/demo-7d4975d656 1 1 1 14m root@k8s-01:~/gitlab# 此时模拟测试人员，访问测试环境域名至此，开发测试阶段演示完成。5.2生产发布阶段接下来演示master分支代码提交后，触发生产环境版本发布流程。#Harbor仓库的权限要提前创建 kubectl create secret docker-registry harbor-pull-secret \ --namespace=test \ --docker-server=192.168.30.180:30003 \ --docker-username=admin \ --docker-password=Harbor12345root@k8s-03:~/work/sprint-boot-demo# git branch -a main * test remotes/origin/HEAD -> origin/main remotes/origin/main remotes/origin/test root@k8s-03:~/work/sprint-boot-demo# git checkout main Switched to branch 'main' Your branch is up to date with 'origin/main'. root@k8s-03:~/work/sprint-boot-demo# vi src/main/java/com/example/springbootdemo/HelloWorldController.java root@k8s-03:~/work/sprint-boot-demo# cat src/main/java/com/example/springbootdemo/HelloWorldController.java package com.example.springbootdemo; import org.springframework.stereotype.Controller; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.ResponseBody; @Controller public class HelloWorldController { @RequestMapping("/") @ResponseBody public String hello() { // 获取环境变量 ENV_NAME，如果不存在则使用默认值 "default" String envName = System.getenv().getOrDefault("ENV_NAME", "default"); return String.format(" Hello SpringBoot Version:v5 Env:%s ", envName); } @RequestMapping("/health") @ResponseBody public String healthy() { return "ok"; } } root@k8s-03:~/work/sprint-boot-demo# git add . root@k8s-03:~/work/sprint-boot-demo# git commit -m "生产环境更新版本v5" [main 617325e] 生产环境更新版本v5 1 file changed, 1 insertion(+), 1 deletion(-) root@k8s-03:~/work/sprint-boot-demo# git push Username for 'http://192.168.30.181': root Password for 'http://root@192.168.30.181': Enumerating objects: 17, done. Counting objects: 100% (17/17), done. Delta compression using up to 8 threads Compressing objects: 100% (6/6), done. Writing objects: 100% (9/9), 704 bytes | 704.00 KiB/s, done. Total 9 (delta 2), reused 0 (delta 0), pack-reused 0 To http://192.168.30.181/develop/sprint-boot-demo.git 912f7b3..617325e main -> main 待收到邮件通知后，查看k8s资源，已经在prod名称空间下创建相关资源此时访问生产环境域名，服务可以正常访问。此时查看Harbor仓库镜像信息，其中p开头的为生产环境镜像，t开头的为测试环境镜像。jenkinsfile pipeline { agent { kubernetes { // 定义要在 Kubernetes 中运行的 Pod 模板 yaml ''' apiVersion: v1 kind: Pod metadata: labels: app: jenkins-slave spec: serviceAccountName: jenkins-admin containers: - name: jnlp image: registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 resources: limits: memory: "512Mi" cpu: "500m" securityContext: privileged: true volumeMounts: - name: buildkit mountPath: "/run/buildkit/" - name: containerd mountPath: "/run/containerd/containerd.sock" - name: kube-config mountPath: "/root/.kube/" readOnly: true - name: maven image: registry.cn-guangzhou.aliyuncs.com/xingcangku/maven3.9.3-openjdk-17:v2.0 resources: limits: memory: "512Mi" cpu: "500m" command: - 'sleep' args: - '9999' volumeMounts: - name: maven-data mountPath: "/root/.m2" - name: buildkitd image: registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2 resources: limits: memory: "256Mi" cpu: "500m" securityContext: privileged: true volumeMounts: - name: buildkit mountPath: "/run/buildkit/" - name: buildkit-data mountPath: "/var/lib/buildkit/" - name: containerd mountPath: "/run/containerd/containerd.sock" volumes: - name: maven-data persistentVolumeClaim: claimName: jenkins-maven - name: buildkit hostPath: path: /run/buildkit/ - name: buildkit-data hostPath: path: /var/lib/buildkit/ - name: containerd hostPath: path: /run/containerd/containerd.sock - name: kube-config secret: secretName: kube-config ''' retries 2 } } environment { // 全局变量 HARBOR_CRED = "harbor-admin" IMAGE_NAME = "" IMAGE_APP = "demo" branchName = "" } stages { stage('拉取代码') { environment { // gitlab仓库信息 GITLAB_CRED = "gitlab-xing-password" GITLAB_URL = "http://192.168.30.181/develop/sprint-boot-demo.git" } steps { echo '开始拉取代码' checkout scmGit(branches: name: '*/*', extensions: [], userRemoteConfigs: [[credentialsId: "${GITLAB_CRED}", url: "${GITLAB_URL}"]]) // 获取当前拉取的分支名称 script { def branch = env.GIT_BRANCH ?: 'main' branchName = branch.split('/')[-1] } echo '拉取代码完成' } } stage('编译打包') { steps { container('maven') { // 指定使用maven container进行打包 echo '开始编译打包' sh 'mvn clean package' echo '编译打包完成' } } } stage('代码审查') { environment { // SonarQube信息 SONARQUBE_SCANNER = "SonarQube" SONARQUBE_SERVER = "SonarQube" } steps { echo '开始代码审查' script { def scannerHome = tool "${SONARQUBE_SCANNER}" withSonarQubeEnv("${SONARQUBE_SERVER}") { // 添加扫描参数 sh """ ${scannerHome}/bin/sonar-scanner \ -Dsonar.projectKey=springbootdemo \ -Dsonar.projectName=SpringBootDemo \ -Dsonar.java.binaries=target/classes \ -Dsonar.sources=src/main/java \ -Dsonar.sourceEncoding=UTF-8 """ } } echo '代码审查完成' } } stage('构建镜像') { environment { // harbor仓库信息 HARBOR_URL = "192.168.30.180:30003" HARBOR_PROJECT = "spring_boot_demo" // 镜像标签 IMAGE_TAG = '' // 镜像名称 IMAGE_NAME = '' } steps { echo '开始构建镜像' script { if (branchName == 'main') { IMAGE_TAG = VersionNumber versionPrefix: 'p', versionNumberString: '${BUILD_DATE_FORMATTED, "yyMMdd"}.${BUILDS_TODAY}' } else if (branchName == 'test') { IMAGE_TAG = VersionNumber versionPrefix: 't', versionNumberString: '${BUILD_DATE_FORMATTED, "yyMMdd"}.${BUILDS_TODAY}' } else { error("Unsupported branch: ${params.BRANCH}") } IMAGE_NAME = "${HARBOR_URL}/${HARBOR_PROJECT}/${IMAGE_APP}:${IMAGE_TAG}" sh "nerdctl build --insecure-registry -t ${IMAGE_NAME} . " } echo '构建镜像完成' echo '开始推送镜像' // 获取harbor账号密码 withCredentials([usernamePassword(credentialsId: "${HARBOR_CRED}", passwordVariable: 'HARBOR_PASSWORD', usernameVariable: 'HARBOR_USERNAME')]) { // 登录Harbor仓库 sh """nerdctl login --insecure-registry ${HARBOR_URL} -u ${HARBOR_USERNAME} -p ${HARBOR_PASSWORD} nerdctl push --insecure-registry ${IMAGE_NAME}""" } echo '推送镜像完成' echo '开始删除镜像' script { sh "nerdctl rmi -f ${IMAGE_NAME}" } echo '删除镜像完成' } } stage('项目部署') { environment { // 资源清单名称 YAML_NAME = "k8s.yaml" } steps { echo '开始修改资源清单' script { if (branchName == 'main' ) { NAME_SPACE = 'prod' DOMAIN_NAME = 'demo.local.com' } else if (branchName == 'test') { NAME_SPACE = 'test' DOMAIN_NAME = 'demo.test.com' } else { error("Unsupported branch: ${params.BRANCH}") } } // 使用Content Replace插件进行k8s资源清单内容替换 contentReplace(configs: [fileContentReplaceConfig(configs: [fileContentReplaceItemConfig(replace: "${IMAGE_NAME}", search: 'IMAGE_NAME'), fileContentReplaceItemConfig(replace: "${NAME_SPACE}", search: 'NAME_SPACE'), fileContentReplaceItemConfig(replace: "${DOMAIN_NAME}", search: 'DOMAIN_NAME')], fileEncoding: 'UTF-8', filePath: "${YAML_NAME}", lineSeparator: 'Unix')]) echo '修改资源清单完成' sh "cat ${YAML_NAME}" echo '开始部署资源清单' sh "kubectl apply -f ${YAML_NAME}" echo '部署资源清单完成' } } } post { always { echo '开始发送邮件通知' emailext(subject: '构建通知：${PROJECT_NAME} - Build # ${BUILD_NUMBER} - ${BUILD_STATUS}!', body: '${FILE,path="email.html"}', to: '7902731@qq.com') echo '邮件通知发送完成' } } }
一、先部署Jenkins和k8s连接和动态slavehttps://axzys.cn/index.php/archives/521/二、流水线的配置cat > Dockerfile << EOF FROM busybox CMD ["echo","hello","container"] EOF nerdctl build -t busybox:v1 . root@k8s-03:~# nerdctl images REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE busybox v1 05610df32232 19 seconds ago linux/amd64 4.338MB 2.146MB buildkitd-test v1 05610df32232 41 hours ago linux/amd64 4.338MB 2.146MB #打标签 root@k8s-03:~# nerdctl tag busybox:v1 192.168.30.180:30003/test/busybox:v1 #推送报错，https连接 要证书 root@k8s-03:~# nerdctl push 192.168.30.180:30003/test/busybox:v1 INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348) manifest-sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348: waiting |--------------------------------------| layer-sha256:90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c: waiting |--------------------------------------| config-sha256:0b44030dca1d1504de8aa100696d5c86f19b06cec660cf55c2ba6c5c36d1fb89: waiting |--------------------------------------| elapsed: 0.1 s total: 0.0 B (0.0 B/s) FATA[0000] failed to do request: Head "https://192.168.30.180:30003/v2/test/busybox/blobs/sha256:90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c": tls: failed to verify certificate: x509: certificate signed by unknown authority #从Harbor中下载ca证书 root@k8s-03:~# sudo cp ca.crt /usr/local/share/ca-certificates/192.168.30.180-registry.crt #更新证书 root@k8s-03:~# sudo update-ca-certificates Updating certificates in /etc/ssl/certs... rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL 1 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d... done. #重启containerd root@k8s-03:~# sudo systemctl restart containerd #推送镜像到Harbor，下面报错了原因是没有登录Harbor root@k8s-03:~# nerdctl push 192.168.30.180:30003/test/busybox:v1 INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348) manifest-sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348: waiting |--------------------------------------| layer-sha256:90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c: waiting |--------------------------------------| config-sha256:0b44030dca1d1504de8aa100696d5c86f19b06cec660cf55c2ba6c5c36d1fb89: waiting |--------------------------------------| elapsed: 0.1 s total: 0.0 B (0.0 B/s) FATA[0000] push access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials 、 #登录Harbor root@k8s-03:~# nerdctl login 192.168.30.180:30003 Enter Username: admin Enter Password: WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded #再次推送成功 root@k8s-03:~# nerdctl push 192.168.30.180:30003/test/busybox:v1 INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.v2+json, sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348) manifest-sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c: done |++++++++++++++++++++++++++++++++++++++| config-sha256:0b44030dca1d1504de8aa100696d5c86f19b06cec660cf55c2ba6c5c36d1fb89: done |++++++++++++++++++++++++++++++++++++++| elapsed: 0.7 s total: 2.0 Mi (2.9 MiB/s) root@k8s-03:~#
一、jenkins免密登录配置 1.1安装插件Publish Over SSH1.2配置SSH免密登录在jenkins主机执行操作。root@k8s-03:~# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa Your public key has been saved in /root/.ssh/id_rsa.pub The key fingerprint is: SHA256:bIagD4/Gebf8fslUICpXfV6Qf6+7nw+Z8vX/5ETwWcc root@k8s-03 The key's randomart image is: +---[RSA 3072]----+ | . .o | | o o o .. | | . o . + o. E| | ...oo o .o=| | o o. S . o+| | . * o . +.| | = + . o .. +.+| | . . o . + o.*o| | ooo. +=@| +----[SHA256]-----+ root@k8s-03:~# ssh-copy-id 192.168.30.181 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@192.168.30.181's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '192.168.30.181'" and check to make sure that only the key(s) you wanted were added. root@k8s-03:~# ssh-copy-id 192.168.30.180 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub" /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys root@192.168.30.180's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh '192.168.30.180'" and check to make sure that only the key(s) you wanted were added. root@k8s-03:~# ssh 192.168.30.180 Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-151-generic x86_64) * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/pro System information as of Sun Aug 3 05:39:01 AM UTC 2025 System load: 0.3642578125 Processes: 314 Usage of /: 28.4% of 49.48GB Users logged in: 1 Memory usage: 13% IPv4 address for docker0: 172.17.0.1 Swap usage: 0% IPv4 address for ens33: 192.168.30.180 * Strictly confined Kubernetes makes edge and IoT secure. Learn how MicroK8s just raised the bar for easy, resilient and secure K8s cluster deployment. https://ubuntu.com/engage/secure-kubernetes-at-the-edge Expanded Security Maintenance for Applications is not enabled. 66 updates can be applied immediately. To see these additional updates run: apt list --upgradable Enable ESM Apps to receive additional future security updates. See https://ubuntu.com/esm or run: sudo pro status New release '24.04.2 LTS' available. Run 'do-release-upgrade' to upgrade to it. Last login: Sun Aug 3 04:48:27 2025 from 192.168.30.1 root@k8s-01:~# exit logout Connection to 192.168.30.180 closed. 1.3插件配置在Jenkins中【系统管理】—【系统配置】，找到“Publish over SSH”来配置该插件信息。#key通过查看jenkins服务器cat .ssh/id_rsa获取。 #或者填写path to key路径/root/.ssh/id_ras。 root@k8s-03:~# cat .ssh/id_rsa id_rsa id_rsa.pub root@k8s-03:~# cat .ssh/id_rsa -----BEGIN OPENSSH PRIVATE KEY----- [binary] -----END OPENSSH PRIVATE KEY-----二、验证测试 2.1创建自由风格项目2.2创建测试脚本在jenkins服务器ssh-test目录下root@k8s-03:~# mkdir -p /var/lib/jenkins/workspace/ssh-test root@k8s-03:~# cd /var/lib/jenkins/workspace/ssh-test/ root@k8s-03:/var/lib/jenkins/workspace/ssh-test# ls root@k8s-03:/var/lib/jenkins/workspace/ssh-test# pwd /var/lib/jenkins/workspace/ssh-test root@k8s-03:/var/lib/jenkins/workspace/ssh-test# vi test.sh root@k8s-03:/var/lib/jenkins/workspace/ssh-test# cat test.sh #!/bin/bash date >> /tmp/date.txt2.3添加构建步骤Name：“系统管理>系统配置”设置的SSH Sverver的名字Name。 Source files：允许为空，复制到远程主机上的文件，**/*意思是当前工作目录下所有问题 Remove prefix：允许为空，文件复制时要过滤的目录。 Remote directory：允许为空，文件得到到远程机上的目录，如果填写目录名则是相对于“SSH Server”中的“Remote directory”的，如果不存在将会自动创建。 Exec command：在这里填写在远程主机上执行的命令。2.4工作空间要理解的细节 1、Jenkins是k8s部署起来的时候，如果没有固定pod的启动的节点则是会变化的。那么pod挂载卷对应的目录也会变化。 2、如果要指定节点则需要配置（General）里面加上对应的节点，节点必须提前配置好https://axzys.cn/index.php/archives/572/2.5构建查看结果由控制台打印内容可知，已经成功传输一个文件。 登录服务器查看执行结果。root@k8s-02:~# cd /opt/jenkins/ root@k8s-02:/opt/jenkins# ls remoting test.sh root@k8s-02:/opt/jenkins# cat test.sh #!/bin/bash date >> /tmp/date.txt root@k8s-02:/opt/jenkins# cat /tmp/date.txt Sun Aug 3 09:35:34 AM UTC 2025 root@k8s-02:/opt/jenkins#
一、Jenkins的Master/Slave机制Jenkins采用Master/Slave架构。Master/Slave相当于Server和agent的概念，Master提供web接口让用户来管理Job和Slave，Job可以运行在Master本机或者被分配到Slave上运行。一个Master可以关联多个Slave用来为不同的Job或相同的Job的不同配置来服务。 Jenkins的Master/Slave机制除了可以并发的执行构建任务，加速构建以外。还可以用于分布式自动化测试，当自动化测试代码非常多或者是需要在多个浏览器上并行的时候，可以把测试代码划分到不同节点上运行，从而加速自动化测试的执行。二、集群角色功能**Master：**Jenkins服务器。主要是处理调度构建作业，把构建分发到Slave节点实际执行，监视Slave节点的状态。当然，也并不是说Master节点不能跑任务。构建结果和构建产物最后还是传回到Master节点，比如说在jenkins工作目录下面的workspace内的内容，在Master节点照样是有一份的。 **Slave：**执行机(奴隶机)。执行Master分配的任务，并返回任务的进度和结果。Jenkins Master/Slave的搭建需要至少两台机器，一台Master节点，一台Slave节点（实际生产中会有多个Slave节点）。三、搭建步骤Master不需要主动去建立，安装Jenkins，在登录到主界面时，这台电脑就已经默认为master。 选择“Manage Jenkins”->“Manage Nodes and Clouds”，可以看到Master节点相关信息：四、为Jenkins添加Slave Node 4.1开启tcp代理端口jenkins web代理是指slave通过jenkins服务端提供的一个tcp端口，与jenkins服务端建立连接，docker版的jenkins默认开启web tcp代理，端口为50000，而自己手动制作的jenkins容器或者在物理机环境部署的jenkins，都需要手动开启web代理端口，如果不开启，slave无法通过web代理的方式与jenkins建立连接。 jenkins web代理的tcp端口不是通过命令启动的而是通过在全局安全设置中配置的，配置成功后会在系统上运行一个指定的端口4.2添加节点信息在Jenkins界面选择“Manage Jenkins”->“Manage Nodes and Clouds”->“New Node配置Agent信息Name：Slave机器的名字 Description：描述 ，不重要 随意填 Number of excutors：允许在这个节点上并发执行任务的数量，即同时可以下发多少个Job到Slave上执行，一般设置为 cpu 支持的线程数。[注：Master Node也可以通过此参数配置Master是否也执行构建任务、还是仅作为Jenkins调度节点] Remote root directory：用来放工程的文件夹，jenkins master上设置的下载的代码会放到这个工作目录下。 Lables：标签，用于实现后续Job调度策略，根据Jobs配置的Label选择Salve Node Usage：支持两种模式“Use this Node as much as possible”、“Only build Jobs with Label expressiong matching this Node”。选择“Only build Jobs with Label expressiong matching this Node”， 添加完毕后，在Jenkins主界面，可以看到新添加的Slave Node，但是红叉表示此时的Slave并未与Master建立起联系。4.3slave节点配置安装jdk#dnf -y install java-17-openjdk root@k8s-02:~# java -version openjdk version "17.0.16" 2025-07-15 LTS OpenJDK Runtime Environment Corretto-17.0.16.8.1 (build 17.0.16+8-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.16.8.1 (build 17.0.16+8-LTS, mixed mode, sharing) 安装agent 点击节点信息，根据控制台提示执行安装agent命令root@k8s-02:~# curl -sO http://192.168.30.180:31530/jnlpJars/agent.jar root@k8s-02:~# java -jar agent.jar -url http://192.168.30.180:31530/ -secret 6e87c37900dfbdcff98099f6681f7b195a141de1cacb157679efc98f8fec2644 -name "k8s-02" -webSocket -workDir "/opt/jenkins" Aug 03, 2025 12:52:47 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /opt/jenkins/remoting as a remoting work directory Aug 03, 2025 12:52:47 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /opt/jenkins/remoting Aug 03, 2025 12:52:47 PM hudson.remoting.Launcher createEngine INFO: Setting up agent: k8s-02 Aug 03, 2025 12:52:47 PM hudson.remoting.Engine startEngine INFO: Using Remoting version: 3309.v27b_9314fd1a_4 Aug 03, 2025 12:52:47 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /opt/jenkins/remoting as a remoting work directory Aug 03, 2025 12:52:47 PM hudson.remoting.Launcher$CuiListener status INFO: WebSocket connection open Aug 03, 2025 12:52:47 PM hudson.remoting.Launcher$CuiListener status INFO: Connected4.4查看agent状态指定Node调度策略 创建Job的页面，“General”下勾选“Restric where this project can be run”，填写Label Expression。
一、jenkins安装插件 1.1下载SonarQube插件进入Jenkins的系统管理->插件管理->可选插件，搜索框输入sonarqube，安装重启。1.2启用SonarQubeJenkins的系统管理->系统配置，添加SonarQube服务。二、SonarQube配置 2.1禁用审查结果上传到SCM功能2.2生成token添加jenkin用户 token：squ_4bc173eb520dd35c176104baa1b899a992e88c88三、jenkins配置 3.1添加令牌Jenkins的系统管理->系统配置->添加token类型切换成Secret text，粘贴token，点击添加。选上刚刚添加的令牌凭证，点击应用保存。3.2SonarQube Scanner 安装进入Jenkins的系统管理->全局工具配置，下滑找到图片里的地方，点击新增SonarQube Scanner，我们选择自动安装并选择最新的版本。四、非流水线项目添加代码审查 4.1添加构建步骤编辑之前的自由风格构建的demo项目，在构建阶段新增步骤。analysis properties参数如下# 项目名称id，全局唯一 sonar.projectKey=sprint_boot_demo # 项目名称 sonar.projectName=sprint_boot_demo sonar.projectVersion=1.0 # 扫描路径，当前项目根目录 sonar.sources=./src # 排除目录 sonar.exclusions=**/test/**,**/target/** # jdk版本 sonar.java.source=1.17 sonar.java.target=1.17 # 字符编码 sonar.sourceEncoding=UTF-8 # binaries路径 sonar.java.binaries=target/classes4.2构建并查看结果jenkins点击立即构建，查看构建结果查看SonarQube扫描结果五、流水线项目添加代码审查 5.1创建sonar-project.properties文件项目根目录下，创建sonar-project.properties文件，内容如下# 项目名称id，全局唯一 sonar.projectKey=sprint_boot_demo # 项目名称 sonar.projectName=sprint_boot_demo sonar.projectVersion=1.0 # 扫描路径，当前项目根目录 sonar.sources=./src # 排除目录 sonar.exclusions=**/test/**,**/target/** # jdk版本 sonar.java.source=1.17 sonar.java.target=1.17 # 字符编码 sonar.sourceEncoding=UTF-8 # binaries路径 sonar.java.binaries=target/classes5.2修改Jenkinsfile加入SonarQube代码审查阶段pipeline { agent any stages { stage('拉取代码') { steps { echo '开始拉取代码' checkout([$class: 'GitSCM', branches: name: '*/master', userRemoteConfigs: url: 'https://gitee.com/axzys/sprint_boot_demo.git']) echo '拉取代码完成' } } stage('打包编译') { steps { echo '开始打包编译' sh 'mvn clean package' echo '打包编译完成' } } stage('代码审查') { steps { echo '开始代码审查' script { // 引入SonarQube scanner，名称与jenkins 全局工具SonarQube Scanner的name保持一致 def scannerHome = tool 'SonarQube' // 引入SonarQube Server，名称与jenkins 系统配置SonarQube servers的name保持一致 withSonarQubeEnv('SonarQube') { sh "${scannerHome}/bin/sonar-scanner" } } echo '代码审查完成' } } stage('部署项目') { steps { echo '开始部署项目' echo '部署项目完成' } } } } 5.3构建测试
一、安装kubernetes插件在Jenkins的插件管理中安装Kubernetes插件 jenkins——>系统管理——>插件管理——>avaliable plugins二、本集群连接 2.1创建sa账号如果jenkins在k8s集群中部署，直接创建sa账号，并进行rbac授权即可,yaml文件参考前面文章。2.2创建cloud资源然后在jenkins——>系统管理——>Clouds——>New cloud——>输入cloud name并勾选类型为kubernetesroot@k8s-01:~/jenkins# cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: cicd spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins spec: securityContext: fsGroup: 1000 # 确保 Jenkins 用户有存储写入权限 serviceAccountName: jenkins-admin automountServiceAccountToken: true # 新增节点选择器，将 Pod 固定在 k8s-03 节点 nodeSelector: kubernetes.io/hostname: k8s-03 containers: - name: jenkins image: registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17:lts-jdk17 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 - containerPort: 50000 # 新增环境变量（可选） env: - name: JAVA_HOME value: "/usr/lib/jvm/amazon-corretto-17.0.16.8.1-linux-x64" volumeMounts: - name: jenkins-data mountPath: /var/jenkins_home # ============ 新增挂载 ============ - name: host-jvm mountPath: /usr/lib/jvm - name: maven-data mountPath: /usr/local/maven resources: limits: cpu: "1" memory: "3Gi" requests: cpu: "0.5" memory: "1Gi" livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 90 periodSeconds: 10 volumes: - name: jenkins-data persistentVolumeClaim: claimName: jenkins-pvc # ============ 新增的卷配置 ============ - name: host-jvm hostPath: path: /usr/lib/jvm type: Directory - name: maven-data hostPath: path: /usr/local/maven type: Directoryroot@k8s-01:~/jenkins# cat new-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-admin namespace: cicd --- # 创建自定义 ClusterRole apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: jenkins-clusterrole rules: - apiGroups: [""] resources: ["pods", "pods/log", "services"] verbs: ["get", "list", "watch", "create", "update", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["*"] --- # 绑定 ServiceAccount 到 ClusterRole apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: jenkins-crb roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins-clusterrole subjects: - kind: ServiceAccount name: jenkins-admin namespace: cicdk8s1.24版本以后​​，行为变更​​：从 Kubernetes v1.24 开始，不再自动为 ServiceAccount 创建 Secret（令牌不再存储在 Secret 中） ​​TokenRequest API​​：现在需要使用 TokenRequest API 获取令牌 root@k8s-01:~/jenkins# cat jenkins-secret.yaml apiVersion: v1 kind: Secret metadata: name: jenkins-admin-token namespace: cicd annotations: kubernetes.io/service-account.name: jenkins-admin type: kubernetes.io/service-account-token root@k8s-01:~/jenkins# kubectl apply -n cicd -f jenkins-secret.yaml secret/jenkins-admin-token created root@k8s-01:~/jenkins# kubectl describe sa jenkins-admin -n cicd Name: jenkins-admin Namespace: cicd Labels: <none> Annotations: <none> Image pull secrets: <none> Mountable secrets: <none> Tokens: jenkins-admin-token Events: <none> root@k8s-01:~/jenkins# kubectl get secret jenkins-admin-token -n cicd -o jsonpath='{.data.token}' | base64 -d eyJhbGciOiJSUzI1NiIsImtpZCI6IjJ0MTFFdDhfdnFBYkNuTnBSSXlyOFIzN1B0MW13cVVJNlFwZDV1VzR1WXcifQ.[binary].qvBiSwdCVbfMDnP0ElhnMcn__q65HDqidWw2JDQbrn7zrgNX2jjXlDxhA8RQFQaHhrTPGrOuP60vzfz4WgvJxwJvwIHaqqAbK8r3t-eTBpXNKltY3GEFEqxyjVlTd8q0DLW0OWZHUVZJrWhYT00Xa1ZViJgwQ2X0ogpAphSlvR351ZEDmDDxwxk4WwioZpmU22_weFamlU0g9SQVW5kBYGw06Tq_dPNL7cB0CDdPy0mSYckquEtG4xh-EIddPs9cGd1_OGurjwEkwX2-HvlfCXfoFkgo42lrOCaouWE4I4e21OXUCg5erFONSWSPKhhmhiLsjikByweCE0qaA2ZIUw#还要另外创建serviceAccount root@k8s-01:~/jenkins# cat jenkins-secret.yaml rbac.yaml apiVersion: v1 kind: Secret metadata: name: jenkins-admin-token namespace: cicd annotations: kubernetes.io/service-account.name: jenkins-admin type: kubernetes.io/service-account-token # rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-admin namespace: cicd --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins-admin-crb roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: edit # 授予命名空间内管理权限 subjects: - kind: ServiceAccount name: jenkins-admin namespace: cicd#Kubernetes 服务证书 key root@k8s-01:~/jenkins# kubectl get secret jenkins-admin-token -n cicd -o jsonpath='{.data.ca\.crt}' | base64 -d -----BEGIN CERTIFICATE----- [binary] -----END CERTIFICATE-----点击kubernetes cloud details填写cloud详细信息 - Kubernetes地址：在集群内部暴露的k8s service名称https://kubernetes.default.svc - Kubernetes命名空间：jenkins sa所属的名称空间cicd - Jenkins地址：jenkins svc的名称:8080端口http://jenkins.cicd.svc:8080 使用刚刚获取的令牌在 Jenkins 中创建凭据： 凭据类型：Secret text Secret：粘贴上面的令牌值 ID：k8s-service-account-token 配置完成后点击连接测试，显示k8s集群版本，证明配置无误。三、跨集群连接在某些情况下，jenkins部署在k8s集群外，通过二进制或者docker方式部署，如果想要连接k8s集群实现资源自动创建。或者当前jenkins部署在k8s集群A中，需要通过jenkins实现集群B资源的自动创建发布，使用此方式连接。3.1配置思路jenkins要想连接并操作k8s集群，需要配置授权，请求k8s集群的kube apiserver的请求，可以和kubectl一样利用config文件用作请求的鉴权，默认在~/.kube/config下，也可以单独严格指定权限细节，生成一个jenkins专用的config文件。 在jenkins中能够识别的证书文件为PKCS#12 certificate，因此需要先将kubeconfig文件中的证书转换生成PKCS#12格式的pfx证书文件3.2生成证书我们可以使用yq命令行工具解析yaml，并提取相关的内容，然后通过base 64解码，最后生成文件 安装yq工具，仓库地址：https://github.com/mikefarah/yq [root@k8s-master ~]# wget https://github.com/mikefarah/yq/releases/download/v4.34.1/yq_linux_amd64.tar.gz [root@k8s-master ~]# tar -zxvf yq_linux_amd64.tar.gz [root@k8s-master ~]# mv yq_linux_amd64 /usr/bin/yq [root@k8s-master ~]# yq --version yq (https://github.com/mikefarah/yq/) version v4.34.1 [root@k8s-master ~]# mkdir -p /opt/jenkins-crt/certificate-authority-data——>base 64解码——>ca.crt client-certificate-data——>base 64解码——>client.crt client-key-data——>base 64解码——>client.key[root@k8s-master ~]# yq e '.clusters[0].cluster.certificate-authority-data' /root/.kube/config | base64 -d > /opt/jenkins-crt/ca.crt [root@k8s-master ~]# yq e '.users[0].user.client-certificate-data' /root/.kube/config | base64 -d > /opt/jenkins-crt/client.crt [root@k8s-master ~]# yq e '.users[0].user.client-key-data' /root/.kube/config | base64 -d > /opt/jenkins-crt/client.key [root@k8s-master ~]# cd /opt/jenkins-crt/ [root@k8s-master jenkins-crt]# ls -la 总用量 12 drwxr-xr-x 2 root root 56 6月 10 20:54 . drwxr-xr-x. 6 root root 65 6月 10 20:37 .. -rw-r--r-- 1 root root 1099 6月 10 20:53 ca.crt -rw-r--r-- 1 root root 1147 6月 10 20:53 client.crt -rw-r--r-- 1 root root 1675 6月 10 20:54 client.key3.3转换证书通过openssl进行证书格式的转换，生成Client P12认证文件cert.pfx，输入两次密码并牢记密码。[root@k8s-master jenkins-crt]# openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt Enter Export Password: Verifying - Enter Export Password: [root@k8s-master jenkins-crt]# ls -la 总用量 16 drwxr-xr-x 2 root root 72 6月 10 20:55 . drwxr-xr-x. 6 root root 65 6月 10 20:37 .. -rw-r--r-- 1 root root 1099 6月 10 20:53 ca.crt -rw------- 1 root root 3221 6月 10 20:55 cert.pfx -rw-r--r-- 1 root root 1147 6月 10 20:53 client.crt -rw-r--r-- 1 root root 1675 6月 10 20:54 client.key3.4导入证书打开jenkins的web界面，系统管理——>Credentials——>添加全局凭据 凭据的类型选择Certificate，证书上传刚才生成的cert.pfx证书文件，输入通过openssl生成证书文件时输入的密码3.5配置远程k8s集群地址jenkins——>系统管理——>Clouds——>New cloud——>输入cloud name并勾选类型为kubernetes 填写cloud详细信息 - Kubernetes地址：/root/.kube/config文件中cluster部分中server的内容 - Kubernetes命名空间：/root/.kube/config文件中cluster部分中name的内容 - Jenkins地址：jenkins服务的地址 - kubernetes服务证书key：ca.crt内容 - 凭据：选择刚刚创建的Certificate凭据配置完成后点击连接测试，显示k8s集群版本，证明配置无误。四、动态slave介绍 4.1为什么需要动态slave目前大多公司都采用 Jenkins 集群来搭建符合需求的 CI/CD 流程，然而传统的 Jenkins Slave 一主多从方式会存在一些痛点，比如： - 主 Master 发生单点故障时，整个流程都不可用了 - 每个 Slave 的配置环境不一样，来完成不同语言的编译打包等操作，但是这些差异化的配置导致管理起来非常不方便，维护起来也是比较费劲 - 资源分配不均衡，有的 Slave 要运行的 job 出现排队等待，而有的 Slave 处于空闲状态 - 资源有浪费，每台 Slave 可能是物理机或者虚拟机，当 Slave 处于空闲状态时，也不会完全释放掉资源。 正因为上面的这些种种痛点，我们渴望一种更高效更可靠的方式来完成这个 CI/CD 流程，而 Docker虚拟化容器技术能很好的解决这个痛点，又特别是在 Kubernetes 集群环境下面能够更好来解决上面的问题，下图是基于 Kubernetes 搭建 Jenkins 集群的简单示意图：从图上可以看到 Jenkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。 这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。4.2Jenkins Slave好处- 服务高可用，当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master 容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用(这是k8s带来的资源控制器带来的优势) - 动态伸缩，合理使用资源，每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave 自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配 Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。 - 扩展性好，当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个 Kubernetes Node 到集群中，从而实现扩展。五、动态slave配置 5.1制作slave镜像slave镜像应该包含以下功能： - 运行jenkins-agent服务 - 使用kubectl命令操作k8s集群 - 使用nerdctl工具管理container镜像 - 使用buildctl构建container镜像。#获取文件 root@k8s-01:~/jenkins/work# cp /usr/bin/kubectl . root@k8s-01:~/jenkins/work# cp /usr/bin/nerdctl . root@k8s-01:~/jenkins/work# cp /usr/local/bin/buildctl . root@k8s-01:~/jenkins/work# ls buildctl Dockerfile kubectl nerdctl#构建镜像 #在构建镜像过程中基于inbound-agent镜像，因为其中已经包含了jenkins-agent服务相关组件，再添加kubectl工具用于操作k8s，nerdctl和buildctl工具用于构建和管理container镜像。 root@k8s-01:~/jenkins/work# cat Dockerfile FROM jenkins/inbound-agent:latest-jdk17 USER root COPY kubectl /usr/bin/kubectl COPY nerdctl /usr/bin/nerdctl COPY buildctl /usr/bin/buildctl root@k8s-01:~/jenkins/work# docker build -t jenkins-agent:v1 . [+] Building 714.4s (9/9) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 181B 0.0s => [internal] load metadata for docker.io/jenkins/inbound-agent:latest-jdk17 4.1s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [1/4] FROM docker.io/jenkins/inbound-agent:latest-jdk17@sha256:591ba0391e1dc47af64432198be00a9e457c74d215970c4f1af592a21 709.8s => => resolve docker.io/jenkins/inbound-agent:latest-jdk17@sha256:591ba0391e1dc47af64432198be00a9e457c74d215970c4f1af592a210a 0.0s => => sha256:a90156da31f7db1823e946282d3743d1f917b9622d122a1d45818ef43c6f5dc9 8.35kB / 8.35kB 0.0s => => sha256:59e22667830bf04fb35e15ed9c70023e9d121719bb87f0db7f3159ee7c7e0b8d 28.23MB / 28.23MB 327.6s => => sha256:9ac051fdbd99f7d8c9e496724860b8ae3373f24d6f8a54f1d9096526df425d3c 43.12MB / 43.12MB 581.2s => => sha256:85a4e35755fb1aa44b91602297dc8d9f10eb8ad3f32baab32094cebc0eda41a4 3.32kB / 3.32kB 0.8s => => sha256:591ba0391e1dc47af64432198be00a9e457c74d215970c4f1af592a210a6c37b 3.14kB / 3.14kB 0.0s => => sha256:7eddb97bde91ed86d7e82e7fa5f23370b33e87882aafdbcfb2782fdec95f6231 2.19kB / 2.19kB 0.0s => => sha256:cef14de45bb7cc343e593f80531764848fe724db9084ae4a3cabacc7a7e24083 1.24MB / 1.24MB 14.3s => => sha256:28f95146d6851ca39a2ce18612cb5e5b19845ef85e4bb95bfa3095193fdf5777 1.24MB / 1.24MB 28.5s => => sha256:68a03bb16ee6a2c356d2e354bab5ec566dd7ef1e4e6daee52c1f87aa9d0cd139 62.99MB / 62.99MB 709.1s => => extracting sha256:59e22667830bf04fb35e15ed9c70023e9d121719bb87f0db7f3159ee7c7e0b8d 2.1s => => sha256:1c55318e78a1d3f438c6ca3cf6532d365a86fabbf7e3ecd14140e455f1489991 161B / 161B 328.8s => => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 329.5s => => sha256:392ad068a4827c711e9af3956bd5d3bcac7e8d66b3dd35bb0a9fae60de47f80d 2.37kB / 2.37kB 330.2s => => extracting sha256:85a4e35755fb1aa44b91602297dc8d9f10eb8ad3f32baab32094cebc0eda41a4 0.0s => => sha256:65375761a96d26587431452e982c657d479c41c41027a7f0acf37a6a21fd1112 180B / 180B 330.9s => => extracting sha256:9ac051fdbd99f7d8c9e496724860b8ae3373f24d6f8a54f1d9096526df425d3c 2.3s => => extracting sha256:cef14de45bb7cc343e593f80531764848fe724db9084ae4a3cabacc7a7e24083 0.0s => => extracting sha256:28f95146d6851ca39a2ce18612cb5e5b19845ef85e4bb95bfa3095193fdf5777 0.0s => => extracting sha256:68a03bb16ee6a2c356d2e354bab5ec566dd7ef1e4e6daee52c1f87aa9d0cd139 0.6s => => extracting sha256:1c55318e78a1d3f438c6ca3cf6532d365a86fabbf7e3ecd14140e455f1489991 0.0s => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.0s => => extracting sha256:392ad068a4827c711e9af3956bd5d3bcac7e8d66b3dd35bb0a9fae60de47f80d 0.0s => => extracting sha256:65375761a96d26587431452e982c657d479c41c41027a7f0acf37a6a21fd1112 0.0s => [internal] load build context 0.7s => => transferring context: 107.93MB 0.7s => [2/4] COPY kubectl /usr/bin/kubectl 0.2s => [3/4] COPY nerdctl /usr/bin/nerdctl 0.1s => [4/4] COPY buildctl /usr/bin/buildctl 0.1s => exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:ad852c7e884a5f9f6e87fcb6112fbe0c616b601a69ae5cf74ba09f2456d4e578 0.0s => => naming to docker.io/library/jenkins-agent:v1 0.0s root@k8s-01:~/jenkins# docker login --username=aliyun3891595718 registry.cn-guangzhou.aliyuncs.com Password: WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded root@k8s-01:~/jenkins# docker push registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 The push refers to repository [registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku] 2de7e3a095aa: Pushed 235bf2aaf23c: Pushed 9c18f60b53bc: Pushed ec380c7051cf: Pushed 0c3e60359c38: Pushed 5f70bf18a086: Pushed 433332898d75: Pushed b2c56f1e2ab9: Pushed 1411e4580b87: Pushed df5f57ff732b: Pushed 3f360379e3d1: Pushed f03c3dd333b7: Pushed 7cc7fe68eff6: Pushed v1: digest: sha256:b3e519ae85d0f05ff170778c8ffae494879397d9881ca8bc905bc889da82fc07 size: 30475.2创建kube-config资源为了能让slave容器中能够使用 kubectl 工具来访问我们的 Kubernetes 集群，需要将其添加为secret资源，并挂载到pod中。root@k8s-01:~/jenkins/work# kubectl create secret generic -n cicd kube-config --from-file=/root/.kube/config secret/kube-config created5.2.1测试创建一个自由风格的流水线 下面是配置截图正常的日志显示，会起一个pod来执行的命令结束以后就会销毁5.3节点开启buildkit服务(可选)container容器运行时仅能运行容器，如果需要在CICD阶段构建镜像，则需要在执行构建镜像的节点手动安装buildkit服务并启用，具体步骤可参考文档：https://axzys.cn/index.php/archives/536/ 也可以在slave pod中新增一个container，运行buildkit服务。5.4配置Pod Template(可选)配置 Pod Template，就是配置 Jenkins Slave 运行的 Pod 模板，命名空间我们同样是用cicd，Labels设置为jenkins-slave，对于后面执行 Job 的时候需要用到该值，容器名称填写jnlp，这样可以替换默认的agent容器。镜像使用的是刚刚我们制作的slave镜像，加入了 kubectl 等一些实用的工具。 运行命令和命令参数为空。另外需要注意我们这里需要在下面挂载三个目录 /run/containerd/containerd.sock：该文件是用于 Pod 中的容器能够共享宿主机的Container，用于管理container镜像。 /root/.kube：将之前创建的kube-config资源挂载到容器的/root/.kube目录下，这样能够在 Pod 的容器中能够使用 kubectl 工具来访问我们的 Kubernetes 集群，方便我们后面在 Slave Pod 部署 Kubernetes 应用 /run/buildkit：该文件是用于 Pod 中的容器能够共享buildkit进程，用于构建container镜像。同时指定Service Accoun为之前创建的jenkins-admin除了在页面配置pod Template外，我们也可以通过pipeline配置。六、测试Kubernetes 插件的配置工作完成了，接下来我们就来添加一个 Job 任务，看是否能够在 Slave Pod 中执行，任务执行完成后看 Pod 是否会被销毁。6.1自由流水线测试创建自由流水线任务，勾选限制项目的运行节点，标签表达式填写我们配置的 Slave Pod 中的 Label，这两个地方必须保持一致。然后往下拉，在 Build 区域选择Execute shellecho "Hello Kubernetes" echo "测试获取Kubernetes信息" kubectl get node echo "测试获取container信息" nerdctl ns ls echo "测试buildkitd构建镜像" echo "FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest" > Dockerfile echo 'CMD ["echo","hello","container"]' >> Dockerfile nerdctl build -t buildkitd-test:v1 . nerdctl images | grep buildkitd-test现在我们直接在页面点击做成的 Build now （立即构建）触发构建即可，然后观察 Kubernetes 集群中 Pod 的变化root@k8s-03:~/bin# kubectl get pods -n cicd -o wide -w NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES jenkins-7d65887794-s4vhr 1/1 Running 0 9h 10.244.2.104 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Pending 0 0s <none> <none> <none> <none> jenkins-agent-8x3mn 0/1 Pending 0 0s <none> k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 ContainerCreating 0 0s <none> k8s-03 <none> <none> jenkins-agent-8x3mn 1/1 Running 0 1s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 1/1 Terminating 0 13s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 13s <none> k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 14s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 14s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 14s 10.244.2.116 k8s-03 <none> <none>6.2pipeline-使用pod Template在流水线中指定pipeline脚本pipeline脚本如下：podTemplate(label: 'jenkins-slave', inheritFrom: 'jenkins-agent', cloud: 'k8s-local'){ node('jenkins-slave') { stage('测试获取Kubernetes信息') { sh 'kubectl get node' } stage('测试获取container信息') { sh 'nerdctl ns ls' } stage('测试buildkitd构建镜像'){ sh '''echo "FROM busybox" > Dockerfile echo \'CMD ["echo","hello","container"]\' >> Dockerfile nerdctl build -t buildkitd-test:v2 . nerdctl images | grep buildkitd-test''' } } }点击立即构建，查看控制台输出。6.3pipeline-自定义pod Template//创建一个Pod的模板，label为jenkins-agent podTemplate(label: 'jenkins-agent', cloud: 'k8s-local', containers: [ containerTemplate( name: 'jnlp', image: "harbor.local.com/cicd/jenkins-agent:v3", workingDir: '/home/jenkins/agent' ), containerTemplate( name: 'buildkitd', image: "harbor.local.com/cicd/buildkit:v0.13.2", privileged: true )], volumes:[ hostPathVolume(mountPath: '/run/containerd/containerd.sock', hostPath:'/run/containerd/containerd.sock'), secretVolume(mountPath: '/root/.kube/', secretName: 'kube-config', defaultMode: '420'), hostPathVolume(mountPath: '/run/buildkit',hostPath: '/run/buildkit') ] ) // 使用上文创建的pod模板 { node('jenkins-agent'){ stage('测试获取Kubernetes信息') { sh 'kubectl get node' } stage('测试获取container信息') { sh 'nerdctl ns ls' } stage('测试buildkitd构建镜像'){ sh '''echo "FROM busybox" > Dockerfile echo 'CMD ["echo","hello","container"]' >> Dockerfile nerdctl build -t buildkitd-test:v2 . nerdctl images | grep buildkitd-test''' } } }运行结果与上文一致。
一、发布与回滚思路正常功能发布时，是基于master分支发布的，所以我在成功发布后，会将当时的master分支自动打上tag，当需要回滚时，则基于tag分支进行发布即可。二、安装配置Git Parameter 2.1安装插件要想出现tag模式的参数，需要安装git Parameter 插件，在Jenkins的Manage Jenkins→Plugins→Available Plugins 中安装2.2验证安装完成后在项目的配置页的This project is parameterized 中可以看到选项2.3仓库添加tag初始化仓库，添加tag并提交root@k8s-03:~/vue3_vite_element-plus# git config --global user.name "xing" root@k8s-03:~/vue3_vite_element-plus# git config --global user.email "7902731@qq.com" root@k8s-03:~/vue3_vite_element-plus# git config --global --list user.name=xing user.email=7902731@qq.com root@k8s-03:~/vue3_vite_element-plus# git tag -a v1.0 -m "1.0版本" root@k8s-03:~/vue3_vite_element-plus# git tag -l v1.0 root@k8s-03:~/vue3_vite_element-plus# git push origin v1.0 Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': Enumerating objects: 1, done. Counting objects: 100% (1/1), done. Writing objects: 100% (1/1), 160 bytes | 160.00 KiB/s, done. Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 To http://192.168.30.181/xing/vue3_vite_element-plus.git * [new tag] v1.0 -> v1.0修改部分代码，并提交新版本。root@k8s-03:~/vue3_vite_element-plus# ls Dockerfile Jenkinsfile package.json README.md src vite.config.mjs index.html nginx.conf public screenshot test webstorm.config.js root@k8s-03:~/vue3_vite_element-plus# vi Dockerfile root@k8s-03:~/vue3_vite_element-plus# git commit -m "更新至v2" . [master 636d69b] 更新至v2 1 file changed, 2 insertions(+), 1 deletion(-) root@k8s-03:~/vue3_vite_element-plus# git tag -a v2.0 -m "2.0版本呢" root@k8s-03:~/vue3_vite_element-plus# git tag -l v1.0 v2.0 root@k8s-03:~/vue3_vite_element-plus# git push origin v2.0 Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': Enumerating objects: 6, done. Counting objects: 100% (6/6), done. Delta compression using up to 8 threads Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 423 bytes | 423.00 KiB/s, done. Total 4 (delta 2), reused 0 (delta 0), pack-reused 0 To http://192.168.30.181/xing/vue3_vite_element-plus.git * [new tag] v2.0 -> v2.0 root@k8s-03:~/vue3_vite_element-plus# 查看gitlab tag信息，发现已经有v1.0，v2.0tag三、使用tag变量发布 3.1发布最新版本生成pipeline，指定分支为${tag} root@k8s-03:~/vue3_vite_element-plus# git ls-remote --tags origin Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': 3afadbd4e09f012ec0cb89e4cae667f8564ba5fa refs/tags/v1.0 a934816659500449a9413148e69d113faf5cccea refs/tags/v1.0^{} cf126d51958d9ed649b6b73e0f90929c6ac58694 refs/tags/v2.0 636d69bff7fb45b6732e977fdeee56c3628a2dcc refs/tags/v2.0^{}整体思路就是发布后就会有tar版本，如果有问题马上可以回滚到稳定的版本
一、安装配置Maven此处以rpm包部署jenkins为例，以下操作在jenkins所在服务器执行。 maven下载地址：https://maven.apache.org/download.cgi1.1安装Maven[root@jenkins ~]# wget https://dlcdn.apache.org/maven/maven-3/3.9.6/binaries/apache-maven-3.9.6-bin.tar.gz [root@jenkins ~]# mkdir /usr/local/maven [root@jenkins ~]# tar -zxf apache-maven-3.9.6-bin.tar.gz -C /usr/local/maven/ [root@jenkins ~]# cd /usr/local/maven/apache-maven-3.9.6/ [root@jenkins apache-maven-3.9.3]# ls bin boot conf lib LICENSE NOTICE README.txt1.2设置maven的阿里云镜像[root@jenkins apache-maven-3.9.6]# vim conf/settings.xml # 在159行的标签为</mirrors>前添加如下阿里云镜像 <mirror> <id>alimaven</id> <name>aliyun maven</name> <url>http://maven.aliyun.com/nexus/content/groups/public/</url> <mirrorOf>central</mirrorOf> </mirror>1.3配置环境变量tar -zxvf amazon-corretto-17.0.16.8.1-linux-x64.tar.gz -C /usr/lib/jvm/ root@k8s-03:/usr/lib/jvm# ls amazon-corretto-17.0.16.8.1-linux-x64 root@k8s-03:/usr/lib/jvm# sudo chown -R root:root /usr/lib/jvm/amazon-corretto-17.0.16.8.1-linux-x64/ root@k8s-03:/usr/lib/jvm# export JAVA_HOME=/usr/lib/jvm/amazon-corretto-17.0.16.8.1-linux-x64/ root@k8s-03:/usr/lib/jvm# export PATH=$JAVA_HOME/bin:$PATH root@k8s-03:/usr/lib/jvm# java -version openjdk version "17.0.16" 2025-07-15 LTS OpenJDK Runtime Environment Corretto-17.0.16.8.1 (build 17.0.16+8-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.16.8.1 (build 17.0.16+8-LTS, mixed mode, sharing) root@k8s-03:/usr/lib/jvm# javac -version javac 17.0.16 root@k8s-03:/usr/lib/jvm# vi /etc/profile root@k8s-03:/usr/lib/jvm# cat /etc/profile # /etc/profile: system-wide .profile file for the Bourne shell (sh(1)) # and Bourne compatible shells (bash(1), ksh(1), ash(1), ...). if [ "${PS1-}" ]; then if [ "${BASH-}" ] && [ "$BASH" != "/bin/sh" ]; then # The file bash.bashrc already sets the default PS1. # PS1='\h:\w\$ ' if [ -f /etc/bash.bashrc ]; then . /etc/bash.bashrc fi else if [ "$(id -u)" -eq 0 ]; then PS1='# ' else PS1='$ ' fi fi fi if [ -d /etc/profile.d ]; then for i in /etc/profile.d/*.sh; do if [ -r $i ]; then . $i fi done unset i fi export MAVEN_HOME=/usr/local/maven/apache-maven-3.9.9 export PATH=${MAVEN_HOME}/bin:${PATH} JAVA_HOME=/usr/lib/jvm/amazon-corretto-17.0.16.8.1-linux-x64 PATH=$JAVA_HOME/bin:$PATH export JAVA_HOME PATH root@k8s-03:/usr/lib/jvm# source /etc/profile root@k8s-03:/usr/lib/jvm# mvn -v Apache Maven 3.9.9 (8e8579a9e76f7d015ee5ec7bfcdc97d260186937) Maven home: /usr/local/maven/apache-maven-3.9.9 Java version: 17.0.16, vendor: Amazon.com Inc., runtime: /usr/lib/jvm/amazon-corretto-17.0.16.8.1-linux-x64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.15.0-151-generic", arch: "amd64", family: "unix" 二、jenkins配置Maven 2.1全局工具配置关联jdk和mavenjenkis——>manage jenkins——>tools 如遇yum安装jdk无法识别，可尝试下载二进制openjdk安装2.2添加jenkins全局变量jenkis——>manage jenkins——>System 新增JAVA_HOME、M2_HOME、PATH+EXTRA三、拉取java项目 3.1创建项目此处以springboot项目为例，项目地址https://gitee.com/axzys/sprint_boot_demo3.2创建凭据依次点击jenkins——>系统管理——>Credentials——> Add Credentials，类型选择username with password四、创建任务 4.1创建流水线任务新建一个类型为自由风格的任务4.2配置git仓库信息gitee的主分支名称为master4.3构建测试点击立即构建，查看构建信息五、打包测试 5.1修改任务修改gitlab与gitee连接中配置的任务，新增构建步骤。配置如下5.2构建测试可以看到控制台成功打印了打包信息六、读取pom.xml参数在执行 Java 项目的流水线时，我们经常要动态获取项目中的属性，很多属性都配置在项目的 pom.xml 中，使用Pipeline Utility Steps 插件提供能够读取 pom.xml 的方法，pipeline如下stage('读取pom.xml参数阶段'){ // 读取 Pom.xml 参数 pom = readMavenPom file: './pom.xml' // 输出读取的参数 print "${pom.artifactId}" print = "${pom.version}" }