Jenkins+k8s项目实战

一、Jenkins动态slave介绍
1.1为什么需要动态slave

1. 配置管理困难：不同项目可能使用不同的编程语言、框架或库，这导致了每个Slave的配置环境各不相同。因此，需要动态Slave能够根据不同的项目需求，灵活配置不同的运行环境，从而简化配置管理和维护工作。
2. 资源分配不均衡：在使用静态Slave时，可能会出现某些Slave处于空闲状态，而其他Slave却处于繁忙状态，导致资源分配不均衡。动态Slave可以根据当前任务的需求自动调配资源，使得任务能够在空闲的Slave上尽快执行，从而提高资源利用率和任务执行效率。
3. 资源浪费：静态Slave在没有任务执行时仍然占用着资源，这导致了资源的浪费。而动态Slave能够根据实际需要自动扩容或缩减，当没有任务执行时会释放资源，从而避免了资源的浪费。

1.2动态slave工作流程

正因为上面的这些种种痛点，我们渴望一种更高效更可靠的方式来完成这个 CI/CD 流程，而 Docker虚拟化容器技术能很好的解决这个痛点，又特别是在 Kubernetes 集群环境下面能够更好来解决上面的问题，下图是基于 Kubernetes 搭建 Jenkins 集群的简单示意图：

从图上可以看到 Jenkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。

这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。

二、服务部署

本项目所有服务均运行在k8s集群上，使用nfs共享存储
nfs共享存储部署
container部署
harbor部署
gitlab部署
jenkins部署
SonarQube部署

三、项目与权限配置
3.1Harbor配置

Harbor的项目分为公开和私有的:
公开项目:所有用户都可以访问，通常存放公共的镜像，默认有一个library公开项目。
私有项目:只有授权用户才可以访问，通常存放项目本身的镜像。 我们可以为微服务项目创建一个新的项目

创建用户
创建一个普通用户xing。

配置项目用户权限
在spring_boot_demo项目中添加普通用户xing，并设置角色为开发者。

权限说明

角色	权限
访客	对项目有只读权限
开发人员	对项目有读写权限
维护人员	对项目有读写权限、创建webhook权限
项目管理员	出上述外，还有用户管理等权限

3.2gitlab项目权限配置

创建组
管理员用户登录，创建群组，组名称为develop，组权限为私有

创建项目
创建sprint boot demo项目，并指定develop，项目类型为私有

创建用户
创建一个普通用户xing

用户添加到组中
将xing添加到群组develop中，cuiliang角色为Developer

配置分支权限

用户权限验证
使用任意一台机器模拟开发人员拉取代码，完成开发后推送至代码仓库。
拉取仓库代码

#拉取代码
root@k8s-03:~/work# git clone https://gitee.com/cuiliang0302/sprint_boot_demo.git
Cloning into 'sprint_boot_demo'...
remote: Enumerating objects: 261, done.
remote: Total 261 (delta 0), reused 0 (delta 0), pack-reused 261 (from 1)
Receiving objects: 100% (261/261), 105.79 KiB | 162.00 KiB/s, done.
Resolving deltas: 100% (116/116), done.
root@k8s-03:~/work# git remote set-url origin http://192.168.30.181/develop/sprint-boot-demo.git
fatal: not a git repository (or any of the parent directories): .git

推送至gitlab仓库

root@k8s-03:~/work/sprint_boot_demo# git remote set-url origin http://192.168.30.181/develop/sprint-boot-demo.git
root@k8s-03:~/work/sprint_boot_demo# git remote -v
origin    http://192.168.30.181/develop/sprint-boot-demo.git (fetch)
origin    http://192.168.30.181/develop/sprint-boot-demo.git (push)
root@k8s-03:~/work/sprint_boot_demo# git push --set-upstream origin --all
Username for 'http://192.168.30.181': xing
Password for 'http://xing@192.168.30.181': 
Enumerating objects: 254, done.
Counting objects: 100% (254/254), done.
Delta compression using up to 8 threads
Compressing objects: 100% (119/119), done.
Writing objects: 100% (254/254), 105.20 KiB | 105.20 MiB/s, done.
Total 254 (delta 111), reused 253 (delta 110), pack-reused 0
remote: Resolving deltas: 100% (111/111), done.
remote: 
remote: To create a merge request for master, visit:
remote:   http://192.168.30.181/develop/sprint-boot-demo/-/merge_requests/new?merge_request%5Bsource_branch%5D=master
remote: 
To http://192.168.30.181/develop/sprint-boot-demo.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

查看验证

四、jenkins配置
4.1插件安装与配置

GitLab插件安装与配置：https://www.cuiliangblog.cn/detail/section/127410630

SonarQube Scanner插件安装与配置：https://www.cuiliangblog.cn/detail/section/165534414

Kubernetes插件安装与配置：https://www.cuiliangblog.cn/detail/section/127230452

Email Extension邮件推送插件安装与配置：https://www.cuiliangblog.cn/detail/section/133029974

Version Number版本号插件安装与配置：https://plugins.jenkins.io/versionnumber/

Content Replace文件内容替换插件安装与配置：https://plugins.jenkins.io/content-replace/

4.2jenkins slave镜像制作

安装完Kubernetes插件后，默认的slave镜像仅包含一些基础功能和软件包，如果需要构建镜像，执行kubectl命令，则需要引入其他container或者自定义slave镜像。

关于镜像构建问题，如果k8s容器运行时为docker，可以直接使用docker in docker方案，启动一个docker:dind容器，通过Docker pipeline插件执行镜像构建与推送操作，具体内容可参考https://www.cuiliangblog.cn/detail/section/166573065。

如果k8s容器运行时为container，则使用nerdctl+buildkitd方案，启动一个buildkit容器，通过nerdctl命令执行镜像构建与推送操作，具体内容可参考:https://axzys.cn/index.php/archives/521/

本次实验以container环境为例，通过nerdctl+buildkitd方案演示如何构建并推送镜像。

构建jenkins-slave镜像

root@k8s-01:~/jenkins/work# cp /usr/bin/kubectl .
root@k8s-01:~/jenkins/work# cp /usr/bin/nerdctl .
root@k8s-01:~/jenkins/work# cp /usr/local/bin/buildctl .
root@k8s-01:~/jenkins/work# ls
buildctl  Dockerfile  kubectl  nerdctl

测试jenkins-slave镜像构建容器与操作k8s
以下操作在k8s集群master机器，容器运行时为container节点执行测试

# 启动buildkit镜像构建服务
# 挂载/run/containerd/containerd.sock方便container调用buildkitd
# 挂载/var/lib/buildkit，以便于将构建过程中下载的镜像持久化存储，方便下次构建时使用缓存
# 挂载/run/buildkit/目录方便nerctl调用buildkitd
root@k8s-03:~/bin# nerdctl run --name buildkit -d --privileged=true \
-v /run/buildkit/:/run/buildkit/ \
-v /var/lib/buildkit:/var/lib/buildkit \
-v /run/containerd/containerd.sock:/run/containerd/containerd.sock \
registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2
registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2:              resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:ff1ed58245d6871cc4bbe07a838603720a90ca33f124484fff2af61b344c3b2f: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:b67e88949be2f8ee8844bbe205c8b0054533a2c8cf35a6bc39ebc1d7cd7ce8f1:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:1479f6e1da81ab38384f704f742a35030364df4c9f9ed65e812a9c921bc20d25:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4abcf20661432fb2d719aaf90656f55c287f8ca915dc1c92ec14ff61e67fbaf8:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:77e3817cafb06118d96cfbd8af2fb7834a03e14d83acbcd9b7ff2a298c292d4d:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:fabeb51fadd96525499e8ce7171956d81bf70c836d4943ff8be9714528da736a:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 19.4s                                                                    total:  87.2 M (4.5 MiB/s)                                       
661d91a4ade1379768948ea962541ce5876e4dc51c14d3673fedfbdb7d142af5
root@k8s-03:~/bin# nerdctl ps 
CONTAINER ID    IMAGE                                                                  COMMAND        CREATED               STATUS    PORTS    NAMES
4778573cadf3    registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2    "buildkitd"    About a minute ago    Up                 buildkit
root@k8s-03:~/bin# 
# 启动jenkins-slave容器
# 挂载/run/containerd/containerd.sock方便netdctl操作container
# 挂载/run/buildkit/目录方便nerctl调用buildkitd构建镜像
# 挂载/root/.kube/目录方便kubectl工具操作k8s
root@k8s-03:~/bin# nerdctl run --name jenkins-slave -it --privileged=true -v /run/buildkit/:/run/buildkit/ -v /root/.kube/:/root/.kube/ -v /run/containerd/containerd.sock:/run/containerd/containerd.sock registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 bash
registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1:                  resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:b3e519ae85d0f05ff170778c8ffae494879397d9881ca8bc905bc889da82fc07: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:ad852c7e884a5f9f6e87fcb6112fbe0c616b601a69ae5cf74ba09f2456d4e578:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:9ac051fdbd99f7d8c9e496724860b8ae3373f24d6f8a54f1d9096526df425d3c:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:85a4e35755fb1aa44b91602297dc8d9f10eb8ad3f32baab32094cebc0eda41a4:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0bdf2c4d1714b5962675435789b4e83edb5aa4d94ec0d7643737940b0e73c4ed:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6b189407e830325d112140133ef8a0adcae1a94a9231cfa56872c78f21886b66:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:28f95146d6851ca39a2ce18612cb5e5b19845ef85e4bb95bfa3095193fdf5777:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:68a03bb16ee6a2c356d2e354bab5ec566dd7ef1e4e6daee52c1f87aa9d0cd139:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:1c55318e78a1d3f438c6ca3cf6532d365a86fabbf7e3ecd14140e455f1489991:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:65375761a96d26587431452e982c657d479c41c41027a7f0acf37a6a21fd1112:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:f26c0daf8d2d5ff437f0eb85dad34fe122657585c3e8e5589fdf4cd007705fbe:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:cef14de45bb7cc343e593f80531764848fe724db9084ae4a3cabacc7a7e24083:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:7db9f9afd5f7e18ef5e410566fe6342de266960271c5ba712b8027e593617498:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:392ad068a4827c711e9af3956bd5d3bcac7e8d66b3dd35bb0a9fae60de47f80d:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 4.9 s                                                                    total:  170.8  (34.9 MiB/s)                                      
root@4da1c7c8f6f4:/home/jenkins# 
# 测试container管理
root@4da1c7c8f6f4:/home/jenkins# nerdctl ps
CONTAINER ID    IMAGE                                                                  COMMAND                   CREATED           STATUS    PORTS    NAMES
4da1c7c8f6f4    registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1        "/usr/local/bin/jenk…"    47 seconds ago    Up                 jenkins-slave
4778573cadf3    registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2    "buildkitd"               6 minutes ago     Up                 buildkit
root@4da1c7c8f6f4:/home/jenkins# 
#测试k8s管理
root@4da1c7c8f6f4:/home/jenkins# kubectl get node
NAME     STATUS   ROLES           AGE   VERSION
k8s-01   Ready    control-plane   5d    v1.27.6
k8s-02   Ready    <none>          5d    v1.27.6
k8s-03   Ready    <none>          5d    v1.27.6
# 测试镜像构建

解释一下下面的操作，如果是宿主机已经开启了buildkitd服务必须先关闭，不然/var/lib/buildkit/buildkitd.lock会生成，这个生成后Docker启动不了buildkitd容器。

#停止buildkitd服务
root@k8s-03:/var/lib/buildkit# sudo systemctl stop buildkitd
sudo systemctl disable buildkitd

# 2. 清理残留文件
sudo rm -f /run/buildkit/buildkitd.sock
sudo rm -f /run/buildkit/otel-grpc.sock
sudo rm -f /var/lib/buildkit/buildkitd.lock

# 3. 创建专用数据目录（避免冲突）
sudo mkdir -p /var/lib/buildkit-container
sudo chmod 700 /var/lib/buildkit-container
Removed /etc/systemd/system/multi-user.target.wants/buildkitd.service.
# 启动buildkit镜像构建服务
# 挂载/run/containerd/containerd.sock方便container调用buildkitd
# 挂载/var/lib/buildkit，以便于将构建过程中下载的镜像持久化存储，方便下次构建时使用缓存
# 挂载/run/buildkit/目录方便nerctl调用buildkitd
root@k8s-03:/var/lib/buildkit# nerdctl run -d --name buildkit \
  --privileged \
  -v /run/buildkit/:/run/buildkit/ \
  -v /var/lib/buildkit-container:/var/lib/buildkit \
  -v /run/containerd/containerd.sock:/run/containerd/containerd.sock \
  registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2
a9d371968d62cb654e50677bb7433b6c58ac25cf177c76dcc81dfb3a311b8289
root@k8s-03:/var/lib/buildkit# nerdctl ps 
CONTAINER ID    IMAGE                                                                  COMMAND        CREATED          STATUS    PORTS    NAMES
a9d371968d62    registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2    "buildkitd"    7 seconds ago    Up                 buildkit
root@k8s-03:/var/lib/buildkit# 
root@k8s-03:/var/lib/buildkit# 
root@k8s-03:/var/lib/buildkit# buildctl --addr unix:///run/buildkit/buildkitd.sock debug workers
ID                PLATFORMS
37647lrzalh7j20qy1hskzedp    linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/386
qi74wmika7i8tvdej2ob0ooj5    linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/386
root@k8s-03:/var/lib/buildkit# 
root@k8s-03:/var/lib/buildkit# 
# 启动jenkins-slave容器
# 挂载/run/containerd/containerd.sock方便netdctl操作container
# 挂载/run/buildkit/目录方便nerctl调用buildkitd构建镜像
# 挂载/root/.kube/目录方便kubectl工具操作k8s
root@k8s-03:/var/lib/buildkit# nerdctl run --name jenkins-slave -it --privileged=true \
-v /run/buildkit/:/run/buildkit/ \
-v /root/.kube/:/root/.kube/ \
-v /run/containerd/containerd.sock:/run/containerd/containerd.sock \
registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 bash
# 测试镜像构建
root@d2f81acc6e0b:/home/jenkins# echo 'FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest' >> Dockerfile
echo 'CMD ["echo","hello","container"]' >> Dockerfile
cat Dockerfile 
FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest
CMD ["echo","hello","container"]
root@d2f81acc6e0b:/home/jenkins# 
root@d2f81acc6e0b:/home/jenkins# 
root@d2f81acc6e0b:/home/jenkins# nerdctl build -t test-test:v1 .
[+] Building 12.4s (3/5)                                                                                                             
[+] Building 12.5s (5/5) FINISHED                                                                                                    
 => [internal] load build definition from Dockerfile                                                                            0.0s
 => => transferring dockerfile: 143B                                                                                            0.0s
 => [internal] load metadata for registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest                           11.6s
 => [internal] load .dockerignore                                                                                               0.0s
 => => transferring context: 2B                                                                                                 0.0s
 => [1/1] FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest@sha256:b41a05bd7a4a32e4c48c284cc2178abe8c11  0.8s
 => => resolve registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest@sha256:b41a05bd7a4a32e4c48c284cc2178abe8c11  0.0s
 => => sha256:90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c 2.15MB / 2.15MB                                  0.8s
 => exporting to docker image format                                                                                            0.8s
 => => exporting layers                                                                                                         0.0s
 => => exporting manifest sha256:05610df32232fdd6d6276d0aa50c628fc3acd75deb010cf15a4ac74cf35ea348                               0.0s
 => => exporting config sha256:0b44030dca1d1504de8aa100696d5c86f19b06cec660cf55c2ba6c5c36d1fb89                                 0.0s
 => => sending tarball                                                                                                          0.0s
Loaded image: docker.io/library/test-test:v1
root@d2f81acc6e0b:/home/jenkins#

4.3job任务创建与配置

配置webhook构建触发器，当分支代码提交时触发构建，具体配置如下：

流水线选择SCM从代码仓库中获取jenkinsfile，脚本路径填写Jenkinsfile-k8s.groov

手动触发完会生成一个pod

4.4部署总结

1. jenkinsfile中如果涉及yaml的代码需要注意权限。
2. 还需要查看ServiceAccount是否跟之前设置的正确
3. 镜像尽量自己做成国内的地址
4. 如果是自定义的Harbor仓库需要提前创建Docker Registry Secret就是Harbor仓库的账号密码，然后在yaml中添加上配置imagePullSecrets。
5. 提前把名称空间创建出来

五、效果演示
5.1开发测试阶段

模拟开发人员完成功能开发后提交代码至test分支，推送以后会自动触发gitlab的webhook然后自动调用拉取jenkins你设置好的job流水线。

root@k8s-03:~/work/sprint-boot-demo# git branch -a
  main
* test
  remotes/origin/HEAD -> origin/main
  remotes/origin/main
root@k8s-03:~/work/sprint-boot-demo# cat src/main/java/com/example/springbootdemo/HelloWorldController.java
package com.example.springbootdemo;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
public class HelloWorldController {
    @RequestMapping("/")
    @ResponseBody
    public String hello() {
        // 获取环境变量 ENV_NAME，如果不存在则使用默认值 "default"
        String envName = System.getenv().getOrDefault("ENV_NAME", "default");
        return String.format("<h1>Hello SpringBoot</h1><p>Version:v1 Env:%s</p>", envName);
    }
    @RequestMapping("/health")
    @ResponseBody
    public String healthy() {
        return "ok";
    }
}
root@k8s-03:~/work/sprint-boot-demo# vi  src/main/java/com/example/springbootdemo/HelloWorldController.java
root@k8s-03:~/work/sprint-boot-demo# git add .
root@k8s-03:~/work/sprint-boot-demo# git commit -m "test环境更新版本至v2"
[test 368ff3d] test环境更新版本至v2
 1 file changed, 1 insertion(+), 1 deletion(-)
root@k8s-03:~/work/sprint-boot-demo# git push
fatal: The current branch test has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin test

root@k8s-03:~/work/sprint-boot-demo# 
root@k8s-03:~/work/sprint-boot-demo# git push --set-upstream origin test
Username for 'http://192.168.30.181': root
Password for 'http://root@192.168.30.181': 
Enumerating objects: 17, done.
Counting objects: 100% (17/17), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 697 bytes | 697.00 KiB/s, done.
Total 9 (delta 2), reused 0 (delta 0), pack-reused 0
remote: 
remote: To create a merge request for test, visit:
remote:   http://192.168.30.181/develop/sprint-boot-demo/-/merge_requests/new?merge_request%5Bsource_branch%5D=test
remote: 
To http://192.168.30.181/develop/sprint-boot-demo.git
 * [new branch]      test -> test
Branch 'test' set up to track remote branch 'test' from 'origin'.

此时查看cicd名称空间下的pod信息，发现已经创建一个名为springbootdemo-275-rf832-h6jkq-630x8的pod，包含3个container，分别是jnlp、maven、buildkitd。

root@k8s-02:~# kubectl get pods -n cicd 
NAME                                  READY   STATUS    RESTARTS       AGE
jenkins-7d65887794-tsqs5              1/1     Running   5 (104m ago)   2d8h
springbootdemo-40-p9smx-4mw1j-z0v30   3/3     Running   0              7m30s

查看jenkins任务信息，已顺利完成了集成部署工作。

并且收到了jenkins自动发出的邮件，内容如下：

查看SonarQube代码扫描信息，未发现异常代码。

查看k8s，已成功创建相关资源

root@k8s-01:~/gitlab# kubectl get all -n test
NAME                        READY   STATUS    RESTARTS   AGE
pod/demo-7d4975d656-cdpk7   1/1     Running   0          4m28s

NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/demo   ClusterIP   10.108.234.81   <none>        8888/TCP   14m

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo   1/1     1            1           14m

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-7d4975d656   1         1         1       14m
root@k8s-01:~/gitlab#

此时模拟测试人员，访问测试环境域名

至此，开发测试阶段演示完成。

5.2生产发布阶段

接下来演示master分支代码提交后，触发生产环境版本发布流程。

#Harbor仓库的权限要提前创建
kubectl create secret docker-registry harbor-pull-secret \
  --namespace=test \
  --docker-server=192.168.30.180:30003 \
  --docker-username=admin \
  --docker-password=Harbor12345

root@k8s-03:~/work/sprint-boot-demo#  git branch -a
  main
* test
  remotes/origin/HEAD -> origin/main
  remotes/origin/main
  remotes/origin/test
root@k8s-03:~/work/sprint-boot-demo# git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
root@k8s-03:~/work/sprint-boot-demo# vi  src/main/java/com/example/springbootdemo/HelloWorldController.java
root@k8s-03:~/work/sprint-boot-demo# cat   src/main/java/com/example/springbootdemo/HelloWorldController.java
package com.example.springbootdemo;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.ResponseBody;

@Controller
public class HelloWorldController {
    @RequestMapping("/")
    @ResponseBody
    public String hello() {
        // 获取环境变量 ENV_NAME，如果不存在则使用默认值 "default"
        String envName = System.getenv().getOrDefault("ENV_NAME", "default");
        return String.format("<h1>Hello SpringBoot</h1><p>Version:v5 Env:%s</p>", envName);
    }
    @RequestMapping("/health")
    @ResponseBody
    public String healthy() {
        return "ok";
    }
}
root@k8s-03:~/work/sprint-boot-demo# git add .
root@k8s-03:~/work/sprint-boot-demo# git commit -m "生产环境更新版本v5"
[main 617325e] 生产环境更新版本v5
 1 file changed, 1 insertion(+), 1 deletion(-)
root@k8s-03:~/work/sprint-boot-demo# git push
Username for 'http://192.168.30.181': root
Password for 'http://root@192.168.30.181': 
Enumerating objects: 17, done.
Counting objects: 100% (17/17), done.
Delta compression using up to 8 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (9/9), 704 bytes | 704.00 KiB/s, done.
Total 9 (delta 2), reused 0 (delta 0), pack-reused 0
To http://192.168.30.181/develop/sprint-boot-demo.git
   912f7b3..617325e  main -> main

待收到邮件通知后，查看k8s资源，已经在prod名称空间下创建相关资源

此时访问生产环境域名，服务可以正常访问。

此时查看Harbor仓库镜像信息，其中p开头的为生产环境镜像，t开头的为测试环境镜像。

jenkinsfile


pipeline {
    agent {
        kubernetes {
            // 定义要在 Kubernetes 中运行的 Pod 模板
            yaml '''
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: jenkins-slave
spec:
  serviceAccountName: jenkins-admin
  containers:
  - name: jnlp
    image: registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1
    resources:
      limits:
        memory: "512Mi"
        cpu: "500m"
    securityContext:
      privileged: true
    volumeMounts:
    - name: buildkit
      mountPath: "/run/buildkit/"
    - name: containerd
      mountPath: "/run/containerd/containerd.sock"
    - name: kube-config
      mountPath: "/root/.kube/"
      readOnly: true
  - name: maven
    image: registry.cn-guangzhou.aliyuncs.com/xingcangku/maven3.9.3-openjdk-17:v2.0
    resources:
      limits:
        memory: "512Mi"
        cpu: "500m"
    command:
      - 'sleep'
    args:
      - '9999'
    volumeMounts:
      - name: maven-data
        mountPath: "/root/.m2"
  - name: buildkitd
    image: registry.cn-guangzhou.aliyuncs.com/xingcangku/moby-buildkit:v0.13.2
    resources:
      limits:
        memory: "256Mi"
        cpu: "500m"
    securityContext:
      privileged: true
    volumeMounts:
    - name: buildkit
      mountPath: "/run/buildkit/"
    - name: buildkit-data
      mountPath: "/var/lib/buildkit/"
    - name: containerd
      mountPath: "/run/containerd/containerd.sock"
  volumes:
  - name: maven-data
    persistentVolumeClaim:
      claimName: jenkins-maven
  - name: buildkit
    hostPath:
      path: /run/buildkit/
  - name: buildkit-data
    hostPath:
      path: /var/lib/buildkit/
  - name: containerd
    hostPath:
      path: /run/containerd/containerd.sock
  - name: kube-config
    secret:
      secretName: kube-config
            '''
      retries 2
        }
    }
    environment {
        // 全局变量
        HARBOR_CRED = "harbor-admin"
        IMAGE_NAME = ""
        IMAGE_APP = "demo"
        branchName = ""
    }
    stages {
        stage('拉取代码') {
            environment {
                // gitlab仓库信息
                GITLAB_CRED = "gitlab-xing-password"
                GITLAB_URL = "http://192.168.30.181/develop/sprint-boot-demo.git"
            }
            steps {
                echo '开始拉取代码'
                checkout scmGit(branches: [[name: '*/*']], extensions: [], userRemoteConfigs: [[credentialsId: "${GITLAB_CRED}", url: "${GITLAB_URL}"]])
                // 获取当前拉取的分支名称
                script {
                    def branch = env.GIT_BRANCH ?: 'main'
                    branchName = branch.split('/')[-1]
                }
                echo '拉取代码完成'
            }
        }
        stage('编译打包') {
            steps {
                container('maven') {
                    // 指定使用maven container进行打包
                    echo '开始编译打包'
                    sh 'mvn clean package'
                    echo '编译打包完成'
                }
            }
        }
        stage('代码审查') {
            environment {
                // SonarQube信息
                SONARQUBE_SCANNER = "SonarQube"
                SONARQUBE_SERVER = "SonarQube"
            }
            steps {
                echo '开始代码审查'
                script {
                    def scannerHome = tool "${SONARQUBE_SCANNER}"
                    withSonarQubeEnv("${SONARQUBE_SERVER}") {
                        // 添加扫描参数
                        sh """
                        ${scannerHome}/bin/sonar-scanner \
                          -Dsonar.projectKey=springbootdemo \
                          -Dsonar.projectName=SpringBootDemo \
                          -Dsonar.java.binaries=target/classes \
                          -Dsonar.sources=src/main/java \
                          -Dsonar.sourceEncoding=UTF-8
                        """
                    }
                }
                echo '代码审查完成'
            }
        }
        stage('构建镜像') {
            environment {
                // harbor仓库信息
                HARBOR_URL = "192.168.30.180:30003"
                HARBOR_PROJECT = "spring_boot_demo"
                // 镜像标签
                IMAGE_TAG = ''
                // 镜像名称
                IMAGE_NAME = ''
            }
            steps {
                echo '开始构建镜像'
                script {
                    if (branchName == 'main') {
                        IMAGE_TAG = VersionNumber versionPrefix: 'p', versionNumberString: '${BUILD_DATE_FORMATTED, "yyMMdd"}.${BUILDS_TODAY}'
                    } else if (branchName == 'test') {
                        IMAGE_TAG = VersionNumber versionPrefix: 't', versionNumberString: '${BUILD_DATE_FORMATTED, "yyMMdd"}.${BUILDS_TODAY}'
                    } else {
                        error("Unsupported branch: ${params.BRANCH}")
                    }
                    IMAGE_NAME = "${HARBOR_URL}/${HARBOR_PROJECT}/${IMAGE_APP}:${IMAGE_TAG}"
                    sh "nerdctl build --insecure-registry -t ${IMAGE_NAME} . "
                }
                echo '构建镜像完成'
                echo '开始推送镜像'
                // 获取harbor账号密码
                withCredentials([usernamePassword(credentialsId: "${HARBOR_CRED}", passwordVariable: 'HARBOR_PASSWORD', usernameVariable: 'HARBOR_USERNAME')]) {
                    // 登录Harbor仓库
                    sh """nerdctl login --insecure-registry ${HARBOR_URL} -u ${HARBOR_USERNAME} -p ${HARBOR_PASSWORD}
          nerdctl push --insecure-registry ${IMAGE_NAME}"""
                }
                echo '推送镜像完成'
                echo '开始删除镜像'
                script {
                    sh "nerdctl rmi -f ${IMAGE_NAME}"
                }
                echo '删除镜像完成'
            }
        }
        stage('项目部署') {
            environment {
                // 资源清单名称
                YAML_NAME = "k8s.yaml"
            }
            steps {
                echo '开始修改资源清单'
                script {
                    if (branchName == 'main' ) {
                        NAME_SPACE = 'prod'
                        DOMAIN_NAME = 'demo.local.com'
                    } else if (branchName == 'test') {
                        NAME_SPACE = 'test'
                        DOMAIN_NAME = 'demo.test.com'
                    } else {
                        error("Unsupported branch: ${params.BRANCH}")
                    }
                }
                // 使用Content Replace插件进行k8s资源清单内容替换
                contentReplace(configs: [fileContentReplaceConfig(configs: [fileContentReplaceItemConfig(replace: "${IMAGE_NAME}", search: 'IMAGE_NAME'),
                                                                            fileContentReplaceItemConfig(replace: "${NAME_SPACE}", search: 'NAME_SPACE'),
                                                                            fileContentReplaceItemConfig(replace: "${DOMAIN_NAME}", search: 'DOMAIN_NAME')],
                        fileEncoding: 'UTF-8',
                        filePath: "${YAML_NAME}",
                        lineSeparator: 'Unix')])
                echo '修改资源清单完成'
                sh "cat ${YAML_NAME}"
                echo '开始部署资源清单'
                sh "kubectl apply -f ${YAML_NAME}"
                echo '部署资源清单完成'
            }
        }
    }
    post {
        always {
            echo '开始发送邮件通知'
            emailext(subject: '构建通知：${PROJECT_NAME} - Build # ${BUILD_NUMBER} - ${BUILD_STATUS}!',
                    body: '${FILE,path="email.html"}',
                    to: '7902731@qq.com')
            echo '邮件通知发送完成'
        }
    }
}

Jenkins+k8s项目实战

评论 (0)