搜索到
116
篇与
的结果
-
jenkins添加节点-slave集群配置 一、Jenkins的Master/Slave机制Jenkins采用Master/Slave架构。Master/Slave相当于Server和agent的概念，Master提供web接口让用户来管理Job和Slave，Job可以运行在Master本机或者被分配到Slave上运行。一个Master可以关联多个Slave用来为不同的Job或相同的Job的不同配置来服务。 Jenkins的Master/Slave机制除了可以并发的执行构建任务，加速构建以外。还可以用于分布式自动化测试，当自动化测试代码非常多或者是需要在多个浏览器上并行的时候，可以把测试代码划分到不同节点上运行，从而加速自动化测试的执行。二、集群角色功能**Master：**Jenkins服务器。主要是处理调度构建作业，把构建分发到Slave节点实际执行，监视Slave节点的状态。当然，也并不是说Master节点不能跑任务。构建结果和构建产物最后还是传回到Master节点，比如说在jenkins工作目录下面的workspace内的内容，在Master节点照样是有一份的。 **Slave：**执行机(奴隶机)。执行Master分配的任务，并返回任务的进度和结果。Jenkins Master/Slave的搭建需要至少两台机器，一台Master节点，一台Slave节点（实际生产中会有多个Slave节点）。三、搭建步骤Master不需要主动去建立，安装Jenkins，在登录到主界面时，这台电脑就已经默认为master。 选择“Manage Jenkins”->“Manage Nodes and Clouds”，可以看到Master节点相关信息：四、为Jenkins添加Slave Node 4.1开启tcp代理端口jenkins web代理是指slave通过jenkins服务端提供的一个tcp端口，与jenkins服务端建立连接，docker版的jenkins默认开启web tcp代理，端口为50000，而自己手动制作的jenkins容器或者在物理机环境部署的jenkins，都需要手动开启web代理端口，如果不开启，slave无法通过web代理的方式与jenkins建立连接。 jenkins web代理的tcp端口不是通过命令启动的而是通过在全局安全设置中配置的，配置成功后会在系统上运行一个指定的端口4.2添加节点信息在Jenkins界面选择“Manage Jenkins”->“Manage Nodes and Clouds”->“New Node配置Agent信息Name：Slave机器的名字 Description：描述 ，不重要 随意填 Number of excutors：允许在这个节点上并发执行任务的数量，即同时可以下发多少个Job到Slave上执行，一般设置为 cpu 支持的线程数。[注：Master Node也可以通过此参数配置Master是否也执行构建任务、还是仅作为Jenkins调度节点] Remote root directory：用来放工程的文件夹，jenkins master上设置的下载的代码会放到这个工作目录下。 Lables：标签，用于实现后续Job调度策略，根据Jobs配置的Label选择Salve Node Usage：支持两种模式“Use this Node as much as possible”、“Only build Jobs with Label expressiong matching this Node”。选择“Only build Jobs with Label expressiong matching this Node”， 添加完毕后，在Jenkins主界面，可以看到新添加的Slave Node，但是红叉表示此时的Slave并未与Master建立起联系。4.3slave节点配置安装jdk#dnf -y install java-17-openjdk root@k8s-02:~# java -version openjdk version "17.0.16" 2025-07-15 LTS OpenJDK Runtime Environment Corretto-17.0.16.8.1 (build 17.0.16+8-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.16.8.1 (build 17.0.16+8-LTS, mixed mode, sharing) 安装agent 点击节点信息，根据控制台提示执行安装agent命令root@k8s-02:~# curl -sO http://192.168.30.180:31530/jnlpJars/agent.jar root@k8s-02:~# java -jar agent.jar -url http://192.168.30.180:31530/ -secret 6e87c37900dfbdcff98099f6681f7b195a141de1cacb157679efc98f8fec2644 -name "k8s-02" -webSocket -workDir "/opt/jenkins" Aug 03, 2025 12:52:47 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /opt/jenkins/remoting as a remoting work directory Aug 03, 2025 12:52:47 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /opt/jenkins/remoting Aug 03, 2025 12:52:47 PM hudson.remoting.Launcher createEngine INFO: Setting up agent: k8s-02 Aug 03, 2025 12:52:47 PM hudson.remoting.Engine startEngine INFO: Using Remoting version: 3309.v27b_9314fd1a_4 Aug 03, 2025 12:52:47 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /opt/jenkins/remoting as a remoting work directory Aug 03, 2025 12:52:47 PM hudson.remoting.Launcher$CuiListener status INFO: WebSocket connection open Aug 03, 2025 12:52:47 PM hudson.remoting.Launcher$CuiListener status INFO: Connected4.4查看agent状态指定Node调度策略 创建Job的页面，“General”下勾选“Restric where this project can be run”，填写Label Expression。 -
jenkins与SonarQube连接 一、jenkins安装插件 1.1下载SonarQube插件进入Jenkins的系统管理->插件管理->可选插件，搜索框输入sonarqube，安装重启。1.2启用SonarQubeJenkins的系统管理->系统配置，添加SonarQube服务。二、SonarQube配置 2.1禁用审查结果上传到SCM功能2.2生成token添加jenkin用户 token：squ_4bc173eb520dd35c176104baa1b899a992e88c88三、jenkins配置 3.1添加令牌Jenkins的系统管理->系统配置->添加token类型切换成Secret text，粘贴token，点击添加。选上刚刚添加的令牌凭证，点击应用保存。3.2SonarQube Scanner 安装进入Jenkins的系统管理->全局工具配置，下滑找到图片里的地方，点击新增SonarQube Scanner，我们选择自动安装并选择最新的版本。四、非流水线项目添加代码审查 4.1添加构建步骤编辑之前的自由风格构建的demo项目，在构建阶段新增步骤。analysis properties参数如下# 项目名称id，全局唯一 sonar.projectKey=sprint_boot_demo # 项目名称 sonar.projectName=sprint_boot_demo sonar.projectVersion=1.0 # 扫描路径，当前项目根目录 sonar.sources=./src # 排除目录 sonar.exclusions=**/test/**,**/target/** # jdk版本 sonar.java.source=1.17 sonar.java.target=1.17 # 字符编码 sonar.sourceEncoding=UTF-8 # binaries路径 sonar.java.binaries=target/classes4.2构建并查看结果jenkins点击立即构建，查看构建结果查看SonarQube扫描结果五、流水线项目添加代码审查 5.1创建sonar-project.properties文件项目根目录下，创建sonar-project.properties文件，内容如下# 项目名称id，全局唯一 sonar.projectKey=sprint_boot_demo # 项目名称 sonar.projectName=sprint_boot_demo sonar.projectVersion=1.0 # 扫描路径，当前项目根目录 sonar.sources=./src # 排除目录 sonar.exclusions=**/test/**,**/target/** # jdk版本 sonar.java.source=1.17 sonar.java.target=1.17 # 字符编码 sonar.sourceEncoding=UTF-8 # binaries路径 sonar.java.binaries=target/classes5.2修改Jenkinsfile加入SonarQube代码审查阶段pipeline { agent any stages { stage('拉取代码') { steps { echo '开始拉取代码' checkout([$class: 'GitSCM', branches: [[name: '*/master']], userRemoteConfigs: [[url: 'https://gitee.com/axzys/sprint_boot_demo.git']]]) echo '拉取代码完成' } } stage('打包编译') { steps { echo '开始打包编译' sh 'mvn clean package' echo '打包编译完成' } } stage('代码审查') { steps { echo '开始代码审查' script { // 引入SonarQube scanner，名称与jenkins 全局工具SonarQube Scanner的name保持一致 def scannerHome = tool 'SonarQube' // 引入SonarQube Server，名称与jenkins 系统配置SonarQube servers的name保持一致 withSonarQubeEnv('SonarQube') { sh "${scannerHome}/bin/sonar-scanner" } } echo '代码审查完成' } } stage('部署项目') { steps { echo '开始部署项目' echo '部署项目完成' } } } } 5.3构建测试
-
buildctl和nerdctl 安装配置 一、安装与使用nerdctlcontainerd虽然可直接提供给终端用户直接使用，也提供了命令行工具(ctr)，但并不是很友好，所以nerdctl应运而生，它也是containerd的命令行工具，支持docker cli关于容器生命周期管理的所有命令，并且支持docker compose (nerdctl compose up)1.1安装nerdctl下载地址：https://github.com/containerd/nerdctl/releases# 下载 [root@k8s-master ~]# wget https://github.com/containerd/nerdctl/releases/download/v2.1.2/nerdctl-2.1.2-linux-amd64.tar.gz # 解压 [root@k8s-master ~]# tar -zxvf nerdctl-2.1.2-linux-amd64.tar.gz nerdctl containerd-rootless-setuptool.sh containerd-rootless.sh # 复制文件 [root@k8s-master ~]# mv nerdctl /usr/bin/ # 配置 nerdctl 参数自动补齐 [root@k8s-master ~]# echo 'source <(nerdctl completion bash)' >> /etc/profile [root@k8s-master ~]# source /etc/profile # 验证 [root@k8s-master ~]# nerdctl -v nerdctl version 2.1.21.2命名空间这个和K8s的名字空间不是一回事，其中default就是containerd的默认名字空间，http://k8s.io是K8s的名字空间root@k8s-03:~/bin# nerdctl ns ls NAME CONTAINERS IMAGES VOLUMES LABELS buildkit 0 0 0 buildkit_history 0 0 0 default 0 1 0 k8s.io 70 66 0 # 创建命名空间 [root@k8s-master ~]# nerdctl ns create test # 删除命名空间 [root@k8s-master ~]# nerdctl ns remove test test # 查看名称空间详情 [root@k8s-master ~]# nerdctl ns inspect k8s.io [ { "Name": "k8s.io", "Labels": null } ]1.3镜像root@k8s-03:~/bin# nerdctl -n k8s.io images REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku <none> b3e519ae85d0 4 hours ago linux/amd64 406.2MB 179.1MB <none> <none> b3e519ae85d0 4 hours ago linux/amd64 406.2MB 179.1MB registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku v1 b3e519ae85d0 4 hours ago linux/amd64 406.2MB 179.1MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-registryctl <none> a13c1fd0b23e 22 hours ago linux/amd64 163.5MB 67.74MB <none> <none> a13c1fd0b23e 22 hours ago linux/amd64 163.5MB 67.74MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-registryctl v2.13.0 a13c1fd0b23e 22 hours ago linux/amd64 163.5MB 67.74MB registry.cn-guangzhou.aliyuncs.com/xingcangku/redis-photon <none> cb5883e8415a 22 hours ago linux/amd64 171.5MB 61MB <none> <none> cb5883e8415a 22 hours ago linux/amd64 171.5MB 61MB registry.cn-guangzhou.aliyuncs.com/xingcangku/redis-photon v2.13.0 cb5883e8415a 22 hours ago linux/amd64 171.5MB 61MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-core <none> d75212166cdb 22 hours ago linux/amd64 202.4MB 63.85MB <none> <none> d75212166cdb 22 hours ago linux/amd64 202.4MB 63.85MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-core v2.13.0 d75212166cdb 22 hours ago linux/amd64 202.4MB 63.85MB registry.cn-guangzhou.aliyuncs.com/xingcangku/registry-photon <none> b9139a9005f9 22 hours ago linux/amd64 87.67MB 33.14MB <none> <none> b9139a9005f9 22 hours ago linux/amd64 87.67MB 33.14MB registry.cn-guangzhou.aliyuncs.com/xingcangku/registry-photon v2.13.0 b9139a9005f9 22 hours ago linux/amd64 87.67MB 33.14MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-portal <none> 19712b3eeee5 22 hours ago linux/amd64 165.2MB 53.6MB <none> <none> 19712b3eeee5 22 hours ago linux/amd64 165.2MB 53.6MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-portal v2.13.0 19712b3eeee5 22 hours ago linux/amd64 165.2MB 53.6MB registry.cn-guangzhou.aliyuncs.com/xingcangku/gitlab-gitlab-ce-16.11.1-ce.0 <none> decbed64a538 2 days ago linux/amd64 3.109GB 1.253GB <none> <none> decbed64a538 2 days ago linux/amd64 3.109GB 1.253GB registry.cn-guangzhou.aliyuncs.com/xingcangku/gitlab-gitlab-ce-16.11.1-ce.0 16.11.1-ce.0 decbed64a538 2 days ago linux/amd64 3.109GB 1.253GB registry.cn-guangzhou.aliyuncs.com/xingcangku/traefik <none> 39f367894114 2 days ago linux/amd64 225.8MB 58.3MB <none> <none> 39f367894114 2 days ago linux/amd64 225.8MB 58.3MB registry.cn-guangzhou.aliyuncs.com/xingcangku/traefik v3.0.0 39f367894114 2 days ago linux/amd64 225.8MB 58.3MB registry.cn-guangzhou.aliyuncs.com/xingcangku/kubernetesui-dashboard <none> e291095692ba 3 days ago linux/amd64 257.7MB 75.79MB <none> <none> e291095692ba 3 days ago linux/amd64 257.7MB 75.79MB registry.cn-guangzhou.aliyuncs.com/xingcangku/kubernetesui-dashboard v2.7.0 e291095692ba 3 days ago linux/amd64 257.7MB 75.79MB registry.cn-guangzhou.aliyuncs.com/xingcangku/kubernetesui-metrics-scraper <none> ca7729489386 3 days ago linux/amd64 43.82MB 19.74MB <none> <none> ca7729489386 3 days ago linux/amd64 43.82MB 19.74MB registry.cn-guangzhou.aliyuncs.com/xingcangku/kubernetesui-metrics-scraper v1.0.8 ca7729489386 3 days ago linux/amd64 43.82MB 19.74MB registry.cn-guangzhou.aliyuncs.com/xingcangku/bitnami-postgresql <none> 94485e7c7d1d 3 days ago linux/amd64 280.1MB 90.55MB <none> <none> 94485e7c7d1d 3 days ago linux/amd64 280.1MB 90.55MB registry.cn-guangzhou.aliyuncs.com/xingcangku/bitnami-postgresql 11.14.0-debian-10-r22 94485e7c7d1d 3 days ago linux/amd64 280.1MB 90.55MB registry.cn-guangzhou.aliyuncs.com/xingcangku/sonarqube-community <none> b5e625526868 3 days ago linux/amd64 1.24GB 957.4MB <none> <none> b5e625526868 3 days ago linux/amd64 1.24GB 957.4MB registry.cn-guangzhou.aliyuncs.com/xingcangku/sonarqube-community 25.5.0.107428-community b5e625526868 3 days ago linux/amd64 1.24GB 957.4MB registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17 <none> bb363b39bef3 3 days ago linux/amd64 483.5MB 271.9MB <none> <none> bb363b39bef3 3 days ago linux/amd64 483.5MB 271.9MB registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17 lts-jdk17 bb363b39bef3 3 days ago linux/amd64 483.5MB 271.9MB registry.cn-guangzhou.aliyuncs.com/xingcangku/trivy-adapter-photon <none> ad014f12e11c 3 days ago linux/amd64 383.3MB 126.1MB <none> <none> ad014f12e11c 3 days ago linux/amd64 383.3MB 126.1MB registry.cn-guangzhou.aliyuncs.com/xingcangku/trivy-adapter-photon v2.13.0 ad014f12e11c 3 days ago linux/amd64 383.3MB 126.1MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-db <none> dc08b59ada6d 3 days ago linux/amd64 285.4MB 108.1MB <none> <none> dc08b59ada6d 3 days ago linux/amd64 285.4MB 108.1MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-db v2.13.0 dc08b59ada6d 3 days ago linux/amd64 285.4MB 108.1MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-jobservice <none> 8ccc99b52f23 3 days ago linux/amd64 178.5MB 72.67MB <none> <none> 8ccc99b52f23 3 days ago linux/amd64 178.5MB 72.67MB registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-jobservice v2.13.0 8ccc99b52f23 3 days ago linux/amd64 178.5MB 72.67MB registry.cn-guangzhou.aliyuncs.com/xingcangku/nginx-photon <none> 87662c08516c 3 days ago linux/amd64 156.5MB 51.41MB <none> <none> 87662c08516c 3 days ago linux/amd64 156.5MB 51.41MB registry.cn-guangzhou.aliyuncs.com/xingcangku/nginx-photon v2.13.0 87662c08516c 3 days ago linux/amd64 156.5MB 51.41MB registry.cn-hangzhou.aliyuncs.com/google_containers/coredns <none> 90d3eeb2e210 3 days ago linux/amd64 53.61MB 16.19MB <none> <none> 90d3eeb2e210 3 days ago linux/amd64 53.61MB 16.19MB registry.cn-hangzhou.aliyuncs.com/google_containers/coredns v1.10.1 90d3eeb2e210 3 days ago linux/amd64 53.61MB 16.19MB registry.cn-guangzhou.aliyuncs.com/xingcangku/cccc <none> f3e2173b0e48 3 days ago linux/amd64 82.5MB 31.09MB <none> <none> f3e2173b0e48 3 days ago linux/amd64 82.5MB 31.09MB registry.cn-guangzhou.aliyuncs.com/xingcangku/cccc 0.25.5 f3e2173b0e48 3 days ago linux/amd64 82.5MB 31.09MB registry.cn-guangzhou.aliyuncs.com/xingcangku/ddd <none> 564119549dd9 3 days ago linux/amd64 10.73MB 4.755MB <none> <none> 564119549dd9 3 days ago linux/amd64 10.73MB 4.755MB registry.cn-guangzhou.aliyuncs.com/xingcangku/ddd 1.5.1 564119549dd9 3 days ago linux/amd64 10.73MB 4.755MB registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy <none> c1fd57dc0883 3 days ago linux/amd64 75.16MB 23.91MB <none> <none> c1fd57dc0883 3 days ago linux/amd64 75.16MB 23.91MB registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy v1.27.0 c1fd57dc0883 3 days ago linux/amd64 75.16MB 23.91MB registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee <none> 0d0658a57932 3 days ago linux/amd64 712.7kB 308.4kB <none> <none> 0d0658a57932 3 days ago linux/amd64 712.7kB 308.4kB registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee 3.8 0d0658a57932 3 days ago linux/amd64 712.7kB 308.4kB # 拉取镜像 [root@k8s-master ~]# nerdctl -n test pull nginx:alpine # 构建镜像 [root@k8s-master ~]# cat Dockerfile FROM debian RUN apt-get install -y --force-yes locales RUN echo "LC_ALL=\"zh_CN.UTF-8\"" >> /etc/default/locale RUN locale-gen "zh_CN.UTF-8" [root@k8s-master ~]# nerdctl -n test build -t abc.com/debian . # 上传镜像 [root@k8s-master ~]# nerdctl -n test push abc.com/debian # 导出镜像 [root@k8s-master ~]# nerdctl -n test save -o debian.tar abc.com/debian # 导入镜像 [root@k8s-master ~]# nerdctl -n test load -i debian.tar 1.4容器root@k8s-03:~/bin# nerdctl -n k8s.io ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 90fbd223a72f registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17:lts-jdk17 "/usr/bin/tini -- /u…" 9 hours ago Up k8s://cicd/jenkins-7d65887794-s4vhr/jenkins 681d3d0f9346 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 9 hours ago Up k8s://cicd/jenkins-7d65887794-s4vhr cfb418a6f445 registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-jobservice:v2.13.0 "/harbor/entrypoint.…" 11 hours ago Up k8s://harbor/harbor-jobservice-6c766cbf57-4t4rv/jobservice 566e8a6194f8 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-jobservice-6c766cbf57-4t4rv 2f44c7d4f045 registry.cn-guangzhou.aliyuncs.com/xingcangku/nginx-photon:v2.13.0 "nginx -g daemon off;" 11 hours ago Up k8s://harbor/harbor-nginx-6569fc6f48-n58m4/nginx 73298e3ed41f registry.cn-guangzhou.aliyuncs.com/xingcangku/bitnami-postgresql:11.14.0-debian-10-r22 "/opt/bitnami/script…" 11 hours ago Up k8s://sonarqube/my-sonarqube-postgresql-0/my-sonarqube-postgresql a3904a1442a7 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://sonarqube/my-sonarqube-postgresql-0 090451297d0e registry.cn-guangzhou.aliyuncs.com/xingcangku/kubernetesui-metrics-scraper:v1.0.8 "/metrics-sidecar" 11 hours ago Up k8s://kubernetes-dashboard/dashboard-metrics-scraper-f9669b96-gqv9b/dashboard-metrics-scraper 5e2e10ece736 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://kubernetes-dashboard/dashboard-metrics-scraper-f9669b96-gqv9b 5fe6f542a5af registry.cn-guangzhou.aliyuncs.com/xingcangku/kubernetesui-dashboard:v2.7.0 "/dashboard --insecu…" 11 hours ago Up k8s://kubernetes-dashboard/kubernetes-dashboard-5d8977b4cd-hn9wj/kubernetes-dashboard 5f5145461dba registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-portal:v2.13.0 "nginx -g daemon off;" 11 hours ago Up k8s://harbor/harbor-portal-7b67bff87d-hhbwf/portal 0c91dee84ef4 registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-db:v2.13.0 "/docker-entrypoint.…" 11 hours ago Up k8s://harbor/harbor-database-0/database 00f905053c35 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://kubernetes-dashboard/kubernetes-dashboard-5d8977b4cd-hn9wj bd3f1dd15a7a registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-portal-7b67bff87d-hhbwf abd9c09c0d84 registry.cn-guangzhou.aliyuncs.com/xingcangku/trivy-adapter-photon:v2.13.0 "/home/scanner/entry…" 11 hours ago Up k8s://harbor/harbor-trivy-0/trivy 398f6f60263a registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-trivy-0 13b06ef3a148 registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-registryctl:v2.13.0 "/home/harbor/start.…" 11 hours ago Up k8s://harbor/harbor-registry-84dc65db77-rq9qc/registryctl da585e4b08bd registry.cn-guangzhou.aliyuncs.com/xingcangku/registry-photon:v2.13.0 "/home/harbor/entryp…" 11 hours ago Up k8s://harbor/harbor-registry-84dc65db77-rq9qc/registry ece6da7ad469 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-nginx-6569fc6f48-n58m4 c0d6e45cec3c registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-registry-84dc65db77-rq9qc 6bcc595f312c registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 "/coredns -conf /etc…" 11 hours ago Up k8s://kube-system/coredns-65dcc469f7-xphsz/coredns 8dd7971b626b registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://kube-system/coredns-65dcc469f7-xphsz e1daa9e322e5 registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1 "/coredns -conf /etc…" 11 hours ago Up k8s://kube-system/coredns-65dcc469f7-fg85n/coredns 6a75b9f4e905 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://kube-system/coredns-65dcc469f7-fg85n bfd37ad46a64 registry.cn-guangzhou.aliyuncs.com/xingcangku/traefik:v3.0.0 "/entrypoint.sh --gl…" 11 hours ago Up k8s://traefik/traefik-release-589c7ff647-ch4cz/traefik-release baacee8d0a07 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://traefik/traefik-release-589c7ff647-ch4cz eca5589418a2 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-database-0 b543a7a9e25c registry.cn-guangzhou.aliyuncs.com/xingcangku/harbor-core:v2.13.0 "/harbor/entrypoint.…" 11 hours ago Up k8s://harbor/harbor-core-797d458f8c-2gcjf/core 558406c8bd5c registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-core-797d458f8c-2gcjf 1a14db05c528 registry.cn-guangzhou.aliyuncs.com/xingcangku/redis-photon:v2.13.0 "redis-server /etc/r…" 11 hours ago Up k8s://harbor/harbor-redis-0/redis 5c518e649a4a registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://harbor/harbor-redis-0 8f75c2fd204b registry.cn-guangzhou.aliyuncs.com/xingcangku/cccc:0.25.5 "/opt/bin/flanneld -…" 11 hours ago Up k8s://kube-flannel/kube-flannel-ds-zp4jv/kube-flannel ea7d12788ef6 registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.27.0 "/usr/local/bin/kube…" 11 hours ago Up k8s://kube-system/kube-proxy-vfcq8/kube-proxy 98f0354f74e5 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://kube-flannel/kube-flannel-ds-zp4jv a9c920f9e7c7 registry.cn-guangzhou.aliyuncs.com/xingcangku/eeeee:3.8 "/pause" 11 hours ago Up k8s://kube-system/kube-proxy-vfcq8 # 启动容器 [root@k8s-master ~]# nerdctl -n test run -d -p 80:80 --name web nginx:alpine # 进入容器 [root@k8s-master ~]# nerdctl -n test exec -it web sh / # # 停止容器 [root@k8s-master ~]# nerdctl -n test stop web web # 删除容器 [root@k8s-master ~]# nerdctl -n test rm web web 1.5其他操作# 查看网络信息 [root@k8s-master ~]# nerdctl network ls NETWORK ID NAME FILE cbr0 /etc/cni/net.d/10-flannel.conflist 17f29b073143 bridge /etc/cni/net.d/nerdctl-bridge.conflist host none # 查看系统信息 [root@k8s-master ~]# nerdctl system info Client: Namespace: default Debug Mode: false Server: Server Version: 1.6.4 Storage Driver: overlayfs Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Log: fluentd journald json-file syslog Storage: native overlayfs Security Options: seccomp Profile: default Kernel Version: 4.18.0-425.13.1.el8_7.x86_64 Operating System: Rocky Linux 8.7 (Green Obsidian) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.618GiB Name: k8s-master ID: d2b76909-9552-4be5-a12a-00b955f756f2 # 清理数据，它不是和Docker那样只是把标签为"none"的镜像清理掉，而是把所有没有"正在使用"的镜像清理了 [root@k8s-master ~]# nerdctl system prune -h二、nerdctl+buildkitd构建镜像 2.1buildkit介绍buildkit 从Docker公司的开源的镜像构建工具包，支持OCI标准的镜像构建 buildkitd组成部分： buildkitd（服务端），目前支持runc和containerd作为镜像构建环境，默认是runc，可以更换containerd。 buildctl（客户端），负责解析Dockerfile文件、并向服务端buildkitd发出构建请求。 构建镜像并推送至Harbor为例，整个服务调用过程如下：2.2安装buildkit软件包下载地址：https://github.com/moby/buildkit/releases[root@master ~]# wget https://github.com/moby/buildkit/releases/download/v0.13.2/buildkit-v0.13.2.linux-amd64.tar.gz [root@master ~]# tar -zxvf buildkit-v0.13.2.linux-amd64.tar.gz bin/ bin/buildctl bin/buildkit-cni-bridge bin/buildkit-cni-firewall bin/buildkit-cni-host-local bin/buildkit-cni-loopback bin/buildkit-qemu-aarch64 bin/buildkit-qemu-arm bin/buildkit-qemu-i386 bin/buildkit-qemu-mips64 bin/buildkit-qemu-mips64el bin/buildkit-qemu-ppc64le bin/buildkit-qemu-riscv64 bin/buildkit-qemu-s390x bin/buildkit-runc bin/buildkitd [root@master ~]# cd bin/ [root@master bin]# cp * /usr/local/bin/创建service脚本[root@master bin]# cat /etc/systemd/system/buildkitd.service [Unit] Description=BuildKit Documentation=https://github.com/moby/buildkit [Service] ExecStart=/usr/local/bin/buildkitd --oci-worker=false --containerd-worker=true [Install] WantedBy=multi-user.target新增buildkitd配置文件，添加镜像仓库使用http访问[root@master bin]# vim /etc/buildkit/buildkitd.toml [registry."harbor.local.com"] http = false insecure = true启动buildkitd[root@master bin]# systemctl daemon-reload [root@master bin]# systemctl start buildkitd [root@master bin]# systemctl enable buildkitd2.3构建镜像并测试[root@master ~]# cat Dockerfile FROM busybox CMD ["echo","hello","container"] [root@master ~]# nerdctl build -t busybox:v1 . [root@master ~]# nerdctl images REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE busybox v1 fb6a2dfc7899 About a minute ago linux/amd64 4.1 MiB 2.1 MiB [root@master ~]# nerdctl run busybox:v1 hello container2.4推送至Harbor仓库[root@master ~]# nerdctl tag busybox:v1 harbor.local.com/app/busybox:v1 [root@master ~]# nerdctl push harbor.local.com/app/busybox:v1此时查看Harbor仓库发现已经推送成功
-
jenkins与k8s连接 一、安装kubernetes插件在Jenkins的插件管理中安装Kubernetes插件 jenkins——>系统管理——>插件管理——>avaliable plugins二、本集群连接 2.1创建sa账号如果jenkins在k8s集群中部署，直接创建sa账号，并进行rbac授权即可,yaml文件参考前面文章。2.2创建cloud资源然后在jenkins——>系统管理——>Clouds——>New cloud——>输入cloud name并勾选类型为kubernetesroot@k8s-01:~/jenkins# cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: cicd spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins spec: securityContext: fsGroup: 1000 # 确保 Jenkins 用户有存储写入权限 serviceAccountName: jenkins-admin automountServiceAccountToken: true # 新增节点选择器，将 Pod 固定在 k8s-03 节点 nodeSelector: kubernetes.io/hostname: k8s-03 containers: - name: jenkins image: registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17:lts-jdk17 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 - containerPort: 50000 # 新增环境变量（可选） env: - name: JAVA_HOME value: "/usr/lib/jvm/amazon-corretto-17.0.16.8.1-linux-x64" volumeMounts: - name: jenkins-data mountPath: /var/jenkins_home # ============ 新增挂载 ============ - name: host-jvm mountPath: /usr/lib/jvm - name: maven-data mountPath: /usr/local/maven resources: limits: cpu: "1" memory: "3Gi" requests: cpu: "0.5" memory: "1Gi" livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 90 periodSeconds: 10 volumes: - name: jenkins-data persistentVolumeClaim: claimName: jenkins-pvc # ============ 新增的卷配置 ============ - name: host-jvm hostPath: path: /usr/lib/jvm type: Directory - name: maven-data hostPath: path: /usr/local/maven type: Directoryroot@k8s-01:~/jenkins# cat new-rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-admin namespace: cicd --- # 创建自定义 ClusterRole apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: jenkins-clusterrole rules: - apiGroups: [""] resources: ["pods", "pods/log", "services"] verbs: ["get", "list", "watch", "create", "update", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["*"] --- # 绑定 ServiceAccount 到 ClusterRole apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: jenkins-crb roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins-clusterrole subjects: - kind: ServiceAccount name: jenkins-admin namespace: cicdk8s1.24版本以后，行为变更：从 Kubernetes v1.24 开始，不再自动为 ServiceAccount 创建 Secret（令牌不再存储在 Secret 中） TokenRequest API：现在需要使用 TokenRequest API 获取令牌 root@k8s-01:~/jenkins# cat jenkins-secret.yaml apiVersion: v1 kind: Secret metadata: name: jenkins-admin-token namespace: cicd annotations: kubernetes.io/service-account.name: jenkins-admin type: kubernetes.io/service-account-token root@k8s-01:~/jenkins# kubectl apply -n cicd -f jenkins-secret.yaml secret/jenkins-admin-token created root@k8s-01:~/jenkins# kubectl describe sa jenkins-admin -n cicd Name: jenkins-admin Namespace: cicd Labels: <none> Annotations: <none> Image pull secrets: <none> Mountable secrets: <none> Tokens: jenkins-admin-token Events: <none> root@k8s-01:~/jenkins# kubectl get secret jenkins-admin-token -n cicd -o jsonpath='{.data.token}' | base64 -d eyJhbGciOiJSUzI1NiIsImtpZCI6IjJ0MTFFdDhfdnFBYkNuTnBSSXlyOFIzN1B0MW13cVVJNlFwZDV1VzR1WXcifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJjaWNkIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImplbmtpbnMtYWRtaW4tdG9rZW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiamVua2lucy1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjRkY2ViZjg1LTI0NzEtNGJjYi04Yzg5LWQ0MWI0NjAzN2RkZSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpjaWNkOmplbmtpbnMtYWRtaW4ifQ.qvBiSwdCVbfMDnP0ElhnMcn__q65HDqidWw2JDQbrn7zrgNX2jjXlDxhA8RQFQaHhrTPGrOuP60vzfz4WgvJxwJvwIHaqqAbK8r3t-eTBpXNKltY3GEFEqxyjVlTd8q0DLW0OWZHUVZJrWhYT00Xa1ZViJgwQ2X0ogpAphSlvR351ZEDmDDxwxk4WwioZpmU22_weFamlU0g9SQVW5kBYGw06Tq_dPNL7cB0CDdPy0mSYckquEtG4xh-EIddPs9cGd1_OGurjwEkwX2-HvlfCXfoFkgo42lrOCaouWE4I4e21OXUCg5erFONSWSPKhhmhiLsjikByweCE0qaA2ZIUw#还要另外创建serviceAccount root@k8s-01:~/jenkins# cat jenkins-secret.yaml rbac.yaml apiVersion: v1 kind: Secret metadata: name: jenkins-admin-token namespace: cicd annotations: kubernetes.io/service-account.name: jenkins-admin type: kubernetes.io/service-account-token # rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: jenkins-admin namespace: cicd --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins-admin-crb roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: edit # 授予命名空间内管理权限 subjects: - kind: ServiceAccount name: jenkins-admin namespace: cicd#Kubernetes 服务证书 key root@k8s-01:~/jenkins# kubectl get secret jenkins-admin-token -n cicd -o jsonpath='{.data.ca\.crt}' | base64 -d -----BEGIN CERTIFICATE----- MIIDBTCCAe2gAwIBAgIIUTF7Qh3Q+fowDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE AxMKa3ViZXJuZXRlczAeFw0yNTA3MzAxMjU3MDdaFw0zNTA3MjgxMzAyMDdaMBUx EzARBgNVBAMTCmt1YmVybmV0ZXMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK AoIBAQCv6lpzywPfjhKfl0HCNA/onj3MJyWI4z1JUwDlukI2j7ygm+kM2rWLpRYY wjNrvzq7R4ZD2SfiFbN/Lo/EV0zM0MXdcsI8hkR6txlKXjRkrC5crCihd9idk2UR Ov25k7bE3JpvG/zVswNMuliHx38gNsv1tfSv75lfCKyiyO3rrnj3LR9iYFvArgzY a1F5FXNjw0HVRVIPeH060i75G3YhKDVAQoZVdMoJfW7wwZDDHnh1/GB16a9Jws+r MjIQVD0YtxhQdg7WeV2nfCs0L8yXInKXwX6MGv+/HRPw140TihZwnC9gYFPisRJ5 EtUV+uq78w3Vj7Y2ANlTTQbN/F1bAgMBAAGjWTBXMA4GA1UdDwEB/wQEAwICpDAP BgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSvRW0owjCv5rHXwOgsyrI2QC0CxDAV BgNVHREEDjAMggprdWJlcm5ldGVzMA0GCSqGSIb3DQEBCwUAA4IBAQBL7BDL9wKu TTZ4Lx8aGAIjuEP1slw8FnDuFY2V2M8cu2wX1tlqYKJkIbwwc/dr59j6LhLgTZaP p02lQJXNRG6vR4zydgzXgsTMDXK1ANI6+7/jYgCwixGgBs/IafISCFkm1IkQ70Cw rgibN4MvybWewmucS21F62HQzO1xvrHGL8YVxvFvaUyhL9+7VhYeIX2MF3E2jLl5 kzT74Awv0O+4zBcULKrYQqNEErf5fpRLi+y9SaKA2e85DWexkhkIb6y1lZScLjRp q1lkgXqcriE653t78WrE1dSaCJY8QI94jYr4B2u7S1sbNJ9vPCDHQfFfl0DgRI/e K8+av0Qro/aA -----END CERTIFICATE-----点击kubernetes cloud details填写cloud详细信息 - Kubernetes地址：在集群内部暴露的k8s service名称https://kubernetes.default.svc - Kubernetes命名空间：jenkins sa所属的名称空间cicd - Jenkins地址：jenkins svc的名称:8080端口http://jenkins.cicd.svc:8080 使用刚刚获取的令牌在 Jenkins 中创建凭据： 凭据类型：Secret text Secret：粘贴上面的令牌值 ID：k8s-service-account-token 配置完成后点击连接测试，显示k8s集群版本，证明配置无误。三、跨集群连接在某些情况下，jenkins部署在k8s集群外，通过二进制或者docker方式部署，如果想要连接k8s集群实现资源自动创建。或者当前jenkins部署在k8s集群A中，需要通过jenkins实现集群B资源的自动创建发布，使用此方式连接。3.1配置思路jenkins要想连接并操作k8s集群，需要配置授权，请求k8s集群的kube apiserver的请求，可以和kubectl一样利用config文件用作请求的鉴权，默认在~/.kube/config下，也可以单独严格指定权限细节，生成一个jenkins专用的config文件。 在jenkins中能够识别的证书文件为PKCS#12 certificate，因此需要先将kubeconfig文件中的证书转换生成PKCS#12格式的pfx证书文件3.2生成证书我们可以使用yq命令行工具解析yaml，并提取相关的内容，然后通过base 64解码，最后生成文件 安装yq工具，仓库地址：https://github.com/mikefarah/yq [root@k8s-master ~]# wget https://github.com/mikefarah/yq/releases/download/v4.34.1/yq_linux_amd64.tar.gz [root@k8s-master ~]# tar -zxvf yq_linux_amd64.tar.gz [root@k8s-master ~]# mv yq_linux_amd64 /usr/bin/yq [root@k8s-master ~]# yq --version yq (https://github.com/mikefarah/yq/) version v4.34.1 [root@k8s-master ~]# mkdir -p /opt/jenkins-crt/certificate-authority-data——>base 64解码——>ca.crt client-certificate-data——>base 64解码——>client.crt client-key-data——>base 64解码——>client.key[root@k8s-master ~]# yq e '.clusters[0].cluster.certificate-authority-data' /root/.kube/config | base64 -d > /opt/jenkins-crt/ca.crt [root@k8s-master ~]# yq e '.users[0].user.client-certificate-data' /root/.kube/config | base64 -d > /opt/jenkins-crt/client.crt [root@k8s-master ~]# yq e '.users[0].user.client-key-data' /root/.kube/config | base64 -d > /opt/jenkins-crt/client.key [root@k8s-master ~]# cd /opt/jenkins-crt/ [root@k8s-master jenkins-crt]# ls -la 总用量 12 drwxr-xr-x 2 root root 56 6月 10 20:54 . drwxr-xr-x. 6 root root 65 6月 10 20:37 .. -rw-r--r-- 1 root root 1099 6月 10 20:53 ca.crt -rw-r--r-- 1 root root 1147 6月 10 20:53 client.crt -rw-r--r-- 1 root root 1675 6月 10 20:54 client.key3.3转换证书通过openssl进行证书格式的转换，生成Client P12认证文件cert.pfx，输入两次密码并牢记密码。[root@k8s-master jenkins-crt]# openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt Enter Export Password: Verifying - Enter Export Password: [root@k8s-master jenkins-crt]# ls -la 总用量 16 drwxr-xr-x 2 root root 72 6月 10 20:55 . drwxr-xr-x. 6 root root 65 6月 10 20:37 .. -rw-r--r-- 1 root root 1099 6月 10 20:53 ca.crt -rw------- 1 root root 3221 6月 10 20:55 cert.pfx -rw-r--r-- 1 root root 1147 6月 10 20:53 client.crt -rw-r--r-- 1 root root 1675 6月 10 20:54 client.key3.4导入证书打开jenkins的web界面，系统管理——>Credentials——>添加全局凭据 凭据的类型选择Certificate，证书上传刚才生成的cert.pfx证书文件，输入通过openssl生成证书文件时输入的密码3.5配置远程k8s集群地址jenkins——>系统管理——>Clouds——>New cloud——>输入cloud name并勾选类型为kubernetes 填写cloud详细信息 - Kubernetes地址：/root/.kube/config文件中cluster部分中server的内容 - Kubernetes命名空间：/root/.kube/config文件中cluster部分中name的内容 - Jenkins地址：jenkins服务的地址 - kubernetes服务证书key：ca.crt内容 - 凭据：选择刚刚创建的Certificate凭据配置完成后点击连接测试，显示k8s集群版本，证明配置无误。四、动态slave介绍 4.1为什么需要动态slave目前大多公司都采用 Jenkins 集群来搭建符合需求的 CI/CD 流程，然而传统的 Jenkins Slave 一主多从方式会存在一些痛点，比如： - 主 Master 发生单点故障时，整个流程都不可用了 - 每个 Slave 的配置环境不一样，来完成不同语言的编译打包等操作，但是这些差异化的配置导致管理起来非常不方便，维护起来也是比较费劲 - 资源分配不均衡，有的 Slave 要运行的 job 出现排队等待，而有的 Slave 处于空闲状态 - 资源有浪费，每台 Slave 可能是物理机或者虚拟机，当 Slave 处于空闲状态时，也不会完全释放掉资源。 正因为上面的这些种种痛点，我们渴望一种更高效更可靠的方式来完成这个 CI/CD 流程，而 Docker虚拟化容器技术能很好的解决这个痛点，又特别是在 Kubernetes 集群环境下面能够更好来解决上面的问题，下图是基于 Kubernetes 搭建 Jenkins 集群的简单示意图：从图上可以看到 Jenkins Master 和 Jenkins Slave 以 Pod 形式运行在 Kubernetes 集群的 Node 上，Master 运行在其中一个节点，并且将其配置数据存储到一个 Volume 上去，Slave 运行在各个节点上，并且它不是一直处于运行状态，它会按照需求动态的创建并自动删除。 这种方式的工作流程大致为：当 Jenkins Master 接受到 Build 请求时，会根据配置的 Label 动态创建一个运行在 Pod 中的 Jenkins Slave 并注册到 Master 上，当运行完 Job 后，这个 Slave 会被注销并且这个 Pod 也会自动删除，恢复到最初状态。4.2Jenkins Slave好处- 服务高可用，当 Jenkins Master 出现故障时，Kubernetes 会自动创建一个新的 Jenkins Master 容器，并且将 Volume 分配给新创建的容器，保证数据不丢失，从而达到集群服务高可用(这是k8s带来的资源控制器带来的优势) - 动态伸缩，合理使用资源，每次运行 Job 时，会自动创建一个 Jenkins Slave，Job 完成后，Slave 自动注销并删除容器，资源自动释放，而且 Kubernetes 会根据每个资源的使用情况，动态分配 Slave 到空闲的节点上创建，降低出现因某节点资源利用率高，还排队等待在该节点的情况。 - 扩展性好，当 Kubernetes 集群的资源严重不足而导致 Job 排队等待时，可以很容易的添加一个 Kubernetes Node 到集群中，从而实现扩展。五、动态slave配置 5.1制作slave镜像slave镜像应该包含以下功能： - 运行jenkins-agent服务 - 使用kubectl命令操作k8s集群 - 使用nerdctl工具管理container镜像 - 使用buildctl构建container镜像。#获取文件 root@k8s-01:~/jenkins/work# cp /usr/bin/kubectl . root@k8s-01:~/jenkins/work# cp /usr/bin/nerdctl . root@k8s-01:~/jenkins/work# cp /usr/local/bin/buildctl . root@k8s-01:~/jenkins/work# ls buildctl Dockerfile kubectl nerdctl#构建镜像 #在构建镜像过程中基于inbound-agent镜像，因为其中已经包含了jenkins-agent服务相关组件，再添加kubectl工具用于操作k8s，nerdctl和buildctl工具用于构建和管理container镜像。 root@k8s-01:~/jenkins/work# cat Dockerfile FROM jenkins/inbound-agent:latest-jdk17 USER root COPY kubectl /usr/bin/kubectl COPY nerdctl /usr/bin/nerdctl COPY buildctl /usr/bin/buildctl root@k8s-01:~/jenkins/work# docker build -t jenkins-agent:v1 . [+] Building 714.4s (9/9) FINISHED docker:default => [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 181B 0.0s => [internal] load metadata for docker.io/jenkins/inbound-agent:latest-jdk17 4.1s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [1/4] FROM docker.io/jenkins/inbound-agent:latest-jdk17@sha256:591ba0391e1dc47af64432198be00a9e457c74d215970c4f1af592a21 709.8s => => resolve docker.io/jenkins/inbound-agent:latest-jdk17@sha256:591ba0391e1dc47af64432198be00a9e457c74d215970c4f1af592a210a 0.0s => => sha256:a90156da31f7db1823e946282d3743d1f917b9622d122a1d45818ef43c6f5dc9 8.35kB / 8.35kB 0.0s => => sha256:59e22667830bf04fb35e15ed9c70023e9d121719bb87f0db7f3159ee7c7e0b8d 28.23MB / 28.23MB 327.6s => => sha256:9ac051fdbd99f7d8c9e496724860b8ae3373f24d6f8a54f1d9096526df425d3c 43.12MB / 43.12MB 581.2s => => sha256:85a4e35755fb1aa44b91602297dc8d9f10eb8ad3f32baab32094cebc0eda41a4 3.32kB / 3.32kB 0.8s => => sha256:591ba0391e1dc47af64432198be00a9e457c74d215970c4f1af592a210a6c37b 3.14kB / 3.14kB 0.0s => => sha256:7eddb97bde91ed86d7e82e7fa5f23370b33e87882aafdbcfb2782fdec95f6231 2.19kB / 2.19kB 0.0s => => sha256:cef14de45bb7cc343e593f80531764848fe724db9084ae4a3cabacc7a7e24083 1.24MB / 1.24MB 14.3s => => sha256:28f95146d6851ca39a2ce18612cb5e5b19845ef85e4bb95bfa3095193fdf5777 1.24MB / 1.24MB 28.5s => => sha256:68a03bb16ee6a2c356d2e354bab5ec566dd7ef1e4e6daee52c1f87aa9d0cd139 62.99MB / 62.99MB 709.1s => => extracting sha256:59e22667830bf04fb35e15ed9c70023e9d121719bb87f0db7f3159ee7c7e0b8d 2.1s => => sha256:1c55318e78a1d3f438c6ca3cf6532d365a86fabbf7e3ecd14140e455f1489991 161B / 161B 328.8s => => sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 329.5s => => sha256:392ad068a4827c711e9af3956bd5d3bcac7e8d66b3dd35bb0a9fae60de47f80d 2.37kB / 2.37kB 330.2s => => extracting sha256:85a4e35755fb1aa44b91602297dc8d9f10eb8ad3f32baab32094cebc0eda41a4 0.0s => => sha256:65375761a96d26587431452e982c657d479c41c41027a7f0acf37a6a21fd1112 180B / 180B 330.9s => => extracting sha256:9ac051fdbd99f7d8c9e496724860b8ae3373f24d6f8a54f1d9096526df425d3c 2.3s => => extracting sha256:cef14de45bb7cc343e593f80531764848fe724db9084ae4a3cabacc7a7e24083 0.0s => => extracting sha256:28f95146d6851ca39a2ce18612cb5e5b19845ef85e4bb95bfa3095193fdf5777 0.0s => => extracting sha256:68a03bb16ee6a2c356d2e354bab5ec566dd7ef1e4e6daee52c1f87aa9d0cd139 0.6s => => extracting sha256:1c55318e78a1d3f438c6ca3cf6532d365a86fabbf7e3ecd14140e455f1489991 0.0s => => extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 0.0s => => extracting sha256:392ad068a4827c711e9af3956bd5d3bcac7e8d66b3dd35bb0a9fae60de47f80d 0.0s => => extracting sha256:65375761a96d26587431452e982c657d479c41c41027a7f0acf37a6a21fd1112 0.0s => [internal] load build context 0.7s => => transferring context: 107.93MB 0.7s => [2/4] COPY kubectl /usr/bin/kubectl 0.2s => [3/4] COPY nerdctl /usr/bin/nerdctl 0.1s => [4/4] COPY buildctl /usr/bin/buildctl 0.1s => exporting to image 0.1s => => exporting layers 0.1s => => writing image sha256:ad852c7e884a5f9f6e87fcb6112fbe0c616b601a69ae5cf74ba09f2456d4e578 0.0s => => naming to docker.io/library/jenkins-agent:v1 0.0s root@k8s-01:~/jenkins# docker login --username=aliyun3891595718 registry.cn-guangzhou.aliyuncs.com Password: WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. Configure a credential helper to remove this warning. See https://docs.docker.com/go/credential-store/ Login Succeeded root@k8s-01:~/jenkins# docker push registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku:v1 The push refers to repository [registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-cangku] 2de7e3a095aa: Pushed 235bf2aaf23c: Pushed 9c18f60b53bc: Pushed ec380c7051cf: Pushed 0c3e60359c38: Pushed 5f70bf18a086: Pushed 433332898d75: Pushed b2c56f1e2ab9: Pushed 1411e4580b87: Pushed df5f57ff732b: Pushed 3f360379e3d1: Pushed f03c3dd333b7: Pushed 7cc7fe68eff6: Pushed v1: digest: sha256:b3e519ae85d0f05ff170778c8ffae494879397d9881ca8bc905bc889da82fc07 size: 30475.2创建kube-config资源为了能让slave容器中能够使用 kubectl 工具来访问我们的 Kubernetes 集群，需要将其添加为secret资源，并挂载到pod中。root@k8s-01:~/jenkins/work# kubectl create secret generic -n cicd kube-config --from-file=/root/.kube/config secret/kube-config created5.2.1测试创建一个自由风格的流水线 下面是配置截图正常的日志显示，会起一个pod来执行的命令结束以后就会销毁5.3节点开启buildkit服务(可选)container容器运行时仅能运行容器，如果需要在CICD阶段构建镜像，则需要在执行构建镜像的节点手动安装buildkit服务并启用，具体步骤可参考文档：https://axzys.cn/index.php/archives/536/ 也可以在slave pod中新增一个container，运行buildkit服务。5.4配置Pod Template(可选)配置 Pod Template，就是配置 Jenkins Slave 运行的 Pod 模板，命名空间我们同样是用cicd，Labels设置为jenkins-slave，对于后面执行 Job 的时候需要用到该值，容器名称填写jnlp，这样可以替换默认的agent容器。镜像使用的是刚刚我们制作的slave镜像，加入了 kubectl 等一些实用的工具。 运行命令和命令参数为空。另外需要注意我们这里需要在下面挂载三个目录 /run/containerd/containerd.sock：该文件是用于 Pod 中的容器能够共享宿主机的Container，用于管理container镜像。 /root/.kube：将之前创建的kube-config资源挂载到容器的/root/.kube目录下，这样能够在 Pod 的容器中能够使用 kubectl 工具来访问我们的 Kubernetes 集群，方便我们后面在 Slave Pod 部署 Kubernetes 应用 /run/buildkit：该文件是用于 Pod 中的容器能够共享buildkit进程，用于构建container镜像。同时指定Service Accoun为之前创建的jenkins-admin除了在页面配置pod Template外，我们也可以通过pipeline配置。六、测试Kubernetes 插件的配置工作完成了，接下来我们就来添加一个 Job 任务，看是否能够在 Slave Pod 中执行，任务执行完成后看 Pod 是否会被销毁。6.1自由流水线测试创建自由流水线任务，勾选限制项目的运行节点，标签表达式填写我们配置的 Slave Pod 中的 Label，这两个地方必须保持一致。然后往下拉，在 Build 区域选择Execute shellecho "Hello Kubernetes" echo "测试获取Kubernetes信息" kubectl get node echo "测试获取container信息" nerdctl ns ls echo "测试buildkitd构建镜像" echo "FROM registry.cn-guangzhou.aliyuncs.com/xingcangku/busybox-latest:latest" > Dockerfile echo 'CMD ["echo","hello","container"]' >> Dockerfile nerdctl build -t buildkitd-test:v1 . nerdctl images | grep buildkitd-test现在我们直接在页面点击做成的 Build now （立即构建）触发构建即可，然后观察 Kubernetes 集群中 Pod 的变化root@k8s-03:~/bin# kubectl get pods -n cicd -o wide -w NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES jenkins-7d65887794-s4vhr 1/1 Running 0 9h 10.244.2.104 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Pending 0 0s <none> <none> <none> <none> jenkins-agent-8x3mn 0/1 Pending 0 0s <none> k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 ContainerCreating 0 0s <none> k8s-03 <none> <none> jenkins-agent-8x3mn 1/1 Running 0 1s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 1/1 Terminating 0 13s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 13s <none> k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 14s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 14s 10.244.2.116 k8s-03 <none> <none> jenkins-agent-8x3mn 0/1 Terminating 0 14s 10.244.2.116 k8s-03 <none> <none>6.2pipeline-使用pod Template在流水线中指定pipeline脚本pipeline脚本如下：podTemplate(label: 'jenkins-slave', inheritFrom: 'jenkins-agent', cloud: 'k8s-local'){ node('jenkins-slave') { stage('测试获取Kubernetes信息') { sh 'kubectl get node' } stage('测试获取container信息') { sh 'nerdctl ns ls' } stage('测试buildkitd构建镜像'){ sh '''echo "FROM busybox" > Dockerfile echo \'CMD ["echo","hello","container"]\' >> Dockerfile nerdctl build -t buildkitd-test:v2 . nerdctl images | grep buildkitd-test''' } } }点击立即构建，查看控制台输出。6.3pipeline-自定义pod Template//创建一个Pod的模板，label为jenkins-agent podTemplate(label: 'jenkins-agent', cloud: 'k8s-local', containers: [ containerTemplate( name: 'jnlp', image: "harbor.local.com/cicd/jenkins-agent:v3", workingDir: '/home/jenkins/agent' ), containerTemplate( name: 'buildkitd', image: "harbor.local.com/cicd/buildkit:v0.13.2", privileged: true )], volumes:[ hostPathVolume(mountPath: '/run/containerd/containerd.sock', hostPath:'/run/containerd/containerd.sock'), secretVolume(mountPath: '/root/.kube/', secretName: 'kube-config', defaultMode: '420'), hostPathVolume(mountPath: '/run/buildkit',hostPath: '/run/buildkit') ] ) // 使用上文创建的pod模板 { node('jenkins-agent'){ stage('测试获取Kubernetes信息') { sh 'kubectl get node' } stage('测试获取container信息') { sh 'nerdctl ns ls' } stage('测试buildkitd构建镜像'){ sh '''echo "FROM busybox" > Dockerfile echo 'CMD ["echo","hello","container"]' >> Dockerfile nerdctl build -t buildkitd-test:v2 . nerdctl images | grep buildkitd-test''' } } }运行结果与上文一致。
-
jenkins根据tag构建 一、发布与回滚思路正常功能发布时，是基于master分支发布的，所以我在成功发布后，会将当时的master分支自动打上tag，当需要回滚时，则基于tag分支进行发布即可。二、安装配置Git Parameter 2.1安装插件要想出现tag模式的参数，需要安装git Parameter 插件，在Jenkins的Manage Jenkins→Plugins→Available Plugins 中安装2.2验证安装完成后在项目的配置页的This project is parameterized 中可以看到选项2.3仓库添加tag初始化仓库，添加tag并提交root@k8s-03:~/vue3_vite_element-plus# git config --global user.name "xing" root@k8s-03:~/vue3_vite_element-plus# git config --global user.email "7902731@qq.com" root@k8s-03:~/vue3_vite_element-plus# git config --global --list user.name=xing user.email=7902731@qq.com root@k8s-03:~/vue3_vite_element-plus# git tag -a v1.0 -m "1.0版本" root@k8s-03:~/vue3_vite_element-plus# git tag -l v1.0 root@k8s-03:~/vue3_vite_element-plus# git push origin v1.0 Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': Enumerating objects: 1, done. Counting objects: 100% (1/1), done. Writing objects: 100% (1/1), 160 bytes | 160.00 KiB/s, done. Total 1 (delta 0), reused 0 (delta 0), pack-reused 0 To http://192.168.30.181/xing/vue3_vite_element-plus.git * [new tag] v1.0 -> v1.0修改部分代码，并提交新版本。root@k8s-03:~/vue3_vite_element-plus# ls Dockerfile Jenkinsfile package.json README.md src vite.config.mjs index.html nginx.conf public screenshot test webstorm.config.js root@k8s-03:~/vue3_vite_element-plus# vi Dockerfile root@k8s-03:~/vue3_vite_element-plus# git commit -m "更新至v2" . [master 636d69b] 更新至v2 1 file changed, 2 insertions(+), 1 deletion(-) root@k8s-03:~/vue3_vite_element-plus# git tag -a v2.0 -m "2.0版本呢" root@k8s-03:~/vue3_vite_element-plus# git tag -l v1.0 v2.0 root@k8s-03:~/vue3_vite_element-plus# git push origin v2.0 Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': Enumerating objects: 6, done. Counting objects: 100% (6/6), done. Delta compression using up to 8 threads Compressing objects: 100% (4/4), done. Writing objects: 100% (4/4), 423 bytes | 423.00 KiB/s, done. Total 4 (delta 2), reused 0 (delta 0), pack-reused 0 To http://192.168.30.181/xing/vue3_vite_element-plus.git * [new tag] v2.0 -> v2.0 root@k8s-03:~/vue3_vite_element-plus# 查看gitlab tag信息，发现已经有v1.0，v2.0tag三、使用tag变量发布 3.1发布最新版本生成pipeline，指定分支为${tag} root@k8s-03:~/vue3_vite_element-plus# git ls-remote --tags origin Username for 'http://192.168.30.181': xing Password for 'http://xing@192.168.30.181': 3afadbd4e09f012ec0cb89e4cae667f8564ba5fa refs/tags/v1.0 a934816659500449a9413148e69d113faf5cccea refs/tags/v1.0^{} cf126d51958d9ed649b6b73e0f90929c6ac58694 refs/tags/v2.0 636d69bff7fb45b6732e977fdeee56c3628a2dcc refs/tags/v2.0^{}整体思路就是发布后就会有tar版本，如果有问题马上可以回滚到稳定的版本
