一、prometheus-operatorroot@k8s-01:/woke/prometheus/feishu# kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 7d23h alertmanager-main-1 2/2 Running 0 7d23h alertmanager-main-2 2/2 Running 0 7d23h blackbox-exporter-7fcbd888d-zv6z6 3/3 Running 0 16d feishu-forwarder-646d54f7cc-ks92j 1/1 Running 0 4m11s grafana-7ff454c477-l9x2k 1/1 Running 0 16d kube-state-metrics-78f95f79bb-wpcln 3/3 Running 0 16d node-exporter-622pm 2/2 Running 24 (36d ago) 40d node-exporter-mp2vg 2/2 Running 0 11h node-exporter-rl67z 2/2 Running 22 (36d ago) 40d prometheus-adapter-585d9c5dd5-bfsxw 1/1 Running 0 8d prometheus-adapter-585d9c5dd5-pcrnd 1/1 Running 0 8d prometheus-k8s-0 2/2 Running 0 7d23h prometheus-k8s-1 2/2 Running 0 7d23h prometheus-operator-78967669c9-5pk25 2/2 Running 0 7d23h prometheus-operator：控制器/管家 负责看你集群里的 CRD（ServiceMonitor/PodMonitor/PrometheusRule/Prometheus/Alertmanager/...），然后生成/维护真正跑起来的 Prometheus、Alertmanager 的 StatefulSet、配置 Secret 等。 prometheus-k8s-0/1：真正的 Prometheus 实例（采集+存储+算告警） prometheus-k8s-0/1：真正的 Prometheus 实例（采集+存储+算告警） 做三件事： 从各种 target 抓取 /metrics（scrape） 存到自己的 TSDB（本地时序库） 按规则（PrometheusRule）持续计算告警表达式，触发后发给 Alertmanager alertmanager-main-0/1/2：告警“中控”（聚合、路由、抑制、去重、静默） Prometheus 只负责“算出告警是否触发”，真正决定“发给谁、怎么合并、怎么抑制、多久发一次”是 Alertmanager。 grafana：出图/看板 Grafana 不采集数据，它是 去 Prometheus 查询（PromQL），然后画图。 node-exporter：采集 Node OS 指标（CPU/内存/磁盘/网卡） 暴露 /metrics 给 Prometheus 抓。 kube-state-metrics：采集 K8s 资源对象状态（Deployment 副本数、Pod 状态、Job 成功失败等） 也是暴露 /metrics 给 Prometheus 抓。 blackbox-exporter：探活（HTTP/TCP/ICMP 探测） Prometheus 调 blackbox-exporter 去探测目标，再把结果当指标存起来。 prometheus-adapter：把 Prometheus 指标转成 K8s HPA 可用的 custom/external metrics（与你问的告警链路不是一条线，但属于体系组件）1.1 Prometheus 怎么知道要抓哪些 target？这里就是 prometheus-operator 的关键：用 CRD 来描述“抓谁”。 常见 CRD： ServiceMonitor：按 label 选择某些 Service，然后抓它们的 endpoints PodMonitor：直接按 label 选择 Pod 抓 Probe：给 blackbox-exporter 用的探测对象（有些栈里会用） 简化理解就是： ServiceMonitor / PodMonitor = “抓取目标清单 + 抓取方式” Prometheus 会通过 Kubernetes API 做服务发现，然后按这些 CRD 去抓。1.2 阈值（告警规则）在哪里配置？怎么“算出来触发”？阈值本质是 PromQL 表达式 + 持续时间。 在 operator 体系里，告警规则通常放在 CRD：PrometheusRule 里。 PrometheusRule 里面一般长这样（示意）： expr: 告警表达式（PromQL） for: 持续多久才算真的触发（防抖） labels: 给告警打标签（比如 severity） annotations: summary/description InfoInhibitor 就是 kube-prometheus 里常见的一条规则：它本身是“用来抑制 info 告警”的辅助告警（触发时提示：现在有 info 告警在 firing）。1.3 Grafana 是怎么“出图”的？Grafana 的数据来源一般配置成一个 Prometheus DataSource（指向 http://prometheus-k8s.monitoring.svc:9090 之类）。 然后每个 Dashboard 的每个 Panel 都是一条或多条 PromQL 查询，例如： CPU 使用率图：rate(node_cpu_seconds_total{mode!="idle"}[5m]) Pod 重启次数：increase(kube_pod_container_status_restarts_total[1h]) 业务 QPS：sum(rate(http_requests_total[1m])) by (service) Grafana 不存数据、不采集数据，只负责向 Prometheus 查询并可视化。1.4 Alertmanager 是怎么收到 Prometheus 的告警？又怎么发给 feishu-forwarder？Prometheus → Alertmanager Prometheus 在运行时会做两件事： 每隔一小段时间计算 PrometheusRule 的 alert 规则 规则满足（pending/firing）后，把告警实例通过 HTTP 发给 Alertmanager（Prometheus 配置里有 alerting.alertmanagers 指向 alertmanager-main） 可以理解为：Prometheus 负责“算出告警”，然后把告警事件 POST 给 Alertmanager。Alertmanager 内部做什么？ 收到告警后，Alertmanager 会： group：按 group_by 合并同类告警（所以你会看到一次来 “(9)” 条） dedup：同一告警别重复刷 inhibit：抑制规则（比如有 warning/critical 时，把 info 抑制掉） route：按 matchers 决定走哪个 receiver（发给哪个渠道）Alertmanager → feishu-forwarder 当路由命中你的 webhook receiver 时，Alertmanager 会对你的 forwarder 发 HTTP POST（就是你代码里 /alertmanager 这个 handler 接收的 AMPayload）。 典型就是这样一个 receiver（示意）： receiver: feishu-forwarder webhook_configs: url: http://feishu-forwarder.monitoring.svc:8080/alertmanager 飞书通知里看到的 Receiver: monitoring/feishu-forwarder/feishu-forwarder 也是这个链路来的。feishu-forwarder → 飞书 forwarder 做的事就是： 解析 Alertmanager webhook 的 JSON（AMPayload） 根据 severity 路由到不同飞书群 webhook 构建飞书消息（text/card/你现在要的折叠面板 card v2） POST 到飞书机器人 webhook（带签名） 最终：飞书群里就收到消息。1.5 整条链路采集链路： Exporter(/metrics) → Prometheus scrape → TSDB 看板链路： Grafana → Prometheus PromQL 查询 → 图表 告警链路： PrometheusRule(阈值/表达式) → Prometheus 计算触发 → POST Alertmanager → Alertmanager(聚合/路由/抑制) → webhook POST feishu-forwarder → feishu-forwarder 转飞书卡片 → 飞书机器人 webhook → 群通知1.6 验证每一段链路Prometheus 抓取目标是否正常： kubectl -n monitoring port-forward pod/prometheus-k8s-0 9090:9090 # 浏览器打开 http://127.0.0.1:9090/targets告警规则是否存在、当前是否 firing： # Prometheus UI: # http://127.0.0.1:9090/rules # http://127.0.0.1:9090/alertsAlertmanager 是否收到告警、路由到哪个 receiver： kubectl -n monitoring port-forward pod/alertmanager-main-0 9093:9093 # 打开 http://127.0.0.1:9093feishu-forwarder 是否收到 webhook： kubectl logs -n monitoring deploy/feishu-forwarder -f二、监控指标详解#列出所有规则对象（所有命名空间） root@k8s-01:/woke/prometheus/feishu# kubectl get prometheusrules -A NAMESPACE NAME AGE monitoring alertmanager-main-rules 7d18h monitoring grafana-rules 7d18h monitoring kube-prometheus-rules 7d18h monitoring kube-state-metrics-rules 7d18h monitoring kubernetes-monitoring-rules 7d18h monitoring node-exporter-rules 7d18h monitoring prometheus-k8s-prometheus-rules 7d18h monitoring prometheus-operator-rules 7d18h node-exporter-rules 来源指标：job="node-exporter"（node_exporter 导出的主机/节点指标） 覆盖内容：CPU、内存、磁盘、inode、网络、系统负载、文件系统只读、时钟偏移等 常见指标前缀：node_cpu_*, node_memory_*, node_filesystem_*, node_network_*, node_load* kube-state-metrics-rules 来源指标：job="kube-state-metrics"（把 K8s 资源状态转成指标） 覆盖内容：Deployment/StatefulSet/DaemonSet 副本不一致、Pod CrashLoopBackOff、Job 失败、HPA 异常、Node 条件（NotReady/Pressure）等 常见指标前缀：kube_pod_*, kube_deployment_*, kube_statefulset_*, kube_daemonset_*, kube_node_*, kube_job_* kubernetes-monitoring-rules 来源指标：Kubernetes 控制面/节点组件（不同集群会略有差异） 典型 job：kube-apiserver、kubelet、coredns、kube-controller-manager、kube-scheduler 等 覆盖内容：API Server 错误率/延迟、kubelet/cadvisor 异常、CoreDNS 错误、控制面组件不可达等 常见指标：apiserver_*, kubelet_*, coredns_*, scheduler_*, workqueue_* 等 kube-prometheus-rules 这是 kube-prometheus 自带的“通用规则包”，通常偏 整套监控栈/集群通用： 一些通用 recording 规则（把原始指标聚合成更好用的序列） 一些跨组件的通用告警（规则名随版本会变） 来源可能混合：node-exporter、kube-state-metrics、apiserver、kubelet 等都会用到 你可以把它理解成“这套监控方案自带的公共配方”，不专属于某一个 exporter。 prometheus-k8s-prometheus-rules 来源指标：job="prometheus-k8s"（Prometheus 自己的 /metrics） 覆盖内容：抓取失败、规则评估失败、远端写入失败、TSDB 问题、磁盘即将满、样本摄入异常、告警发送到 Alertmanager 失败等 常见指标前缀：prometheus_*, prometheus_tsdb_*, prometheus_rule_* prometheus-operator-rules 来源指标：job="prometheus-operator" 覆盖内容：operator reconcile 失败/错误率、资源同步问题、配置 reload 问题等 常见指标前缀：prometheus_operator_* alertmanager-main-rules 来源指标：job="alertmanager-main"（Alertmanager 自己的 /metrics） 覆盖内容：通知发送失败、集群 peer 同步问题、silence/通知队列相关异常等 常见指标前缀：alertmanager_* grafana-rules 来源指标：job="grafana"（Grafana /metrics，若开启） 覆盖内容：Grafana 自身可用性/HTTP 错误、（有些环境也会加数据源连接类告警） 常见指标前缀：grafana_*#看详细监控数据 kubectl get prometheusrule node-exporter-rules -n monitoring -o yaml apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"monitoring.coreos.com/v1","kind":"PrometheusRule","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"exporter","app.kubernetes.io/name":"node-exporter","app.kubernetes.io/part-of":"kube-prometheus","app.kubernetes.io/version":"1.8.2","prometheus":"k8s","role":"alert-rules"},"name":"node-exporter-rules","namespace":"monitoring"},"spec":{"groups":[{"name":"node-exporter","rules":[{"alert":"NodeFilesystemSpaceFillingUp","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available space left and is filling up.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemspacefillingup","summary":"Filesystem is predicted to run out of space within the next 24 hours."},"expr":"(\n node_filesystem_avail_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 15\nand\n predict_linear(node_filesystem_avail_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"}[6h], 24*60*60) \u003c 0\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"1h","labels":{"severity":"warning"}},{"alert":"NodeFilesystemSpaceFillingUp","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available space left and is filling up fast.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemspacefillingup","summary":"Filesystem is predicted to run out of space within the next 4 hours."},"expr":"(\n node_filesystem_avail_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 10\nand\n predict_linear(node_filesystem_avail_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"}[6h], 4*60*60) \u003c 0\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"1h","labels":{"severity":"critical"}},{"alert":"NodeFilesystemAlmostOutOfSpace","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available space left.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutofspace","summary":"Filesystem has less than 5% space left."},"expr":"(\n node_filesystem_avail_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 5\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"30m","labels":{"severity":"warning"}},{"alert":"NodeFilesystemAlmostOutOfSpace","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available space left.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutofspace","summary":"Filesystem has less than 3% space left."},"expr":"(\n node_filesystem_avail_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_size_bytes{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 3\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"30m","labels":{"severity":"critical"}},{"alert":"NodeFilesystemFilesFillingUp","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available inodes left and is filling up.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemfilesfillingup","summary":"Filesystem is predicted to run out of inodes within the next 24 hours."},"expr":"(\n node_filesystem_files_free{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_files{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 40\nand\n predict_linear(node_filesystem_files_free{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"}[6h], 24*60*60) \u003c 0\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"1h","labels":{"severity":"warning"}},{"alert":"NodeFilesystemFilesFillingUp","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available inodes left and is filling up fast.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemfilesfillingup","summary":"Filesystem is predicted to run out of inodes within the next 4 hours."},"expr":"(\n node_filesystem_files_free{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_files{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 20\nand\n predict_linear(node_filesystem_files_free{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"}[6h], 4*60*60) \u003c 0\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"1h","labels":{"severity":"critical"}},{"alert":"NodeFilesystemAlmostOutOfFiles","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available inodes left.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutoffiles","summary":"Filesystem has less than 5% inodes left."},"expr":"(\n node_filesystem_files_free{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_files{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 5\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"1h","labels":{"severity":"warning"}},{"alert":"NodeFilesystemAlmostOutOfFiles","annotations":{"description":"Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf \"%.2f\" $value }}% available inodes left.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutoffiles","summary":"Filesystem has less than 3% inodes left."},"expr":"(\n node_filesystem_files_free{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} / node_filesystem_files{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} * 100 \u003c 3\nand\n node_filesystem_readonly{job=\"node-exporter\",fstype!=\"\",mountpoint!=\"\"} == 0\n)\n","for":"1h","labels":{"severity":"critical"}},{"alert":"NodeNetworkReceiveErrs","annotations":{"description":"{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \"%.0f\" $value }} receive errors in the last two minutes.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodenetworkreceiveerrs","summary":"Network interface is reporting many receive errors."},"expr":"rate(node_network_receive_errs_total{job=\"node-exporter\"}[2m]) / rate(node_network_receive_packets_total{job=\"node-exporter\"}[2m]) \u003e 0.01\n","for":"1h","labels":{"severity":"warning"}},{"alert":"NodeNetworkTransmitErrs","annotations":{"description":"{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf \"%.0f\" $value }} transmit errors in the last two minutes.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodenetworktransmiterrs","summary":"Network interface is reporting many transmit errors."},"expr":"rate(node_network_transmit_errs_total{job=\"node-exporter\"}[2m]) / rate(node_network_transmit_packets_total{job=\"node-exporter\"}[2m]) \u003e 0.01\n","for":"1h","labels":{"severity":"warning"}},{"alert":"NodeHighNumberConntrackEntriesUsed","annotations":{"description":"{{ $value | humanizePercentage }} of conntrack entries are used.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused","summary":"Number of conntrack are getting close to the limit."},"expr":"(node_nf_conntrack_entries{job=\"node-exporter\"} / node_nf_conntrack_entries_limit) \u003e 0.75\n","labels":{"severity":"warning"}},{"alert":"NodeTextFileCollectorScrapeError","annotations":{"description":"Node Exporter text file collector on {{ $labels.instance }} failed to scrape.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodetextfilecollectorscrapeerror","summary":"Node Exporter text file collector failed to scrape."},"expr":"node_textfile_scrape_error{job=\"node-exporter\"} == 1\n","labels":{"severity":"warning"}},{"alert":"NodeClockSkewDetected","annotations":{"description":"Clock at {{ $labels.instance }} is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodeclockskewdetected","summary":"Clock skew detected."},"expr":"(\n node_timex_offset_seconds{job=\"node-exporter\"} \u003e 0.05\nand\n deriv(node_timex_offset_seconds{job=\"node-exporter\"}[5m]) \u003e= 0\n)\nor\n(\n node_timex_offset_seconds{job=\"node-exporter\"} \u003c -0.05\nand\n deriv(node_timex_offset_seconds{job=\"node-exporter\"}[5m]) \u003c= 0\n)\n","for":"10m","labels":{"severity":"warning"}},{"alert":"NodeClockNotSynchronising","annotations":{"description":"Clock at {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodeclocknotsynchronising","summary":"Clock not synchronising."},"expr":"min_over_time(node_timex_sync_status{job=\"node-exporter\"}[5m]) == 0\nand\nnode_timex_maxerror_seconds{job=\"node-exporter\"} \u003e= 16\n","for":"10m","labels":{"severity":"warning"}},{"alert":"NodeRAIDDegraded","annotations":{"description":"RAID array '{{ $labels.device }}' at {{ $labels.instance }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/noderaiddegraded","summary":"RAID Array is degraded."},"expr":"node_md_disks_required{job=\"node-exporter\",device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"} - ignoring (state) (node_md_disks{state=\"active\",job=\"node-exporter\",device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}) \u003e 0\n","for":"15m","labels":{"severity":"critical"}},{"alert":"NodeRAIDDiskFailure","annotations":{"description":"At least one device in RAID array at {{ $labels.instance }} failed. Array '{{ $labels.device }}' needs attention and possibly a disk swap.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/noderaiddiskfailure","summary":"Failed device in RAID array."},"expr":"node_md_disks{state=\"failed\",job=\"node-exporter\",device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"} \u003e 0\n","labels":{"severity":"warning"}},{"alert":"NodeFileDescriptorLimit","annotations":{"description":"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \"%.2f\" $value }}%.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefiledescriptorlimit","summary":"Kernel is predicted to exhaust file descriptors limit soon."},"expr":"(\n node_filefd_allocated{job=\"node-exporter\"} * 100 / node_filefd_maximum{job=\"node-exporter\"} \u003e 70\n)\n","for":"15m","labels":{"severity":"warning"}},{"alert":"NodeFileDescriptorLimit","annotations":{"description":"File descriptors limit at {{ $labels.instance }} is currently at {{ printf \"%.2f\" $value }}%.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodefiledescriptorlimit","summary":"Kernel is predicted to exhaust file descriptors limit soon."},"expr":"(\n node_filefd_allocated{job=\"node-exporter\"} * 100 / node_filefd_maximum{job=\"node-exporter\"} \u003e 90\n)\n","for":"15m","labels":{"severity":"critical"}},{"alert":"NodeCPUHighUsage","annotations":{"description":"CPU usage at {{ $labels.instance }} has been above 90% for the last 15 minutes, is currently at {{ printf \"%.2f\" $value }}%.\n","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodecpuhighusage","summary":"High CPU usage."},"expr":"sum without(mode) (avg without (cpu) (rate(node_cpu_seconds_total{job=\"node-exporter\", mode!=\"idle\"}[2m]))) * 100 \u003e 90\n","for":"15m","labels":{"severity":"info"}},{"alert":"NodeSystemSaturation","annotations":{"description":"System load per core at {{ $labels.instance }} has been above 2 for the last 15 minutes, is currently at {{ printf \"%.2f\" $value }}.\nThis might indicate this instance resources saturation and can cause it becoming unresponsive.\n","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodesystemsaturation","summary":"System saturated, load per core is very high."},"expr":"node_load1{job=\"node-exporter\"}\n/ count without (cpu, mode) (node_cpu_seconds_total{job=\"node-exporter\", mode=\"idle\"}) \u003e 2\n","for":"15m","labels":{"severity":"warning"}},{"alert":"NodeMemoryMajorPagesFaults","annotations":{"description":"Memory major pages are occurring at very high rate at {{ $labels.instance }}, 500 major page faults per second for the last 15 minutes, is currently at {{ printf \"%.2f\" $value }}.\nPlease check that there is enough memory available at this instance.\n","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodememorymajorpagesfaults","summary":"Memory major page faults are occurring at very high rate."},"expr":"rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[5m]) \u003e 500\n","for":"15m","labels":{"severity":"warning"}},{"alert":"NodeMemoryHighUtilization","annotations":{"description":"Memory is filling up at {{ $labels.instance }}, has been above 90% for the last 15 minutes, is currently at {{ printf \"%.2f\" $value }}%.\n","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodememoryhighutilization","summary":"Host is running out of memory."},"expr":"100 - (node_memory_MemAvailable_bytes{job=\"node-exporter\"} / node_memory_MemTotal_bytes{job=\"node-exporter\"} * 100) \u003e 90\n","for":"15m","labels":{"severity":"warning"}},{"alert":"NodeDiskIOSaturation","annotations":{"description":"Disk IO queue (aqu-sq) is high on {{ $labels.device }} at {{ $labels.instance }}, has been above 10 for the last 30 minutes, is currently at {{ printf \"%.2f\" $value }}.\nThis symptom might indicate disk saturation.\n","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodediskiosaturation","summary":"Disk IO queue is high."},"expr":"rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}[5m]) \u003e 10\n","for":"30m","labels":{"severity":"warning"}},{"alert":"NodeSystemdServiceFailed","annotations":{"description":"Systemd service {{ $labels.name }} has entered failed state at {{ $labels.instance }}","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodesystemdservicefailed","summary":"Systemd service has entered failed state."},"expr":"node_systemd_unit_state{job=\"node-exporter\", state=\"failed\"} == 1\n","for":"5m","labels":{"severity":"warning"}},{"alert":"NodeBondingDegraded","annotations":{"description":"Bonding interface {{ $labels.master }} on {{ $labels.instance }} is in degraded state due to one or more slave failures.","runbook_url":"https://runbooks.prometheus-operator.dev/runbooks/node/nodebondingdegraded","summary":"Bonding interface is degraded"},"expr":"(node_bonding_slaves - node_bonding_active) != 0\n","for":"5m","labels":{"severity":"warning"}}]},{"name":"node-exporter.rules","rules":[{"expr":"count without (cpu, mode) (\n node_cpu_seconds_total{job=\"node-exporter\",mode=\"idle\"}\n)\n","record":"instance:node_num_cpu:sum"},{"expr":"1 - avg without (cpu) (\n sum without (mode) (rate(node_cpu_seconds_total{job=\"node-exporter\", mode=~\"idle|iowait|steal\"}[5m]))\n)\n","record":"instance:node_cpu_utilisation:rate5m"},{"expr":"(\n node_load1{job=\"node-exporter\"}\n/\n instance:node_num_cpu:sum{job=\"node-exporter\"}\n)\n","record":"instance:node_load1_per_cpu:ratio"},{"expr":"1 - (\n (\n node_memory_MemAvailable_bytes{job=\"node-exporter\"}\n or\n (\n node_memory_Buffers_bytes{job=\"node-exporter\"}\n +\n node_memory_Cached_bytes{job=\"node-exporter\"}\n +\n node_memory_MemFree_bytes{job=\"node-exporter\"}\n +\n node_memory_Slab_bytes{job=\"node-exporter\"}\n )\n )\n/\n node_memory_MemTotal_bytes{job=\"node-exporter\"}\n)\n","record":"instance:node_memory_utilisation:ratio"},{"expr":"rate(node_vmstat_pgmajfault{job=\"node-exporter\"}[5m])\n","record":"instance:node_vmstat_pgmajfault:rate5m"},{"expr":"rate(node_disk_io_time_seconds_total{job=\"node-exporter\", device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}[5m])\n","record":"instance_device:node_disk_io_time_seconds:rate5m"},{"expr":"rate(node_disk_io_time_weighted_seconds_total{job=\"node-exporter\", device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}[5m])\n","record":"instance_device:node_disk_io_time_weighted_seconds:rate5m"},{"expr":"sum without (device) (\n rate(node_network_receive_bytes_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n","record":"instance:node_network_receive_bytes_excluding_lo:rate5m"},{"expr":"sum without (device) (\n rate(node_network_transmit_bytes_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n","record":"instance:node_network_transmit_bytes_excluding_lo:rate5m"},{"expr":"sum without (device) (\n rate(node_network_receive_drop_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n","record":"instance:node_network_receive_drop_excluding_lo:rate5m"},{"expr":"sum without (device) (\n rate(node_network_transmit_drop_total{job=\"node-exporter\", device!=\"lo\"}[5m])\n)\n","record":"instance:node_network_transmit_drop_excluding_lo:rate5m"}]}]}} creationTimestamp: "2026-01-29T13:10:34Z" generation: 1 labels: app.kubernetes.io/component: exporter app.kubernetes.io/name: node-exporter app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 1.8.2 prometheus: k8s role: alert-rules name: node-exporter-rules namespace: monitoring resourceVersion: "17509681" uid: 8f17f249-40fd-4bef-839b-d9389947b19d spec: groups: - name: node-exporter rules: - alert: NodeFilesystemSpaceFillingUp annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left and is filling up. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemspacefillingup summary: Filesystem is predicted to run out of space within the next 24 hours. expr: | ( node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 15 and predict_linear(node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""}[6h], 24*60*60) < 0 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 1h labels: severity: warning - alert: NodeFilesystemSpaceFillingUp annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left and is filling up fast. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemspacefillingup summary: Filesystem is predicted to run out of space within the next 4 hours. expr: | ( node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 10 and predict_linear(node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""}[6h], 4*60*60) < 0 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 1h labels: severity: critical - alert: NodeFilesystemAlmostOutOfSpace annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutofspace summary: Filesystem has less than 5% space left. expr: | ( node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 5 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 30m labels: severity: warning - alert: NodeFilesystemAlmostOutOfSpace annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available space left. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutofspace summary: Filesystem has less than 3% space left. expr: | ( node_filesystem_avail_bytes{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_size_bytes{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 3 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 30m labels: severity: critical - alert: NodeFilesystemFilesFillingUp annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left and is filling up. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemfilesfillingup summary: Filesystem is predicted to run out of inodes within the next 24 hours. expr: | ( node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 40 and predict_linear(node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""}[6h], 24*60*60) < 0 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 1h labels: severity: warning - alert: NodeFilesystemFilesFillingUp annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left and is filling up fast. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemfilesfillingup summary: Filesystem is predicted to run out of inodes within the next 4 hours. expr: | ( node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 20 and predict_linear(node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""}[6h], 4*60*60) < 0 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 1h labels: severity: critical - alert: NodeFilesystemAlmostOutOfFiles annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutoffiles summary: Filesystem has less than 5% inodes left. expr: | ( node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 5 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 1h labels: severity: warning - alert: NodeFilesystemAlmostOutOfFiles annotations: description: Filesystem on {{ $labels.device }}, mounted on {{ $labels.mountpoint }}, at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefilesystemalmostoutoffiles summary: Filesystem has less than 3% inodes left. expr: | ( node_filesystem_files_free{job="node-exporter",fstype!="",mountpoint!=""} / node_filesystem_files{job="node-exporter",fstype!="",mountpoint!=""} * 100 < 3 and node_filesystem_readonly{job="node-exporter",fstype!="",mountpoint!=""} == 0 ) for: 1h labels: severity: critical - alert: NodeNetworkReceiveErrs annotations: description: '{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} receive errors in the last two minutes.' runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodenetworkreceiveerrs summary: Network interface is reporting many receive errors. expr: | rate(node_network_receive_errs_total{job="node-exporter"}[2m]) / rate(node_network_receive_packets_total{job="node-exporter"}[2m]) > 0.01 for: 1h labels: severity: warning - alert: NodeNetworkTransmitErrs annotations: description: '{{ $labels.instance }} interface {{ $labels.device }} has encountered {{ printf "%.0f" $value }} transmit errors in the last two minutes.' runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodenetworktransmiterrs summary: Network interface is reporting many transmit errors. expr: | rate(node_network_transmit_errs_total{job="node-exporter"}[2m]) / rate(node_network_transmit_packets_total{job="node-exporter"}[2m]) > 0.01 for: 1h labels: severity: warning - alert: NodeHighNumberConntrackEntriesUsed annotations: description: '{{ $value | humanizePercentage }} of conntrack entries are used.' runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodehighnumberconntrackentriesused summary: Number of conntrack are getting close to the limit. expr: | (node_nf_conntrack_entries{job="node-exporter"} / node_nf_conntrack_entries_limit) > 0.75 labels: severity: warning - alert: NodeTextFileCollectorScrapeError annotations: description: Node Exporter text file collector on {{ $labels.instance }} failed to scrape. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodetextfilecollectorscrapeerror summary: Node Exporter text file collector failed to scrape. expr: | node_textfile_scrape_error{job="node-exporter"} == 1 labels: severity: warning - alert: NodeClockSkewDetected annotations: description: Clock at {{ $labels.instance }} is out of sync by more than 0.05s. Ensure NTP is configured correctly on this host. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodeclockskewdetected summary: Clock skew detected. expr: | ( node_timex_offset_seconds{job="node-exporter"} > 0.05 and deriv(node_timex_offset_seconds{job="node-exporter"}[5m]) >= 0 ) or ( node_timex_offset_seconds{job="node-exporter"} < -0.05 and deriv(node_timex_offset_seconds{job="node-exporter"}[5m]) <= 0 ) for: 10m labels: severity: warning - alert: NodeClockNotSynchronising annotations: description: Clock at {{ $labels.instance }} is not synchronising. Ensure NTP is configured on this host. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodeclocknotsynchronising summary: Clock not synchronising. expr: | min_over_time(node_timex_sync_status{job="node-exporter"}[5m]) == 0 and node_timex_maxerror_seconds{job="node-exporter"} >= 16 for: 10m labels: severity: warning - alert: NodeRAIDDegraded annotations: description: RAID array '{{ $labels.device }}' at {{ $labels.instance }} is in degraded state due to one or more disks failures. Number of spare drives is insufficient to fix issue automatically. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/noderaiddegraded summary: RAID Array is degraded. expr: | node_md_disks_required{job="node-exporter",device=~"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)"} - ignoring (state) (node_md_disks{state="active",job="node-exporter",device=~"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)"}) > 0 for: 15m labels: severity: critical - alert: NodeRAIDDiskFailure annotations: description: At least one device in RAID array at {{ $labels.instance }} failed. Array '{{ $labels.device }}' needs attention and possibly a disk swap. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/noderaiddiskfailure summary: Failed device in RAID array. expr: | node_md_disks{state="failed",job="node-exporter",device=~"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)"} > 0 labels: severity: warning - alert: NodeFileDescriptorLimit annotations: description: File descriptors limit at {{ $labels.instance }} is currently at {{ printf "%.2f" $value }}%. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefiledescriptorlimit summary: Kernel is predicted to exhaust file descriptors limit soon. expr: | ( node_filefd_allocated{job="node-exporter"} * 100 / node_filefd_maximum{job="node-exporter"} > 70 ) for: 15m labels: severity: warning - alert: NodeFileDescriptorLimit annotations: description: File descriptors limit at {{ $labels.instance }} is currently at {{ printf "%.2f" $value }}%. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodefiledescriptorlimit summary: Kernel is predicted to exhaust file descriptors limit soon. expr: | ( node_filefd_allocated{job="node-exporter"} * 100 / node_filefd_maximum{job="node-exporter"} > 90 ) for: 15m labels: severity: critical - alert: NodeCPUHighUsage annotations: description: | CPU usage at {{ $labels.instance }} has been above 90% for the last 15 minutes, is currently at {{ printf "%.2f" $value }}%. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodecpuhighusage summary: High CPU usage. expr: | sum without(mode) (avg without (cpu) (rate(node_cpu_seconds_total{job="node-exporter", mode!="idle"}[2m]))) * 100 > 90 for: 15m labels: severity: info - alert: NodeSystemSaturation annotations: description: | System load per core at {{ $labels.instance }} has been above 2 for the last 15 minutes, is currently at {{ printf "%.2f" $value }}. This might indicate this instance resources saturation and can cause it becoming unresponsive. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodesystemsaturation summary: System saturated, load per core is very high. expr: | node_load1{job="node-exporter"} / count without (cpu, mode) (node_cpu_seconds_total{job="node-exporter", mode="idle"}) > 2 for: 15m labels: severity: warning - alert: NodeMemoryMajorPagesFaults annotations: description: | Memory major pages are occurring at very high rate at {{ $labels.instance }}, 500 major page faults per second for the last 15 minutes, is currently at {{ printf "%.2f" $value }}. Please check that there is enough memory available at this instance. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodememorymajorpagesfaults summary: Memory major page faults are occurring at very high rate. expr: | rate(node_vmstat_pgmajfault{job="node-exporter"}[5m]) > 500 for: 15m labels: severity: warning - alert: NodeMemoryHighUtilization annotations: description: | Memory is filling up at {{ $labels.instance }}, has been above 90% for the last 15 minutes, is currently at {{ printf "%.2f" $value }}%. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodememoryhighutilization summary: Host is running out of memory. expr: | 100 - (node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} * 100) > 90 for: 15m labels: severity: warning - alert: NodeDiskIOSaturation annotations: description: | Disk IO queue (aqu-sq) is high on {{ $labels.device }} at {{ $labels.instance }}, has been above 10 for the last 30 minutes, is currently at {{ printf "%.2f" $value }}. This symptom might indicate disk saturation. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodediskiosaturation summary: Disk IO queue is high. expr: | rate(node_disk_io_time_weighted_seconds_total{job="node-exporter", device=~"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)"}[5m]) > 10 for: 30m labels: severity: warning - alert: NodeSystemdServiceFailed annotations: description: Systemd service {{ $labels.name }} has entered failed state at {{ $labels.instance }} runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodesystemdservicefailed summary: Systemd service has entered failed state. expr: | node_systemd_unit_state{job="node-exporter", state="failed"} == 1 for: 5m labels: severity: warning - alert: NodeBondingDegraded annotations: description: Bonding interface {{ $labels.master }} on {{ $labels.instance }} is in degraded state due to one or more slave failures. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodebondingdegraded summary: Bonding interface is degraded expr: | (node_bonding_slaves - node_bonding_active) != 0 for: 5m labels: severity: warning - name: node-exporter.rules rules: - expr: | count without (cpu, mode) ( node_cpu_seconds_total{job="node-exporter",mode="idle"} ) record: instance:node_num_cpu:sum - expr: | 1 - avg without (cpu) ( sum without (mode) (rate(node_cpu_seconds_total{job="node-exporter", mode=~"idle|iowait|steal"}[5m])) ) record: instance:node_cpu_utilisation:rate5m - expr: | ( node_load1{job="node-exporter"} / instance:node_num_cpu:sum{job="node-exporter"} ) record: instance:node_load1_per_cpu:ratio - expr: | 1 - ( ( node_memory_MemAvailable_bytes{job="node-exporter"} or ( node_memory_Buffers_bytes{job="node-exporter"} + node_memory_Cached_bytes{job="node-exporter"} + node_memory_MemFree_bytes{job="node-exporter"} + node_memory_Slab_bytes{job="node-exporter"} ) ) / node_memory_MemTotal_bytes{job="node-exporter"} ) record: instance:node_memory_utilisation:ratio - expr: | rate(node_vmstat_pgmajfault{job="node-exporter"}[5m]) record: instance:node_vmstat_pgmajfault:rate5m - expr: | rate(node_disk_io_time_seconds_total{job="node-exporter", device=~"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)"}[5m]) record: instance_device:node_disk_io_time_seconds:rate5m - expr: | rate(node_disk_io_time_weighted_seconds_total{job="node-exporter", device=~"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)"}[5m]) record: instance_device:node_disk_io_time_weighted_seconds:rate5m - expr: | sum without (device) ( rate(node_network_receive_bytes_total{job="node-exporter", device!="lo"}[5m]) ) record: instance:node_network_receive_bytes_excluding_lo:rate5m - expr: | sum without (device) ( rate(node_network_transmit_bytes_total{job="node-exporter", device!="lo"}[5m]) ) record: instance:node_network_transmit_bytes_excluding_lo:rate5m - expr: | sum without (device) ( rate(node_network_receive_drop_total{job="node-exporter", device!="lo"}[5m]) ) record: instance:node_network_receive_drop_excluding_lo:rate5m - expr: | sum without (device) ( rate(node_network_transmit_drop_total{job="node-exporter", device!="lo"}[5m]) ) record: instance:node_network_transmit_drop_excluding_lo:rate5m#kubectl edit prometheusrule node-exporter-rules -n monitoring 可以直接修改 - alert: NodeMemoryHighUtilization annotations: description: | Memory is filling up at {{ $labels.instance }}, has been above 90% for the last 15 minutes, is currently at {{ printf "%.2f" $value }}%. runbook_url: https://runbooks.prometheus-operator.dev/runbooks/node/nodememoryhighutilization summary: Host is running out of memory. expr: | 100 - (node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} * 100) > 90 for: 15m labels: severity: warning annotations: 是注释信息/展示信息，不会参与告警匹配逻辑，但会出现在告警详情、通知内容里。常见用来放：描述、摘要、排障链接、负责人等。 alert: NodeMemoryHighUtilization 含义：告警规则的名字（Alert name）。在告警列表里显示的名称； Alertmanager 路由/分组/抑制（silence）时经常用它做匹配条件；下游通知（钉钉/飞书/Slack/邮件）里也会带这个名字。 description: | | 表示 多行字符串（保留换行）。 内容： Memory is filling up at {{ $labels.instance }}, has been above 90% for the last 15 minutes, is currently at {{ printf "%.2f" $value }}%. 这里用的是 Alertmanager 的模板变量（Go template 风格）： {{ $labels.instance }} 会被替换成该时间序列的 label 值，比如 10.0.0.12:9100。 这个 label 来自 metrics 本身（node-exporter 指标通常会带 instance）。 {{ $value }} 触发告警时表达式的计算值（这里是“内存使用率百分比”）。 printf "%.2f" 把数值格式化为 保留 2 位小数（比如 91.23%）。 注意：模板渲染发生在 告警发送/展示阶段，不影响 expr 的计算。 runbook_url: ... 含义：Runbook（排障手册）链接。 用途：当值班同学收到告警，点进去能看到： 常见原因； 排查步骤； 缓解/修复方法； 需要升级到谁。 这里链接指向 prometheus-operator 官方 runbook：nodememoryhighutilization。 summary: Host is running out of memory. 含义：一句话摘要。 用途：很多通知渠道会优先展示 summary，适合“短、明确”。 expr: | expr 是 PromQL 表达式，决定“什么时候触发告警”。 表达式： 100 - (node_memory_MemAvailable_bytes{job="node-exporter"} / node_memory_MemTotal_bytes{job="node-exporter"} node_memory_MemAvailable_bytes 来自：node-exporter（Linux 主机）。 含义：系统“可用内存”字节数（大致等于在不严重影响性能前提下可立即分配的内存，包括可回收 page cache 等）。 比 MemFree 更实用：MemFree 只算完全空闲，不算 cache/buffer 的可回收部分，容易误判。 node_memory_MemTotal_bytes 含义：总内存字节数。 node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100计算：可用内存占比（%）。 100 - (...)计算：已使用内存占比（%）。 使用率 = 100% - 可用率 > 90 阈值：使用率超过 90% 才算触发条件成立。 {job="node-exporter"} label 过滤器：只选 job label 为 node-exporter 的时间序列。 为什么需要：如果同名指标来自多个采集 job，用它避免混杂；也能减少误匹配。 可调整：如果你在 Prometheus 里 job 名不是 node-exporter，就得改这里，否则永远算不出来/不触发。 for: 15m 含义：表达式条件需要 连续成立 15 分钟 才“真正触发”（从 Pending 变成 Firing）。 用途： 抑制抖动（比如短时间内存尖峰、瞬时 load）。 避免频繁通知。 与你的文案对应：description 写了 “above 90% for the last 15 minutes”，就是呼应 for: 15m。 注意： 如果指标中断、抓取失败、或者值掉下阈值又上来，计时会重置。 labels: labels 是告警标签，会参与 Alertmanager 路由、分组、去重。 severity: warning 含义：告警级别。 用途： Alertmanager 根据 severity 把告警发到不同渠道（warning 走群通知，critical 走电话/短信）。 也可用于抑制策略（warning 被 critical 覆盖等）。 常见约定：info / warning / critical（团队可自定义，但要统一）。 #或者直接改yaml文件 然后apply 我是没有helm部署的 直接apply https://axzys.cn/index.php/archives/423/ root@k8s-01:/woke/prometheus/kube-prometheus/manifests# grep -R "NodeMemoryHighUtilization" nodeExporter-prometheusRule.yaml: - alert: NodeMemoryHighUtilization root@k8s-01:/woke/prometheus/kube-prometheus/manifests# 改了 直接apply 也可以
一、代码#初始化目录 cd ~ mkdir feishu-forwarder cd feishu-forwarder go mod init feishu-forwarder #Dockerfile FROM golang:1.24.2-alpine AS build WORKDIR /src COPY go.mod ./ RUN go mod download COPY . . RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" -o /out/feishu-forwarder . FROM gcr.io/distroless/static:nonroot COPY --from=build /out/feishu-forwarder /feishu-forwarder EXPOSE 8080 USER nonroot:nonroot ENTRYPOINT ["/feishu-forwarder"] #main.go package main import ( "bytes" "context" "crypto/hmac" "crypto/sha256" "encoding/base64" "encoding/hex" "encoding/json" "io" "log" "math/rand" "net/http" "os" "sort" "strings" "sync" "time" ) type Alert struct { Status string `json:"status"` Labels map[string]string `json:"labels"` Annotations map[string]string `json:"annotations"` StartsAt string `json:"startsAt"` EndsAt string `json:"endsAt"` GeneratorURL string `json:"generatorURL"` Fingerprint string `json:"fingerprint"` // Alertmanager webhook 通常会带；没有也没关系 } type AMPayload struct { Status string `json:"status"` Receiver string `json:"receiver"` ExternalURL string `json:"externalURL"` GroupKey string `json:"groupKey"` CommonLabels map[string]string `json:"commonLabels"` Alerts []Alert `json:"alerts"` } // webhook body：text 用 content；card 用 card type FeishuBody struct { Timestamp string `json:"timestamp,omitempty"` Sign string `json:"sign,omitempty"` MsgType string `json:"msg_type"` Content map[string]interface{} `json:"content,omitempty"` Card map[string]interface{} `json:"card,omitempty"` } type Target struct { Webhook string Secret string Name string // 用于日志 } type Config struct { // 路由：按 severity 选择目标群 WebhooksCritical string WebhooksWarning string WebhooksDefault string SecretCritical string SecretWarning string SecretDefault string // 消息类型：card / text MsgType string // 去重窗口 DedupTTL time.Duration // 重试 RetryMax int RetryBase time.Duration // 限流（全局） RateQPS float64 RateBurst int // HTTP SendTimeout time.Duration } func loadConfig() Config { cfg := Config{ WebhooksCritical: strings.TrimSpace(os.Getenv("FEISHU_WEBHOOKS_CRITICAL")), WebhooksWarning: strings.TrimSpace(os.Getenv("FEISHU_WEBHOOKS_WARNING")), WebhooksDefault: strings.TrimSpace(os.Getenv("FEISHU_WEBHOOKS_DEFAULT")), SecretCritical: strings.TrimSpace(os.Getenv("FEISHU_SECRET_CRITICAL")), SecretWarning: strings.TrimSpace(os.Getenv("FEISHU_SECRET_WARNING")), SecretDefault: strings.TrimSpace(os.Getenv("FEISHU_SECRET_DEFAULT")), MsgType: strings.ToLower(strings.TrimSpace(os.Getenv("FEISHU_MSG_TYPE"))), DedupTTL: mustParseDuration(getEnvDefault("DEDUP_TTL", "10m")), RetryMax: mustParseInt(getEnvDefault("RETRY_MAX", "3")), RetryBase: mustParseDuration(getEnvDefault("RETRY_BASE", "300ms")), RateQPS: mustParseFloat(getEnvDefault("RATE_QPS", "2")), // 每秒2条 RateBurst: mustParseInt(getEnvDefault("RATE_BURST", "5")), // 突发5条 SendTimeout: mustParseDuration(getEnvDefault("SEND_TIMEOUT", "6s")), } if cfg.MsgType != "text" && cfg.MsgType != "card" { cfg.MsgType = "card" } // 兼容：如果你只设置了旧的 FEISHU_WEBHOOKS/FEISHU_SECRET if cfg.WebhooksDefault == "" { cfg.WebhooksDefault = strings.TrimSpace(os.Getenv("FEISHU_WEBHOOKS")) } if cfg.SecretDefault == "" { cfg.SecretDefault = strings.TrimSpace(os.Getenv("FEISHU_SECRET")) } return cfg } /* ============ 去重（内存 TTL） ============ */ type Deduper struct { mu sync.Mutex ttl time.Duration data map[string]time.Time } func NewDeduper(ttl time.Duration) *Deduper { d := &Deduper{ ttl: ttl, data: make(map[string]time.Time), } go d.gcLoop() return d } func (d *Deduper) Allow(key string) bool { if d.ttl <= 0 { return true } now := time.Now() d.mu.Lock() defer d.mu.Unlock() if t, ok := d.data[key]; ok { if now.Sub(t) < d.ttl { return false } } d.data[key] = now return true } func (d *Deduper) gcLoop() { t := time.NewTicker(1 * time.Minute) defer t.Stop() for range t.C { now := time.Now() d.mu.Lock() for k, v := range d.data { if now.Sub(v) > d.ttl*2 { delete(d.data, k) } } d.mu.Unlock() } } /* ============ 简单全局限流（token bucket） ============ */ type RateLimiter struct { ch chan struct{} stop chan struct{} } func NewRateLimiter(qps float64, burst int) *RateLimiter { if qps <= 0 { return nil } if burst < 1 { burst = 1 } rl := &RateLimiter{ ch: make(chan struct{}, burst), stop: make(chan struct{}), } // 初始塞满 burst for i := 0; i < burst; i++ { rl.ch <- struct{}{} } interval := time.Duration(float64(time.Second) / qps) if interval < 10*time.Millisecond { interval = 10 * time.Millisecond } go func() { t := time.NewTicker(interval) defer t.Stop() for { select { case <-t.C: select { case rl.ch <- struct{}{}: default: } case <-rl.stop: return } } }() return rl } func (rl *RateLimiter) Acquire(ctx context.Context) error { if rl == nil { return nil } select { case <-ctx.Done(): return ctx.Err() case <-rl.ch: return nil } } /* ============ 飞书签名 ============ */ func genFeishuSign(secret, timestamp string) string { stringToSign := timestamp + "\n" + secret mac := hmac.New(sha256.New, []byte(stringToSign)) sum := mac.Sum(nil) return base64.StdEncoding.EncodeToString(sum) } /* ============ severity 路由 ============ */ func getSeverity(p AMPayload) string { s := strings.ToLower(strings.TrimSpace(p.CommonLabels["severity"])) if s == "" && len(p.Alerts) > 0 { s = strings.ToLower(strings.TrimSpace(p.Alerts[0].Labels["severity"])) } switch s { case "critical", "fatal", "sev0", "sev1": return "critical" case "warning", "warn", "sev2": return "warning" default: return "default" } } func splitWebhooks(s string) []string { s = strings.TrimSpace(s) if s == "" { return nil } parts := strings.Split(s, ",") var out []string for _, x := range parts { x = strings.TrimSpace(x) if x != "" { out = append(out, x) } } return out } func selectTargets(cfg Config, sev string) []Target { var whs []string var secret string var name string switch sev { case "critical": whs = splitWebhooks(cfg.WebhooksCritical) secret = cfg.SecretCritical name = "critical" case "warning": whs = splitWebhooks(cfg.WebhooksWarning) secret = cfg.SecretWarning name = "warning" default: whs = splitWebhooks(cfg.WebhooksDefault) secret = cfg.SecretDefault name = "default" } // fallback：如果 critical/warning 没配，则退回 default if len(whs) == 0 { whs = splitWebhooks(cfg.WebhooksDefault) secret = cfg.SecretDefault name = "default(fallback)" } var out []Target for _, w := range whs { out = append(out, Target{Webhook: w, Secret: secret, Name: name}) } return out } /* ============ 去重 key ============ */ func alertKey(a Alert) string { if a.Fingerprint != "" { return a.Fingerprint } // 没 fingerprint 就用 labels 自己算一个稳定 hash keys := make([]string, 0, len(a.Labels)) for k := range a.Labels { keys = append(keys, k) } sort.Strings(keys) var b strings.Builder for _, k := range keys { b.WriteString(k) b.WriteString("=") b.WriteString(a.Labels[k]) b.WriteString(";") } sum := sha256.Sum256([]byte(b.String())) return hex.EncodeToString(sum[:]) } /* ============ 消息构建 ============ */ func formatText(p AMPayload, alerts []Alert) string { var b strings.Builder b.WriteString("【Alertmanager】" + strings.ToUpper(p.Status) + "\n") if v := p.CommonLabels["alertname"]; v != "" { b.WriteString("alertname: " + v + "\n") } if v := p.CommonLabels["severity"]; v != "" { b.WriteString("severity: " + v + "\n") } b.WriteString("alerts: ") b.WriteString(intToString(len(alerts))) b.WriteString("\n\n") for i, a := range alerts { b.WriteString("#") b.WriteString(intToString(i + 1)) b.WriteString(" ") if an := a.Labels["alertname"]; an != "" { b.WriteString(an) } if inst := a.Labels["instance"]; inst != "" { b.WriteString(" @ " + inst) } b.WriteString("\n") if s := a.Annotations["summary"]; s != "" { b.WriteString("summary: " + s + "\n") } if d := a.Annotations["description"]; d != "" { b.WriteString("desc: " + d + "\n") } if a.GeneratorURL != "" { b.WriteString("url: " + a.GeneratorURL + "\n") } b.WriteString("\n") } return b.String() } func buildCard(p AMPayload, sev string, alerts []Alert) map[string]interface{} { alertname := p.CommonLabels["alertname"] if alertname == "" && len(alerts) > 0 { alertname = alerts[0].Labels["alertname"] } // 颜色：FIRING + critical 用 red；warning 用 orange；resolved 用 green template := "blue" if strings.ToLower(p.Status) == "firing" { if sev == "critical" { template = "red" } else if sev == "warning" { template = "orange" } else { template = "blue" } } else { template = "green" } title := "[" + strings.ToUpper(p.Status) + "][" + sev + "] " + alertname + " (" + intToString(len(alerts)) + ")" // 内容用 markdown，最多展示前 5 条（避免卡片过长） maxShow := 5 show := alerts more := 0 if len(alerts) > maxShow { show = alerts[:maxShow] more = len(alerts) - maxShow } var md strings.Builder md.WriteString("**Receiver:** " + p.Receiver + "\n") if p.ExternalURL != "" { md.WriteString("**Alertmanager:** " + p.ExternalURL + "\n") } md.WriteString("\n") for i, a := range show { an := a.Labels["alertname"] inst := a.Labels["instance"] md.WriteString("**#" + intToString(i+1) + "** " + an) if inst != "" { md.WriteString(" @ `" + inst + "`") } md.WriteString("\n") if s := a.Annotations["summary"]; s != "" { md.WriteString("- **summary:** " + s + "\n") } if d := a.Annotations["description"]; d != "" { md.WriteString("- **desc:** " + d + "\n") } if a.StartsAt != "" { md.WriteString("- **startsAt:** " + a.StartsAt + "\n") } if a.GeneratorURL != "" { md.WriteString("- **url:** " + a.GeneratorURL + "\n") } md.WriteString("\n") } if more > 0 { md.WriteString("…还有 **" + intToString(more) + "** 条未展开\n") } // 如果有 generatorURL，就加一个按钮（取第一条有 url 的） btnURL := "" for _, a := range alerts { if a.GeneratorURL != "" { btnURL = a.GeneratorURL break } } elements := []interface{}{ map[string]interface{}{"tag": "markdown", "content": md.String()}, } if btnURL != "" { elements = append(elements, map[string]interface{}{"tag": "hr"}, map[string]interface{}{ "tag": "action", "actions": []interface{}{ map[string]interface{}{ "tag": "button", "text": map[string]interface{}{"tag": "plain_text", "content": "打开规则/图表"}, "type": "primary", "url": btnURL, }, }, }, ) } card := map[string]interface{}{ "config": map[string]interface{}{ "wide_screen_mode": true, }, "header": map[string]interface{}{ "title": map[string]interface{}{"tag": "plain_text", "content": title}, "template": template, }, "elements": elements, } return card } /* ============ 发送：重试 + 限流 ============ */ type httpError struct { code int body string } func (e *httpError) Error() string { if e.body != "" { return "http status " + intToString(e.code) + " body=" + e.body } return "http status " + intToString(e.code) } func isRetryableStatus(code int) bool { return code == 429 || code >= 500 } func doPost(ctx context.Context, client *http.Client, url string, payload []byte) error { req, _ := http.NewRequestWithContext(ctx, "POST", url, bytes.NewReader(payload)) req.Header.Set("Content-Type", "application/json") resp, err := client.Do(req) if err != nil { return err } defer resp.Body.Close() if resp.StatusCode/100 == 2 { return nil } // 读一点返回体，方便排查（最多 2KB） b, _ := io.ReadAll(io.LimitReader(resp.Body, 2048)) return &httpError{code: resp.StatusCode, body: strings.TrimSpace(string(b))} } func sendWithRetry(ctx context.Context, rl *RateLimiter, cfg Config, t Target, body FeishuBody) error { client := &http.Client{Timeout: cfg.SendTimeout} payload, _ := json.Marshal(body) r := rand.New(rand.NewSource(time.Now().UnixNano())) var lastErr error for i := 0; i < cfg.RetryMax; i++ { // 限流：每次尝试都要拿 token（避免重试时把飞书打爆） if err := rl.Acquire(ctx); err != nil { return err } err := doPost(ctx, client, t.Webhook, payload) if err == nil { return nil } lastErr = err // 只有可重试错误才重试 retry := false if he, ok := err.(*httpError); ok { retry = isRetryableStatus(he.code) } else { retry = true // 网络错误等 } if !retry || i == cfg.RetryMax-1 { break } // 退避：base * 2^i + jitter(0~100ms) sleep := cfg.RetryBase * time.Duration(1<<i) sleep += time.Duration(r.Intn(100)) * time.Millisecond select { case <-ctx.Done(): return ctx.Err() case <-time.After(sleep): } } return lastErr } func buildFeishuBody(cfg Config, sev string, p AMPayload, alerts []Alert, webhookSecret string) FeishuBody { ts := "" sign := "" if webhookSecret != "" { ts = int64ToString(time.Now().Unix()) sign = genFeishuSign(webhookSecret, ts) } if cfg.MsgType == "text" { return FeishuBody{ Timestamp: ts, Sign: sign, MsgType: "text", Content: map[string]interface{}{ "text": formatText(p, alerts), }, } } // card return FeishuBody{ Timestamp: ts, Sign: sign, MsgType: "interactive", Card: buildCard(p, sev, alerts), } } /* ============ helpers ============ */ func getEnvDefault(k, def string) string { v := strings.TrimSpace(os.Getenv(k)) if v == "" { return def } return v } func mustParseDuration(s string) time.Duration { d, err := time.ParseDuration(s) if err != nil { return 0 } return d } func mustParseInt(s string) int { n := 0 for _, ch := range s { if ch >= '0' && ch <= '9' { n = n*10 + int(ch-'0') } } if n == 0 { return 0 } return n } func mustParseFloat(s string) float64 { // 简易 parse：支持 "2" "2.5" var n, frac, div float64 div = 1 seenDot := false for _, ch := range s { if ch == '.' { seenDot = true continue } if ch < '0' || ch > '9' { continue } if !seenDot { n = n*10 + float64(ch-'0') } else { frac = frac*10 + float64(ch-'0') div *= 10 } } return n + frac/div } func intToString(x int) string { return int64ToString(int64(x)) } func int64ToString(x int64) string { if x == 0 { return "0" } neg := x < 0 if neg { x = -x } var buf [32]byte i := len(buf) for x > 0 { i-- buf[i] = byte('0' + x%10) x /= 10 } if neg { i-- buf[i] = '-' } return string(buf[i:]) } /* ============ main ============ */ func main() { log.SetFlags(log.LstdFlags | log.Lmicroseconds) cfg := loadConfig() deduper := NewDeduper(cfg.DedupTTL) rl := NewRateLimiter(cfg.RateQPS, cfg.RateBurst) http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) w.Write([]byte("ok")) }) http.HandleFunc("/alertmanager", func(w http.ResponseWriter, r *http.Request) { if r.Method != "POST" { w.WriteHeader(405) return } var p AMPayload if err := json.NewDecoder(r.Body).Decode(&p); err != nil { log.Printf("decode failed: remote=%s err=%v", r.RemoteAddr, err) w.WriteHeader(400) w.Write([]byte(err.Error())) return } sev := getSeverity(p) log.Printf("recv webhook: remote=%s status=%s receiver=%s alerts=%d severity=%s alertname=%s", r.RemoteAddr, p.Status, p.Receiver, len(p.Alerts), sev, p.CommonLabels["alertname"]) // 去重（只对 firing 做，resolved 不去重，避免“恢复消息被吞”） alertsToSend := make([]Alert, 0, len(p.Alerts)) suppressed := 0 for _, a := range p.Alerts { if strings.ToLower(p.Status) == "firing" && cfg.DedupTTL > 0 { key := "firing:" + alertKey(a) if !deduper.Allow(key) { suppressed++ continue } } alertsToSend = append(alertsToSend, a) } if len(alertsToSend) == 0 { log.Printf("dedup suppressed all alerts: suppressed=%d total=%d", suppressed, len(p.Alerts)) w.WriteHeader(200) w.Write([]byte("ok")) return } targets := selectTargets(cfg, sev) if len(targets) == 0 { log.Printf("no targets configured for severity=%s", sev) w.WriteHeader(500) w.Write([]byte("no targets configured")) return } okCount := 0 failCount := 0 // 给本次请求一个总体 deadline（避免 handler 卡太久） ctx, cancel := context.WithTimeout(r.Context(), cfg.SendTimeout*time.Duration(cfg.RetryMax+1)) defer cancel() for _, t := range targets { body := buildFeishuBody(cfg, sev, p, alertsToSend, t.Secret) err := sendWithRetry(ctx, rl, cfg, t, body) if err != nil { log.Printf("send failed: group=%s target=%s err=%v", t.Name, t.Webhook, err) failCount++ continue } log.Printf("send ok: group=%s target=%s", t.Name, t.Webhook) okCount++ } log.Printf("send summary: ok=%d fail=%d targets=%d suppressed=%d severity=%s", okCount, failCount, len(targets), suppressed, sev) if okCount == 0 { w.WriteHeader(502) w.Write([]byte("failed")) return } w.WriteHeader(200) w.Write([]byte("ok")) }) addr := ":8080" log.Printf("listening on %s msgType=%s dedup=%s retryMax=%d rateQps=%.2f burst=%d", addr, cfg.MsgType, cfg.DedupTTL, cfg.RetryMax, cfg.RateQPS, cfg.RateBurst) log.Fatal(http.ListenAndServe(addr, nil)) } 二、构建部署docker build -t harbor.axzys.cn/monitoring/feishu-forwarder:0.1.0 . docker push harbor.axzys.cn/monitoring/feishu-forwarder:0.1.0 .cat feishu.yaml apiVersion: apps/v1 kind: Deployment metadata: name: feishu-forwarder namespace: monitoring spec: replicas: 1 selector: matchLabels: { app: feishu-forwarder } template: metadata: labels: { app: feishu-forwarder } spec: containers: - name: app image: harbor.axzys.cn/monitoring/feishu-forwarder:0.1.2 imagePullPolicy: Always ports: - containerPort: 8080 env: - name: FEISHU_WEBHOOKS_CRITICAL valueFrom: { secretKeyRef: { name: feishu-forwarder-secret, key: FEISHU_WEBHOOKS_CRITICAL } } - name: FEISHU_WEBHOOKS_WARNING valueFrom: { secretKeyRef: { name: feishu-forwarder-secret, key: FEISHU_WEBHOOKS_WARNING } } - name: FEISHU_WEBHOOKS_DEFAULT valueFrom: { secretKeyRef: { name: feishu-forwarder-secret, key: FEISHU_WEBHOOKS_DEFAULT } } - name: FEISHU_SECRET_CRITICAL valueFrom: { secretKeyRef: { name: feishu-forwarder-secret, key: FEISHU_SECRET_CRITICAL } } - name: FEISHU_SECRET_WARNING valueFrom: { secretKeyRef: { name: feishu-forwarder-secret, key: FEISHU_SECRET_WARNING } } - name: FEISHU_SECRET_DEFAULT valueFrom: { secretKeyRef: { name: feishu-forwarder-secret, key: FEISHU_SECRET_DEFAULT } } - name: FEISHU_MSG_TYPE value: "card" # card / text - name: DEDUP_TTL value: "10m" - name: RETRY_MAX value: "3" - name: RETRY_BASE value: "300ms" - name: RATE_QPS value: "2" - name: RATE_BURST value: "5" - name: SEND_TIMEOUT value: "6s" readinessProbe: httpGet: { path: /healthz, port: 8080 } initialDelaySeconds: 2 periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: name: feishu-forwarder namespace: monitoring spec: selector: { app: feishu-forwarder } ports: - name: http port: 8080 targetPort: 8080 type: ClusterIP cat feishu-secret.yaml apiVersion: v1 kind: Secret metadata: name: feishu-forwarder-secret namespace: monitoring type: Opaque stringData: FEISHU_WEBHOOKS_CRITICAL: "https://open.feishu.cn/open-apis/bot/v2/hook/飞书token" FEISHU_WEBHOOKS_WARNING: "https://open.feishu.cn/open-apis/bot/v2/hook/飞书token" FEISHU_WEBHOOKS_DEFAULT: "https://open.feishu.cn/open-apis/bot/v2/hook/xxxx" # 如果你开了“签名校验”，不同群机器人 secret 往往不同 FEISHU_SECRET_CRITICAL: "SEC_xxx" FEISHU_SECRET_WARNING: "SEC_yyy" FEISHU_SECRET_DEFAULT: "SEC_zzz" cat alertmanagerConfig.yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name: feishu-forwarder namespace: monitoring labels: alertmanagerConfig: main # 这行是否需要，取决于你的 Alertmanager 选择器 spec: route: receiver: feishu-forwarder groupWait: 30s groupInterval: 5m repeatInterval: 30m receivers: - name: feishu-forwarder webhookConfigs: - url: http://feishu-forwarder.monitoring.svc:8080/alertmanager sendResolved: true root@k8s-01:/woke/prometheus/feishu# kubectl get pod -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 0 7d2h alertmanager-main-1 2/2 Running 0 7d2h alertmanager-main-2 2/2 Running 0 7d2h blackbox-exporter-7fcbd888d-zv6z6 3/3 Running 0 15d feishu-forwarder-8559bf6b68-njzvq 1/1 Running 0 68m grafana-7ff454c477-l9x2k 1/1 Running 0 15d kube-state-metrics-78f95f79bb-wpcln 3/3 Running 0 15d node-exporter-2vq26 2/2 Running 2 (35d ago) 39d node-exporter-622pm 2/2 Running 24 (35d ago) 39d node-exporter-rl67z 2/2 Running 22 (35d ago) 39d prometheus-adapter-585d9c5dd5-bfsxw 1/1 Running 0 8d prometheus-adapter-585d9c5dd5-pcrnd 1/1 Running 0 8d prometheus-k8s-0 2/2 Running 0 7d2h prometheus-k8s-1 2/2 Running 0 7d2h prometheus-operator-78967669c9-5pk25 2/2 Running 0 7d2h 三、调试 kubectl -n monitoring port-forward svc/alertmanager-main 19093:9093 kubectl -n monitoring logs -f deploy/feishu-forwarder curl -i -X POST http://127.0.0.1:19093/api/v2/alerts \ -H 'Content-Type: application/json' \ -d '[{ "labels":{"alertname":"axingWarning","severity":"warning","instance":"demo:9100","namespace":"monitoring"}, "annotations":{"summary":"injected warning","description":"from alertmanager api"}, "startsAt":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'" }]'操作过程中出现了 prometheus 已经告警 并且已经到Alertmanager了 但是 feishu-forwarder 没有收到 解决方法 是加上 不然他只会通知monitoring这个名称空间的告警信息 matchers: - name: prometheus value: monitoring/k8s 还有就是InfoInhibitor 一直刷屏通知告警 有两种解决方法 一、就是改上面的feishu-secret把FEISHU_WEBHOOKS_DEFAULT注释掉 还有一种就是下面这种只通知warning/critical root@k8s-01:/woke/prometheus/feishu# cat alertmanagerConfig.yaml apiVersion: monitoring.coreos.com/v1alpha1 kind: AlertmanagerConfig metadata: name: feishu-forwarder namespace: monitoring labels: alertmanagerConfig: main spec: route: receiver: feishu-forwarder groupWait: 30s groupInterval: 5m repeatInterval: 30m matchers: - name: prometheus value: monitoring/k8s matchType: "=" # 建议：只推 warning/critical，InfoInhibitor(severity=none) 就不会再刷飞书 - name: severity value: warning|critical matchType: "=~" receivers: - name: feishu-forwarder webhookConfigs: - url: http://feishu-forwarder.monitoring.svc:8080/alertmanager sendResolved: true
一、升级最稳妥的 LTS 逐级升级（推荐） 先补齐本线最后一个补丁：2.516.3（把 2.516 线能拿到的修复先吃满） 升到下一条 LTS 基线：2.528.3（过渡 LTS 线） 升到当前在维护的 LTS：2.541.1（截至 2026-02-05 仍是主流 LTS） 这么做的好处：每一步跨度小，遇到插件/鉴权问题更容易定位；并且升级指南也建议如果跳 LTS，要把中间每段的升级注意事项都看一遍。root@k8s-01:/woke/jenkins# cat deployment.yaml # jenkins-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: jenkins namespace: jenkins spec: replicas: 1 selector: matchLabels: app: jenkins template: metadata: labels: app: jenkins spec: securityContext: fsGroup: 1000 serviceAccountName: jenkins-admin containers: - name: jenkins image: registry.cn-guangzhou.aliyuncs.com/xingcangku/jenkins-jenkins-lts-jdk17:lts-jdk17 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 - containerPort: 50000 # ★ JVM 参数 env: - name: JENKINS_JAVA_OPTIONS value: "-Djava.net.preferIPv4Stack=true" - name: JAVA_OPTS value: "-Djava.net.preferIPv4Stack=true" volumeMounts: - name: jenkins-data mountPath: /var/jenkins_home # ★ 新增挂载，把每个节点上的 /root/cicd 挂进 Jenkins 家目录 - name: cicd-tools mountPath: /var/jenkins_home/cicd resources: limits: cpu: "1" memory: "4Gi" requests: cpu: "0.5" memory: "1Gi" livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 90 periodSeconds: 10 volumes: - name: jenkins-data persistentVolumeClaim: claimName: jenkins-pvc # ★ 新增 volume，使用 hostPath 指向每个节点的 /root/cicd - name: cicd-tools hostPath: path: /root/cicd type: Directory # 如果想让 k8s 自动创建目录，也可以用： # type: DirectoryOrCreate
一、流量走势NodePort → ingressgateway Service → ingressgateway Pod(Envoy) → Gateway 筛选 → VirtualService 路由 → productpage Service:9080 → productpage Pod #如果不是NodePort 就需要kubectl port-forward -n istio-system --address 0.0.0.0 svc/istio-ingressgateway 8080:80 进行端口暴露 只需要记住一点svc只是负责把流量转发到对应的pod 而网关和路由的匹配则是istio-ingressgateway进行处理二、对应的yaml文件cat gateway.yaml apiVersion: networking.istio.io/v1beta1 kind: Gateway metadata: name: bookinfo-gateway namespace: default spec: selector: istio: ingressgateway servers: - port: number: 80 name: http protocol: HTTP hosts: - "*" root@k8s-01:/woke/istio# kubectl get virtualservice bookinfo -n default -o yaml apiVersion: networking.istio.io/v1 kind: VirtualService metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"annotations":{},"name":"bookinfo","namespace":"default"},"spec":{"gateways":["bookinfo-gateway"],"hosts":["*"],"http":[{"match":[{"uri":{"prefix":"/"}}],"route":[{"destination":{"host":"productpage","port":{"number":9080}}}]}]}} creationTimestamp: "2025-12-31T05:30:04Z" generation: 1 name: bookinfo namespace: default resourceVersion: "8121204" uid: b0fd2c20-35ab-475c-8c1a-d6dc36749ce8 spec: gateways: - bookinfo-gateway hosts: - '*' http: - match: - uri: prefix: / route: - destination: host: productpage port: number: 9080三、对应svc和podroot@k8s-01:/woke/istio# kubectl get pod NAME READY STATUS RESTARTS AGE bookinfo-gateway-istio-6c4c8d9b74-z7c4j 1/1 Running 0 18d details-v1-7b88fb8889-5qt26 2/2 Running 0 33d nginx-5fd58574d-2c9z9 2/2 Running 0 13d nginx-test-78b8db46d6-cfd98 2/2 Running 0 13d productpage-v1-79d79d6-mktpd 2/2 Running 0 33d ratings-v1-5b89cf4bbf-c9xdg 2/2 Running 0 33d reviews-v1-ffd57b847-w9dkr 2/2 Running 0 33d reviews-v2-8cc6fd8dc-q9bd9 2/2 Running 0 33d reviews-v3-cf48455-znts5 2/2 Running 0 33d traefik-gw-istio-7867b4f544-hkknj 1/1 Running 0 18d root@k8s-01:/woke/istio# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bookinfo-gateway-istio ClusterIP 10.105.38.228 <none> 15021/TCP,80/TCP 18d details ClusterIP 10.100.97.40 <none> 9080/TCP 35d kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 78d nginx ClusterIP 10.110.220.61 <none> 80/TCP 60d nginx-svc NodePort 10.108.4.45 <none> 80:32574/TCP 77d productpage ClusterIP 10.111.145.105 <none> 9080/TCP 35d ratings ClusterIP 10.98.94.10 <none> 9080/TCP 35d reviews ClusterIP 10.97.19.38 <none> 9080/TCP 35d traefik-gw-istio LoadBalancer 10.103.179.128 <pending> 15021:31055/TCP,8000:30447/TCP 18d root@k8s-01:/woke/istio# kubectl get pod -n istio-system NAME READY STATUS RESTARTS AGE istio-egressgateway-596f455c4f-7qh6w 1/1 Running 0 13d istio-ingressgateway-796f5cf647-tcpph 1/1 Running 0 13d istiod-5c84f8c79d-ddxkz 1/1 Running 0 13d kiali-68f9949bf5-8n4tk 1/1 Running 6 (13d ago) 13d prometheus-7f6bc65664-29c7c 0/2 Completed 0 18d prometheus-7f6bc65664-llp2q 2/2 Running 0 5d root@k8s-01:/woke/istio# kubectl get svc -n istio-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE istio-egressgateway ClusterIP 10.106.183.187 <none> 80/TCP,443/TCP 35d istio-ingressgateway NodePort 10.99.189.246 <none> 15021:31689/TCP,80:32241/TCP,443:30394/TCP,31400:31664/TCP,15443:32466/TCP 35d istiod ClusterIP 10.104.189.5 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 35d istiod-revision-tag-default ClusterIP 10.102.152.242 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 35d kiali NodePort 10.103.234.52 <none> 20001:30318/TCP,9090:30682/TCP 34d prometheus ClusterIP 10.100.183.52 <none> 9090/TCP 34d root@k8s-01:/woke/istio#
一、找回密码root@k8s-01:~# kubectl -n monitoring get pod | grep grafana grafana-7ff454c477-l9x2k 1/1 Running 0 8d root@k8s-01:~# kubectl -n monitoring exec -it grafana-7ff454c477-l9x2k -- sh /usr/share/grafana $ grafana-cli admin reset-admin-password '33070595Abc' Deprecation warning: The standalone 'grafana-cli' program is deprecated and will be removed in the future. Please update all uses of 'grafana-cli' to 'grafana cli' INFO [01-29|12:45:37] Starting Grafana logger=settings version= commit= branch= compiled=1970-01-01T00:00:00Z INFO [01-29|12:45:37] Config loaded from logger=settings file=/usr/share/grafana/conf/defaults.ini INFO [01-29|12:45:37] Config overridden from Environment variable logger=settings var="GF_PATHS_DATA=/var/lib/grafana" INFO [01-29|12:45:37] Config overridden from Environment variable logger=settings var="GF_PATHS_LOGS=/var/log/grafana" INFO [01-29|12:45:37] Config overridden from Environment variable logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" INFO [01-29|12:45:37] Config overridden from Environment variable logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" INFO [01-29|12:45:37] Target logger=settings target=[all] INFO [01-29|12:45:37] Path Home logger=settings path=/usr/share/grafana INFO [01-29|12:45:37] Path Data logger=settings path=/var/lib/grafana INFO [01-29|12:45:37] Path Logs logger=settings path=/var/log/grafana INFO [01-29|12:45:37] Path Plugins logger=settings path=/var/lib/grafana/plugins INFO [01-29|12:45:37] Path Provisioning logger=settings path=/etc/grafana/provisioning INFO [01-29|12:45:37] App mode production logger=settings INFO [01-29|12:45:37] FeatureToggles logger=featuremgmt recoveryThreshold=true panelMonitoring=true lokiQuerySplitting=true nestedFolders=true logsContextDatasourceUi=true cloudWatchNewLabelParsing=true logRowsPopoverMenu=true kubernetesPlaylists=true dataplaneFrontendFallback=true recordedQueriesMulti=true transformationsVariableSupport=true addFieldFromCalculationStatFunctions=true cloudWatchCrossAccountQuerying=true prometheusAzureOverrideAudience=true lokiQueryHints=true logsExploreTableVisualisation=true annotationPermissionUpdate=true lokiMetricDataplane=true prometheusMetricEncyclopedia=true lokiStructuredMetadata=true topnav=true alertingInsights=true exploreMetrics=true formatString=true ssoSettingsApi=true autoMigrateXYChartPanel=true tlsMemcached=true prometheusConfigOverhaulAuth=true logsInfiniteScrolling=true alertingSimplifiedRouting=true awsAsyncQueryCaching=true managedPluginsInstall=true cloudWatchRoundUpEndTime=true transformationsRedesign=true alertingNoDataErrorExecution=true dashgpt=true influxdbBackendMigration=true prometheusDataplane=true groupToNestedTableTransformation=true correlations=true publicDashboards=true angularDeprecationUI=true INFO [01-29|12:45:37] Connecting to DB logger=sqlstore dbtype=sqlite3 INFO [01-29|12:45:37] Locking database logger=migrator INFO [01-29|12:45:37] Starting DB migrations logger=migrator INFO [01-29|12:45:37] migrations completed logger=migrator performed=0 skipped=594 duration=359.617µs INFO [01-29|12:45:37] Unlocking database logger=migrator INFO [01-29|12:45:37] Envelope encryption state logger=secrets enabled=true current provider=secretKey.v1 Admin password changed successfully ✔ /usr/share/grafana $ exit root@k8s-01:~# cat <<'EOF' | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-traefik-to-grafana namespace: monitoring spec: podSelector: matchLabels: app.kubernetes.io/component: grafana app.kubernetes.io/name: grafana app.kubernetes.io/part-of: kube-prometheus policyTypes: ["Ingress"] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: traefik ports: - protocol: TCP port: 3000 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-traefik-to-prometheus namespace: monitoring spec: podSelector: matchLabels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus policyTypes: ["Ingress"] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: traefik ports: - protocol: TCP port: 9090 --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-traefik-to-alertmanager namespace: monitoring spec: podSelector: matchLabels: app.kubernetes.io/component: alert-router app.kubernetes.io/instance: main app.kubernetes.io/name: alertmanager app.kubernetes.io/part-of: kube-prometheus policyTypes: ["Ingress"] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: traefik ports: - protocol: TCP port: 9093 EOF 需要放行 monitoring 里原来的 NetworkPolicy 没放行 traefik → 导致 Traefik 转发到 grafana/prometheus/alertmanager 全部被丢包，最后表现成 504/超时。#添加节点 创建 additionalScrapeConfigs 的 Secret（把外部节点加进去） #我这里用 job_name: node-exporter 是为了让你现成的 Node Exporter / Nodes 仪表盘直接复用（很多面板按 job 过滤）。 cat <<'EOF' | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: prometheus-additional-scrape-configs namespace: monitoring type: Opaque stringData: additional-scrape-configs.yaml: | - job_name: node-exporter static_configs: - targets: - 192.168.1.12:9100 - 192.168.1.15:9100 - 192.168.1.30:9100 labels: origin: external relabel_configs: - source_labels: [__address__] target_label: instance regex: '([^:]+):\d+' replacement: '$1' EOF #把 Secret 挂到 Prometheus（prometheus-k8s） kubectl -n monitoring patch prometheus prometheus-k8s --type merge -p ' { "spec": { "additionalScrapeConfigs": { "name": "prometheus-additional-scrape-configs", "key": "additional-scrape-configs.yaml" } } }' #验证 kubectl -n monitoring port-forward svc/prometheus-k8s 9090:9090
istio是什么？Istio 是一种开源服务网格，可透明地分层到现有的分布式应用程序上。 Istio 的强大功能提供了一种统一且更高效的方式来保护、连接和监控服务。 Istio 是实现负载均衡、服务到服务身份验证和监控的途径 - 几乎无需更改服务代码。 功能： 使用双向 TLS 加密、强大的基于身份的身份验证和鉴权在集群中保护服务到服务通信 HTTP、gRPC、WebSocket 和 TCP 流量的自动负载均衡 使用丰富的路由规则、重试、故障转移和故障注入对流量行为进行细粒度控制 支持访问控制、限流和配额的可插入策略层和配置 API 集群内所有流量（包括集群入口和出口）的自动指标、日志和链路追踪 如何工作？ Istio 使用代理来拦截您的所有网络流量，从而根据您设置的配置允许使用一系列应用程序感知功能。 控制平面采用您所需的配置及其对服务的视图，并动态地编程代理服务器，并根据规则或环境的变化对其进行更新。 数据平面是服务之间的通信。如果没有服务网格，网络就无法理解正在发送的流量，也无法根据流量类型、流量来源或目的地做出任何决策。 Istio 支持两种数据平面模式： Sidecar 模式，它会与您在集群中启动的每个 Pod 一起部署一个 Envoy 代理，或者与在虚拟机上运行的服务一同运行。（下面使用的了sidecar模式） Ambient 模式，它使用每个节点的四层代理，并且可选地使用每个命名空间的 Envoy 代理来实现七层功能。 一、部署#官网 https://istio.io/#下载 https://github.com/istio/istio/releases/download/1.28.2/istio-1.28.2-linux-amd64.tar.gz #解压 tar -zxvf istio-1.28.2-linux-amd64.tar.gz #部署 root@k8s-01:/woke/istio# ls istio-1.28.2 istio-1.28.2-linux-amd64.tar.gz root@k8s-01:/woke/istio# cd istio-1.28.2/ root@k8s-01:/woke/istio/istio-1.28.2# export PATH=$PWD/bin:$PATH root@k8s-01:/woke/istio/istio-1.28.2# istioctl install -f samples/bookinfo/demo-profile-no-gateways.yaml -y |\ | \ | \ | \ /|| \ / || \ / || \ / || \ / || \ / || \ /______||__________\ ____________________ \__ _____/ \_____/ ✔ Istio core installed ⛵️ Processing resources for Istiod. Waiting for Deployment/istio-system/istiod ✔ Istiod installed 🧠 ✔ Installation complete root@k8s-01:/woke/istio/istio-1.28.2# kubectl label namespace default istio-injection=enabled namespace/default labeled root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# kubectl get crd gateways.gateway.networking.k8s.io NAME CREATED AT gateways.gateway.networking.k8s.io 2025-11-17T15:05:26Z root@k8s-01:/woke/istio/istio-1.28.2# kubectl get crd httproutes.gateway.networking.k8s.io NAME CREATED AT httproutes.gateway.networking.k8s.io 2025-11-17T15:05:26Z root@k8s-01:/woke/istio/istio-1.28.2# kubectl get crd gatewayclasses.gateway.networking.k8s.io NAME CREATED AT gatewayclasses.gateway.networking.k8s.io 2025-11-17T15:05:26Z root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# kubectl get crd | egrep 'tlsroutes|tcproutes|udproutes|grpcroutes\.gateway\.networking\.k8s\.io' grpcroutes.gateway.networking.k8s.io 2025-11-17T15:05:26Z root@k8s-01:/woke/istio/istio-1.28.2# kubectl exec "$(kubectl get pod -l app=ratings -o jsonpath='{.items[0].metadata.name}')" -c ratings -- curl -sS productpage:9080/productpage | grep -o " .* " Simple Bookstore App root@k8s-01:/woke/istio/istio-1.28.2# kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml gateway.networking.istio.io/bookinfo-gateway created virtualservice.networking.istio.io/bookinfo created root@k8s-01:/woke/istio/istio-1.28.2# kubectl get gateway NAME CLASS ADDRESS PROGRAMMED AGE traefik-gw traefik True 25d root@k8s-01:/woke/istio/istio-1.28.2# kubectl get gateway NAME CLASS ADDRESS PROGRAMMED AGE traefik-gw traefik True 25d root@k8s-01:/woke/istio/istio-1.28.2# kubectl get pod -A | grep gateway root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# kubectl get gateways.networking.istio.io -A kubectl get virtualservices.networking.istio.io -A NAMESPACE NAME AGE default bookinfo-gateway 2m55s NAMESPACE NAME GATEWAYS HOSTS AGE default bookinfo ["bookinfo-gateway"] ["*"] 2m55s root@k8s-01:/woke/istio/istio-1.28.2# kubectl get svc -n istio-system | egrep 'ingress|gateway' kubectl get pods -n istio-system | egrep 'ingress|gateway' root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# istioctl install -y --set profile=demo istioctl: command not found root@k8s-01:/woke/istio/istio-1.28.2# ls bin LICENSE manifests manifest.yaml README.md samples tools root@k8s-01:/woke/istio/istio-1.28.2# export PATH=$PWD/bin:$PATH istioctl version client version: 1.28.2 control plane version: 1.28.2 data plane version: 1.28.2 (6 proxies) root@k8s-01:/woke/istio/istio-1.28.2# istioctl install -y --set profile=demo |\ | \ | \ | \ /|| \ / || \ / || \ / || \ / || \ / || \ /______||__________\ ____________________ \__ _____/ \_____/ ✔ Istio core installed ⛵️ ✔ Istiod installed 🧠 ✔ Egress gateways installed 🛫 ✔ Ingress gateways installed 🛬 ✔ Installation complete root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# root@k8s-01:/woke/istio/istio-1.28.2# kubectl get pods -n istio-system | egrep 'istio-ingressgateway|istiod' kubectl get svc -n istio-system | egrep 'istio-ingressgateway' istio-ingressgateway-796f5cf647-n28c8 1/1 Running 0 107s istiod-5c84f8c79d-q7p2x 1/1 Running 0 91m istio-ingressgateway LoadBalancer 10.99.189.246 <pending> 15021:31689/TCP,80:32241/TCP,443:30394/TCP,31400:31664/TCP,15443:32466/TCP 107s #调通 root@k8s-01:/woke/istio/istio-1.28.2# kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80 Forwarding from 127.0.0.1:8080 -> 8080 Forwarding from [::1]:8080 -> 8080 Handling connection for 8080 #访问 root@k8s-01:/woke/istio/istio-1.28.2# curl -sS http://127.0.0.1:8080/productpage | grep -o " .* " Simple Bookstore App 1）更细的发布策略（不止权重灰度） 基于请求特征的灰度（更精准） 按 Header（用户/设备/地区/灰度标记） 按 Cookie（特定用户群） 按 URI（某些路径走新版本） 例子：带 x-canary: true 的请求走 v2，其余走 v1。 镜像流量（Shadow / Mirroring） 真实流量仍走 v1 同时“复制一份”给 v2 做压测/验证（不影响线上返回） 适合：验证新版本行为、性能。 2）限流与保护（Resilience） 超时 / 重试 / 连接池 timeout: 2s retries: attempts: 3 连接池限制（避免下游被打爆） 熔断 / 异常剔除（Outlier Detection） 连续 5xx 多就把某个 pod/实例踢出一段时间 故障注入（Chaos Testing） 人为注入延迟/错误（只对某类请求），验证系统韧性 3）流量控制与安全 mTLS 加密（服务到服务） 自动给网格内东西向流量加密 可做 STRICT/ PERMISSIVE 模式 配 PeerAuthentication / DestinationRule 细粒度访问控制（RBAC） 谁可以访问谁、什么路径、什么方法 支持基于 JWT claim、source principal、namespace、ip 等条件 用 AuthorizationPolicy JWT/OIDC 校验（API 鉴权） 在网关或服务侧校验 token（不必应用自己写） 用 RequestAuthentication + AuthorizationPolicy 4）入口网关能力（North-South） 多域名/多证书（HTTPS/TLS） Gateway 配多 SNI、证书、重定向 HTTP→HTTPS 路径重写/跳转/多服务路由 /api 去 A 服务、/web 去 B 服务 rewrite /v1 → / 等 速率限制（通常需配合 EnvoyFilter 或外部组件） 入口处做全局/按 key（IP、用户）限流 （Istio 原生 CRD 的“全局限流”需要看你用的扩展方案，常见是 Envoy ext-authz / ratelimit service） 5）可观测性（你在 Kiali/Prometheus 上看到的背后能力） 指标：QPS、P99、错误率、TCP 指标 分布式追踪：Jaeger/Zipkin/Tempo（需要装 tracing） 访问日志：Envoy access log（可输出到 stdout 或日志系统） 拓扑：Kiali 图里展示服务调用关系 6）多集群/多网格（更高级） 多集群同网格 跨集群服务发现与 mTLS East-West Gateway 给你一个“从 Bookinfo 出发最值得尝试”的 6 个玩法（上手快） Header 灰度：指定用户走 reviews v3 流量镜像：把部分流量镜像到 v3（不影响返回） 超时+重试：让 productpage 调 reviews 更稳定 熔断：reviews 出现错误时自动隔离坏实例 故障注入：只对某路径注入 2s 延迟测试体验 mTLS STRICT + 授权策略：禁止网格外/未授权服务访问 reviews
调用 grafana 接口失败 error trying to reach service: dial tcp 10.244.2.60:3000: i/o timeout① 背景（30 秒） “有一次我们在 Kubernetes 集群里部署监控体系（Prometheus + Grafana），Grafana Pod 本身是 Running 的，但从节点或者其他组件访问 Grafana 的 Pod IP 的时候一直超时，curl 会直接 i/o timeout，影响监控平台的接入和联调。” “表面看 Grafana 是正常的，但网络层面存在异常。”② 问题拆解与排查思路（1 分钟） “我当时按分层排查的思路来做，而不是直接改网络策略。” 1️⃣ 先排除应用问题 我先 kubectl exec 进 Grafana Pod，直接 curl 127.0.0.1:3000/api/health 返回 200，说明 Grafana 本身没问题，端口监听正常 👉 这一步可以明确：不是应用、不是容器启动问题。 2️⃣ 再定位是 Service 层还是 Pod 网络问题 我确认 Service / Endpoint 都指向正确的 Pod IP 但从节点直接 curl PodIP 仍然超时 在同节点起一个临时 curl Pod 访问 PodIP，也同样超时 👉 这一步可以确定： 不是宿主机路由问题，而是 Pod 网络路径被拦截。 3️⃣ 确认 CNI 和隧道是否正常 集群使用的是 Cilium + VXLAN 我抓包看到 8472/UDP 的 VXLAN 流量是存在的 说明 overlay 隧道本身工作正常，否则整个集群通信都会异常 👉 到这里，网络连通性问题基本被排除。③ 关键突破点（40 秒） “真正的突破点是在 Cilium 的可观测工具 上。” 我在节点上执行了： cilium monitor --type drop 发现每一次访问 Grafana 3000 端口的 TCP SYN 都被丢弃 丢包原因非常明确：Policy denied 源身份显示为 remote-node 或 host 👉 这一步明确告诉我： 不是网络问题，而是被 Cilium 的策略明确拒绝了。④ 根因分析（30 秒） “回头检查发现，Grafana 所在命名空间里存在默认的 NetworkPolicy，只允许 Prometheus Pod 访问 Grafana 的 3000 端口。” 节点 / 宿主机访问 Grafana 时： 在 Cilium 里会被标记为 host 或 remote-node 并不属于普通 Pod identity 所以即使我用 Kubernetes NetworkPolicy 加了 ipBlock，依然无法命中这类流量 👉 本质原因是： Cilium 是以 identity/entity 为核心做策略决策，而不是单纯基于 IP。⑤ 解决方案（30 秒） “最终我使用的是 CiliumNetworkPolicy，显式放行来自节点的流量。” 在策略里允许： fromEntities: host fromEntities: remote-node 只放行到 Grafana 的 3000/TCP 应用策略后，从节点 curl PodIP 立刻恢复正常。⑥ 总结与价值（20 秒） “这个问题让我印象很深的一点是： 在使用 Cilium 这类 eBPF CNI 时，排查网络问题一定要用它自己的观测工具，而不是只停留在 Kubernetes NetworkPolicy 的抽象层。” “后续我也把这个问题整理成了 SOP，明确： Pod→Pod Node→Pod Service→Pod 在 Cilium 下对应的 identity 和策略差异，避免再踩坑。”面试官常见追问 & 标准答法（加分项） ❓追问 1：为什么 ipBlock 不生效？ 答： “因为在 Cilium 里，Node / Host 流量优先按 identity（host / remote-node）匹配策略，而不是单纯按源 IP。 NetworkPolicy 的 ipBlock 在这类流量场景下不会命中，所以需要用 CiliumNetworkPolicy 的 fromEntities。” ❓追问 2：为什么不用 NodePort / LoadBalancer？ 答： “NodePort 确实能绕开 Pod 网络策略，但会引入额外暴露面，不符合最小权限原则。 这里的目标只是节点内部访问 Pod 服务，用 CNI 级别策略精确放行更安全、更可控。” ❓追问 3：如何避免以后再发生？ 答： 明确区分 Pod 流量 vs Node/Host 流量 在启用 Cilium 的集群里： 默认 NetworkPolicy + 例外的 CiliumNetworkPolicy 遇到网络问题第一时间用 cilium monitor / hubble observe 定位丢包原因，而不是盲改规则 🧠 面试官听到的“隐藏信号” 你这段回答会传递出几个非常强的信号： 你理解 CNI 底层实现，不只是会写 YAML 你会用 可观测工具定位问题 你做的是 体系化排障，而不是拍脑袋改配置 你有 复盘和沉淀能力 如果你愿意，下一步我还能帮你： 把这个案例改成 简历上的一条高质量项目经历 或模拟一次面试官“深挖追问”的对话，帮你把回答打磨到非常顺 你这个案例，放在 K8s 运维 / 平台工程 / SRE 面试里，是妥妥的加分项。root@k8s-01:~# cat > /tmp/cnp-allow-grafana-3000-from-nodes.yaml <<'EOF' apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-grafana-3000-from-nodes namespace: kuboard spec: endpointSelector: matchLabels: app.kubernetes.io/name: grafana ingress: - fromEntities: - host - remote-node toPorts: - ports: - port: "3000" protocol: TCP EOF kubectl apply -f /tmp/cnp-allow-grafana-3000-from-nodes.yaml kubectl -n kuboard get cnp | grep allow-grafana-3000-from-nodes ciliumnetworkpolicy.cilium.io/allow-grafana-3000-from-nodes created allow-grafana-3000-from-nodes 0s True root@k8s-01:~# curl -sS -m 3 http://10.244.2.60:3000/api/health { "database": "ok", "version": "11.2.0", "commit": "2a88694fd3ced0335bf3726cc5d0adc2d1858855" }root@k8s-01:~# 可以了 你现在是一个资深的k8s运维工程师 你帮我详细复盘分析之前的上下文 然后总结根因是什么？怎么解决的？为什么会出现这种情况 然后我想把这个问题也在面试上比如面试官问我 工作当中遇到了什么问题 可以说这个 我是怎么遇到这个问题的 过程尝试了用什么方法解决 然后最后怎么找到问题的 怎么解决的 下面我按“资深 K8s 运维复盘”的方式，把你这次 Grafana Pod IP 不通的问题完整复盘一遍：现象 → 排查路径 → 关键证据 → 根因 → 解决方案 → 为什么会这样。最后再给你一段面试可直接复述的 STAR/故事版本。 1）现象是什么 Pod 里 Grafana 本身是好的： kubectl exec grafana -- curl 127.0.0.1:3000/api/health 返回 OK 日志正常、监听正常（Grafana 启动成功） 但是从 宿主机 / 节点 访问 Pod IP 超时： curl http://10.244.2.60:3000/api/health → i/o timeout 从 同节点的测试 Pod 访问 Pod IP 也超时： 你在 k8s-01 强制调度了 curlimages/curl 去 curl PodIP，仍然 timeout 这说明：不是“宿主机路由不通”，而是“到 Pod 的流量在某处被拦截/丢弃” 2）你做了哪些正确的排查动作（过程复盘） A. 先确认应用是否健康（排除应用问题） 你先在 grafana 容器内 curl 自己： 容器内 OK → Grafana 服务与端口没问题 这一步很关键：把“应用/容器问题”先排掉。 B. 再验证网络路径（区分 host、pod、service） kubectl get svc/ep 确认 endpoint 就是 10.244.2.60:3000 curl PodIP 超时，说明不是 Service 负载均衡层的问题，而是到 endpoint 的流量被拦截 C. 检查 CNI 模式与隧道 你确认了 Cilium 配置： routing-mode: tunnel tunnel-protocol: vxlan 并且抓包能看到 8472/UDP 的 VXLAN 流量 进出（说明隧道本身在工作） 这一点也非常关键：overlay 通道是通的，否则集群内大量 Pod 通信都会异常。 D. 最关键的一步：用 Cilium 观测“丢包原因” 你执行： cilium monitor --type drop 立刻得到核心证据： xx drop (Policy denied) ... identity 6820->29952: 10.244.0.206 -> 10.244.2.60:3000 tcp SYN 以及后面： xx drop (Policy denied) ... identity remote-node->29952: 10.244.0.18 -> 10.244.2.60:3000 tcp SYN 这两条信息极其重要，等价于告诉你： 不是路由问题 不是端口没监听 是 Cilium 的策略（Policy）明确拒绝了 SYN 到这一步，根因已经锁定在 NetworkPolicy / CiliumPolicy 方向。 3）根因是什么（总结一句话版） Grafana 所在命名空间存在默认更严格的 NetworkPolicy（只允许 Prometheus Pod 访问 Grafana:3000），而节点/宿主机流量在 Cilium 里被标记成 remote-node/host 身份，不会被普通 K8s NetworkPolicy 的 ipBlock 命中，最终被 Cilium 以 Policy denied 丢弃。 再拆开说得更专业一点： 你原来的 NetworkPolicy（kuboard/grafana）是这样： ingress: - from: - podSelector: matchLabels: app.kubernetes.io/name: prometheus ports: - port: 3000 含义是： 只允许 prometheus 的 Pod 去访问 grafana 的 3000 其他来源（包括你从节点直接 curl PodIP）默认拒绝 为什么你加了 K8s NetworkPolicy（ipBlock）也没效果？ 因为你的访问路径是“节点/宿主机 → Pod”： 从节点发起的流量到了 Cilium，会以 entity identity 来归类： host（本节点宿主机） remote-node（其他节点过来的 node traffic） 这类流量不是普通 Pod identity，很多情况下也不是单纯按源 IP去匹配 NetworkPolicy 的 ipBlock（尤其在 Cilium 的实现/模式下，优先按 identity 进行策略决策）。 所以你看到的现象就是： 你用 ipBlock: 192.168.1.0/24 放行 —— 依然被 remote-node identity 拒绝 你用 ipBlock: 10.244.0.0/16 放行 —— 依然被 remote-node identity 拒绝 而 cilium monitor 也明确显示拒绝原因是：Policy denied + identity remote-node->... 4）怎么解决的（你最终的正确做法） 你最终用了 CiliumNetworkPolicy： apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy spec: endpointSelector: matchLabels: app.kubernetes.io/name: grafana ingress: - fromEntities: - host - remote-node toPorts: - ports: - port: "3000" protocol: TCP 这条规则的效果是： 明确放行 Cilium 的 entity host：本机宿主机进来的流量 remote-node：其他节点过来的 node 流量 只放行到 grafana pod 的 3000/TCP 所以你立刻在 k8s-01 上 curl PodIP 成功 5）为什么会出现这种情况（更“体系化”的解释） 1）装监控栈/平台时经常自带 NetworkPolicy 很多 kube-prometheus-stack / 平台化套件会默认创建 NetworkPolicy，实现“默认拒绝 + 白名单放行”，比如： Prometheus 可 scrape 其他来源默认不允许访问 UI/端口 这属于“安全默认值”，但会让排障时出现“明明 Pod Running，却访问超时”。 2）Cilium 对 Node/Host 流量有自己的 identity 体系 Cilium 的策略引擎不是只看 IP，而是以 identity/entity 为第一等公民： Pod 有 identity（基于 labels） Node/Host/Remote-node 是 entity 所以要放通 node→pod，最稳的是用 fromEntities: host/remote-node 这也是你最终方案成功的根本原因。 3）“Pod->Pod 通，Node->Pod 不通”是典型特征 因为： Pod→Pod 可能被某些 policy 放行了（或在同 namespace 里满足 selector） Node→Pod 不属于 podSelector 能匹配的对象，常被默认 deny 6）面试：如何把这件事讲成一个漂亮的“排障故事” 下面给你一个可直接背的版本（STAR + 技术细节），你按自己实际环境改两句就能用。 面试复述版本（建议 2~3 分钟） 背景（S） “我们在 K8s 集群里部署监控（Grafana/Prometheus），Grafana Pod Running，但业务方/运维侧从节点访问 Grafana API 一直 i/o timeout，影响监控平台接入和联调。” 任务（T） “我需要快速确认是应用问题、服务暴露问题还是集群网络策略问题，并给出可控的修复方案。” 行动（A） 先排除应用故障：进入 Grafana Pod 内部 curl 127.0.0.1:3000/api/health，确认服务健康、端口正常监听。 定位网络层级：从节点 curl PodIP 超时，同时在同节点跑一个临时 curl Pod 访问 PodIP 也超时，说明不是宿主机网络工具问题，而是集群网络路径被拦。 检查 CNI 与隧道：确认集群使用 Cilium，VXLAN 隧道 8472/UDP 有流量，说明 overlay 本身工作正常。 用可观测手段找“丢包理由”：运行 cilium monitor --type drop，发现对 Grafana:3000 的 SYN 被明确标记为 Policy denied，源身份显示为 remote-node/host。 回查 NetworkPolicy：发现 Grafana 命名空间有默认 NetworkPolicy，只允许 Prometheus Pod 访问 Grafana 3000，节点/宿主机流量不在白名单内，因此被 Cilium 拒绝。 修复策略：使用 CiliumNetworkPolicy 放行 fromEntities: host, remote-node 到 Grafana 的 3000/TCP，修复后节点 curl PodIP 立刻恢复。 结果（R） “Grafana API 访问恢复，监控平台联调继续推进。过程中我把排查步骤固化成 SOP：先应用健康检查→再分层定位→最后用 CNI 可观测工具确认策略/丢包原因，避免盲目改防火墙或路由。” 补充亮点（技术总结） “这类问题的关键点在于：Cilium 以 identity/entity 为核心做策略决策，Node/Host 流量常显示为 remote-node/host，单纯用 K8s NetworkPolicy 的 ipBlock 不一定能匹配到，需要用 CiliumNetworkPolicy 的 fromEntities 精准放行。”你是一个资深的k8s云原生运维工程师 你觉得碰到的这个问题有深度吗？ 有，而且深度不在“Grafana 访问不通”这件事本身，而在你最后定位到的点：Cilium（eBPF）语义下，Host/Node 流量与 Pod 流量的身份模型差异，这已经超出“会用 K8s”的层级了。 我用面试官视角给你拆一下它为什么算“有深度”。 为什么这个问题有深度 1) 不是“网络不通”，是“策略语义不一致” 你一开始看到的是 timeout，很容易被带偏去查： VXLAN 8472/UDP、防火墙、路由、MTU、kube-proxy、Service/Endpoint 但最终根因是： NetworkPolicy 只放了 Prometheus 节点访问在 Cilium 里变成 host/remote-node identity 所以被 Policy denied 掉了 这属于 “抽象层（K8s NP） vs 实现层（Cilium identity/entity）”的差异问题，很典型也很难踩得准。 2) 你用的是“正确的高级排障工具链” 真正有经验的人会很快转向： cilium monitor --type drop / hubble 而不是一直“猜”。 你抓到的那句： drop (Policy denied) ... identity remote-node->... tcp SYN 这就是一锤定音的证据链。很多人排障卡住就是因为拿不到“谁丢的、为什么丢”。 3) 这是“集群行为”层面的知识点，不是某个组件 bug 你解决的不是 Grafana，也不是 curl，而是： CNI datapath identity-based enforcement node/host traffic classification NetworkPolicy 与 CiliumNetworkPolicy 的边界 这类问题在生产里经常发生在： metrics/health check 从节点探测 Pod ingress controller / node-local agent 访问业务 Pod 节点上跑的守护进程（daemonset 外部探测）访问 Pod 属于“平台层运维”常见痛点。 面试视角：它值不值得讲？ 值得，尤其适合这些岗位： Kubernetes 运维 / 平台工程 / 云原生 SRE 网络/安全/可观测性方向 用 Cilium、Calico、Istio 的团队 但要注意一点： 你讲的重点一定要是“如何定位到 Policy denied + identity”，而不是“我改了很多 yaml”。 这题在“深度等级”大概在哪？ 我会给它一个比较客观的分级： 初级：只会说“加个 NetworkPolicy 放行” 中级：知道分层排查（Pod 内 curl、Service、路由） 高级：能用 Cilium/Hubble 看到 Policy denied 并解释 host/remote-node identity 更高级：能进一步讲“为什么 ipBlock 不命中、为什么需要 fromEntities、如何标准化策略/避免误伤” 你这次已经到 高级门槛了。 如果你愿意，我还可以帮你把它再“拔高一档”——补上两句很能镇住面试官的话，比如： “我们在 Cilium 下把 Node->Pod 探测流量单独建了 fromEntities: host/remote-node 的 allowlist，避免跟业务 NP 混在一起。” “排查网络我会先看 datapath drop reason，避免在 overlay、iptables、policy 之间盲猜。” 你想把这题定位成你“最有含金量的一个案例”吗？我可以顺手帮你润色成简历 bullet。
一、cilium介绍Cilium 本质上是一个 Kubernetes 的网络与安全组件（通常以 CNI 插件的身份安装），它用 eBPF 在 Linux 内核里实现：Pod/Service 的转发、网络策略（NetworkPolicy）、可观测性（流量可视化/追踪）、负载均衡，甚至部分服务网格能力。 可以把它理解成：“把 Kubernetes 网络、负载均衡和安全策略，尽量下沉到内核里做”，相比传统 iptables 方案更高性能、更可观测、策略能力更强。二、cilium是什么在 K8s 里，想让 Pod 能互通、能访问 Service、能做 NetworkPolicy，需要一个网络方案。Cilium 通常承担这些职责： 1. CNI（容器网络接口）：给 Pod 分配 IP、配置路由/隧道/网卡，把 Pod 接入集群网络 2. Service 负载均衡实现：让访问 ClusterIP/NodePort/LoadBalancer 服务能正确转发到后端 Pod 3. 网络安全策略：实现 Kubernetes NetworkPolicy，甚至更强的 L3/L4/L7（HTTP/gRPC/DNS）策略 4. 网络可观测性：看到“谁在跟谁说话、延迟多少、丢包在哪里、命中了哪些策略” 5. （可选）替代 kube-proxy：用 eBPF 实现 Service 转发（kube-proxy replacement）三、cilium有什么用（解决了哪些痛点）A. 更快、更稳的转发与负载均衡 传统 kube-proxy 依赖 iptables/ipvs，规则多了会膨胀、调试困难。Cilium 用 eBPF 做转发和 LB： 1. 内核态快速转发（少走用户态/少遍历规则） 2. 大规模 Service/Endpoint 场景下更稳定 3. 更容易观测“为什么这次转发去了哪个后端、命中什么规则” B. 更强的安全能力（不仅是“允许/拒绝”） K8s 原生 NetworkPolicy 更多是 L3/L4（IP/端口）。Cilium 能做到更细： 1. 按 Pod 标签、命名空间、服务账户 做策略（身份感更强） 2. L7 策略（可选，通常依赖 Envoy）： 3. 允许某些 HTTP Path / Method 4. gRPC 服务/方法级别控制 5. DNS 解析策略（允许访问哪些域名） C. 可观测性（排障神器） Cilium 自带/配套 Hubble（可选）可以做到： 1. 实时看流量（flow logs）：谁访问了谁、是允许还是被拒绝、RTT、丢包 2. 快速定位：是 DNS 问题？策略拦了？还是 Service LB 选后端异常？ D. 可选的高级能力（看你是否启用） 不同环境会选用其中一部分能力： 1. 透明加密（节点间/Pod 间加密） 2. BGP/路由通告（部分场景下替代或配合传统 LB） 3. Gateway API / Ingress（通过 cilium-envoy） 4. 多集群互联（ClusterMesh）三、cilium是怎么工作的（核心：eBPF 数据面）关键点：eBPF eBPF 是 Linux 内核机制，允许把程序挂到内核的网络钩子上（如 tc、XDP、cgroup），从而： 1. 在包进入/离开网卡、进入容器、进入 socket 前后做处理 2. 实现 NAT、负载均衡、策略过滤、可观测统计 3. 尽量减少 iptables 链式规则带来的复杂性 Cilium 常见组件（你 kubectl get pod -n kube-system 看到的那些） 1. cilium (DaemonSet)：每个节点一个 agent，负责该节点上的转发/策略/eBPF 程序装载 2. cilium-operator (Deployment)：做控制面工作（例如管理 IPAM、节点资源、状态同步等） 3. cilium-envoy (DaemonSet，可选)：用于 L7 能力/Gateway/Ingress 等（不是所有场景都必须）四、IPAM 是什么IPAM = IP Address Management，就是“Pod 的 IP 从哪里来、怎么分配、怎么回收”。 Cilium 支持多种 IPAM 模式（不同环境选不同方式），常见两类理解： 1. kubernetes 模式：Pod IP 通常来自每个节点的 PodCIDR（例如你看到的 10.244.x.x） 2. cluster-pool / 等模式：由 Cilium 自己维护一个地址池分配（适合一些特定架构或需要 Cilium 管理地址的场景） 遇到的现象（Helm 改了 ipam.mode=kubernetes 但不重启不生效）很符合实际（后面案例）： ConfigMap 改了 ≠ agent 自动 reload，Cilium agent 还是用旧配置跑着，所以要 rollout restart 让它重新读取配置并重新装载相关 datapath这两个模式本质上都在做一件事：给每个节点准备一段“Pod 子网（PodCIDR）”，然后由该节点上的 cilium-agent 在这段子网里给 Pod 分 IP。真正的核心区别只有一个： “每个节点的 PodCIDR 由谁来分配/作为事实来源（Source of Truth）？ - cluster-pool / cluster-scope（默认）：PodCIDR 由 Cilium Operator 分配，写到 v2.CiliumNode 里。 - kubernetes（Kubernetes Host Scope）：PodCIDR 由 Kubernetes 分配，写到 v1.Node.spec.podCIDR(s) 里（依赖 kube-controller-manager 分配 Node CIDR）。cluster-pool / cluster-scope（Cilium 管 PodCIDR） 机制: - Cilium Operator 从你配置的 clusterPoolIPv4PodCIDRList 里切块，给每个节点分一个 PodCIDR，记录在 CiliumNode。 - 节点上的 cilium-agent 再从该 PodCIDR 里发放 Pod IP。 优点: - 不依赖 Kubernetes 给 Node 下发 PodCIDR：很多环境（尤其是你没法改 controller-manager 参数、或 K8s 不分配 PodCIDR 的场景）会更省事。 - 集群地址池扩容相对直观：池不够了可以往 clusterPoolIPv4PodCIDRList 追加新的 CIDR（注意是“追加”，不要改已有元素）。 - 支持 “集群多 CIDR”（多段池）——官方 IPAM 能力表里，cluster-scope 这一项是 缺点/坑点: - 和“标准 Kubernetes 语义”有点脱钩：因为 Node 的 spec.podCIDR 不是事实来源（可能为空或不一致），某些依赖 Node PodCIDR 的外部组件/脚本可能不适配（例如你自己写的自动化、某些网络周边控制器）。 - 默认池容易和节点网段冲突：文档明确提醒默认 10.0.0.0/8 如果跟节点网络重叠会直接炸（跨节点通信会被误判成 Pod 流量）。 - 控制面多了一个 CiliumNode（CRD）对象：排障时要懂它（但对运维来说也不算坏事）。ipam.mode=kubernetes（Kubernetes Host Scope，K8s 管 PodCIDR） 机制: - Cilium 从 Kubernetes Node 上读 spec.podCIDR(s)，在该范围内分配 Pod IP。 - 文档强调：需要 kube-controller-manager 开启 --allocate-node-cidrs 才能给节点分 PodCIDR。 优点: - 最“原生 Kubernetes”：Node 的 spec.podCIDR 就是标准字段，很多周边组件、工具链、排障习惯都默认看它。 - 和集群初始化参数（如 kubeadm 的 --pod-network-cidr）天然一致：你想让 Pod 都落在 10.244.0.0/16 这种规划网段时更符合直觉。 - 配合云厂商/托管集群时，若平台本身就按 Node 分配 PodCIDR，这个模式更顺滑。 缺点: - 依赖 Kubernetes 能正确分配 Node PodCIDR：如果你的集群没开 allocate-node-cidrs、或托管环境不给 PodCIDR，这个模式会卡 Pod 分配/导致 Pod 起不来。 - “集群多 CIDR”不支持（能力表里 Kubernetes Host Scope 这项是 ❌）。 - 想扩容地址空间通常会牵涉更大范围的 K8s 网络规划调整（成本更高）。选哪个？ 你能控制/确认 Kubernetes 会给 Node 分 PodCIDR（kubeadm 常见） → 优先选 ipam.mode=kubernetes：标准、兼容性好、排障直观。 你没法/不方便让 Kubernetes 分 PodCIDR，或希望 Cilium 自己掌控地址池（并可能需要多段 CIDR） → 选 cluster-pool/cluster-scope。Cilium 官方明确建议：不要在“存量集群”里随意切换 IPAM 模式，可能造成持续的网络中断风险；最安全是新集群按目标 IPAM 重装。你这次切换后通过重启恢复正常，说明你场景比较“干净”，但生产环境还是要非常谨慎。五、什么时候适合用 Cilium很适合这些场景： 1. 集群规模比较大、Service/Endpoint 多，追求性能和稳定性 2. 对 NetworkPolicy 有较强需求，想要更细粒度的安全控制 3. 经常排查网络问题，希望“看得见流量” 4. 想减少 kube-proxy/iptables 的复杂度 不一定要上满功能（比如 L7、Gateway、加密、BGP），很多团队只用它当高性能 CNI + NetworkPolicy + 可观测性，就已经很值了。六、ubuntu 22.04 部署Cilium前置条件（每个节点都要满足） Linux kernel ≥ 5.10（Ubuntu 22.04 默认一般是 5.15，通常没问题）。 uname -r6.1 方案 A：用 cilium-cli 快速安装（最省事）（我使用的是方案B）1) 安装 cilium-cli（在有 kubectl 的那台机器上） 按官方 quick install 文档的 Linux 安装方式： CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) CLI_ARCH=amd64 if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi curl -L --fail --remote-name-all \ https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum} 2) 安装 Cilium（作为 CNI） 官方示例（以文档当时的 stable 版本为例 1.18.5）： cilium install --version 1.18.53) 验证 cilium status --wait cilium connectivity test6.2 方案 B：用 Helm 安装（方便走 GitOps/精细配置）官方 Helm 安装入口： 1) 添加 Helm 仓库并安装 helm repo add cilium https://helm.cilium.io/ helm repo update helm install cilium cilium/cilium --version 1.18.5 --namespace kube-system2) 验证 cilium status --wait cilium connectivity test # 安装前 root@k8s-master-01:~# ls -l /etc/cni/net.d/ total 8 -rw-r--r-- 1 root root 664 Dec 23 01:17 10-calico.conflist -rw------- 1 root root 2796 Dec 23 01:17 calico-kubeconfig # 安装后 会把calico重命名 默认使用的cilium root@k8s-master-02:~# ls -l /etc/cni/net.d/ total 12 -rw------- 1 root root 191 Dec 23 02:01 05-cilium.conflist -rw-r--r-- 1 root root 664 Dec 23 01:17 10-calico.conflist.cilium_bak -rw------- 1 root root 2796 Dec 23 01:17 calico-kubeconfig 6.3 此时集群有两个cni插件Calico 没删时，Cilium + Calico 会不会冲突？ 默认会（除非你明确配置了 CNI chaining 让两者协同）。常见情况是： 1. 现有 Pod：基本还是走 Calico（它们创建时用的就是 Calico 的 CNI），短时间内通常还能通信。 2. 新建 Pod：如果 Cilium 已经把自己的 CNI 配置写进了 /etc/cni/net.d/，而且文件排序优先级比 Calico 高，那么新 Pod 可能会用 Cilium。这样集群就会出现“一部分 Pod 用 Calico，一部分用 Cilium”的混用状态——这时 Pod-Pod 通信非常容易出问题（跨节点路由/overlay/iptables/BPF 规则不一致）。 3. 另外，两套 agent 都在节点上跑，也可能同时改动路由、iptables、BPF 挂载/系统参数，带来不可预期的问题。 需要提前做好预案 删除calico的同时需要把所有的控制器资源如deploy 同时rollout restart 把cni切换成cilium否则就会出现上面说的情况6.4 升级切换# 这个看你需求上面有说 要说的一点是 改 ConfigMap 之后，Cilium agent / operator 通常需要重启才能读取新配置并重建 datapath。 root@k8s-master-01:~# helm upgrade cilium cilium/cilium \ -n kube-system \ --version 1.18.5 \ --reuse-values \ --set ipam.mode=kubernetes Release "cilium" has been upgraded. Happy Helming! NAME: cilium LAST DEPLOYED: Tue Dec 23 02:28:01 2025 NAMESPACE: kube-system STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: You have successfully installed Cilium with Hubble. Your release version is 1.18.5. For any further help, visit https://docs.cilium.io/en/v1.18/gettinghelp kubectl -n kube-system rollout restart ds/cilium kubectl -n kube-system rollout restart ds/cilium-envoy kubectl -n kube-system rollout restart deploy/cilium-operator# 查看 给每个node节点分配的IP段 IPAM root@k8s-master-01:~# kubectl get node -o custom-columns=NAME:.metadata.name,PODCIDR:.spec.podCIDR,PODCIDRS:.spec.podCIDRs NAME PODCIDR PODCIDRS k8s-master-01 10.244.0.0/24 [10.244.0.0/24] k8s-master-02 10.244.1.0/24 [10.244.1.0/24] k8s-node-03 10.244.2.0/24 [10.244.2.0/24] k8s-node-04 10.244.3.0/24 [10.244.3.0/24] #为什么会有PODCIDRS和PODCIDR 这是因为k8s后面支持IPV4和IPV6 老的支持支单IP 新的支持数组 多IP配置root@k8s-master-01:~# kubectl get pod -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cilium-envoy-655sx 1/1 Running 0 6h50m 192.168.30.160 k8s-master-01 <none> <none> cilium-envoy-6f6jv 1/1 Running 0 6h50m 192.168.30.163 k8s-node-04 <none> <none> cilium-envoy-7w54q 1/1 Running 0 6h50m 192.168.30.161 k8s-master-02 <none> <none> cilium-envoy-fg7q6 1/1 Running 0 6h50m 192.168.30.162 k8s-node-03 <none> <none> cilium-fsq6s 1/1 Running 0 6h50m 192.168.30.161 k8s-master-02 <none> <none> cilium-g4c79 1/1 Running 0 6h50m 192.168.30.162 k8s-node-03 <none> <none> cilium-jxmh5 1/1 Running 0 6h50m 192.168.30.160 k8s-master-01 <none> <none> cilium-operator-84cd4f9cb8-nqtlp 1/1 Running 0 6h50m 192.168.30.161 k8s-master-02 <none> <none> cilium-operator-84cd4f9cb8-vtphd 1/1 Running 0 6h50m 192.168.30.160 k8s-master-01 <none> <none> cilium-w4jqv 1/1 Running 0 6h50m 192.168.30.163 k8s-node-04 <none> <none> coredns-6d58d46f65-2sbtg 1/1 Running 0 6h50m 10.244.2.180 k8s-node-03 <none> <none> coredns-6d58d46f65-z4l4l 1/1 Running 0 6h50m 10.244.3.158 k8s-node-04 <none> <none> etcd-k8s-master-01 1/1 Running 5 (8h ago) 2d4h 192.168.30.160 k8s-master-01 <none> <none> etcd-k8s-master-02 1/1 Running 1 (8h ago) 2d3h 192.168.30.161 k8s-master-02 <none> <none> kube-apiserver-k8s-master-01 1/1 Running 2 (8h ago) 2d4h 192.168.30.160 k8s-master-01 <none> <none> kube-apiserver-k8s-master-02 1/1 Running 1 (8h ago) 2d3h 192.168.30.161 k8s-master-02 <none> <none> kube-controller-manager-k8s-master-01 1/1 Running 7 (8h ago) 2d4h 192.168.30.160 k8s-master-01 <none> <none> kube-controller-manager-k8s-master-02 1/1 Running 1 (8h ago) 2d3h 192.168.30.161 k8s-master-02 <none> <none> kube-proxy-44cbs 1/1 Running 1 (8h ago) 2d4h 192.168.30.160 k8s-master-01 <none> <none> kube-proxy-bjjb2 1/1 Running 1 (8h ago) 2d3h 192.168.30.163 k8s-node-04 <none> <none> kube-proxy-bvptw 1/1 Running 1 (8h ago) 2d3h 192.168.30.161 k8s-master-02 <none> <none> kube-proxy-m5dt2 1/1 Running 1 (8h ago) 2d3h 192.168.30.162 k8s-node-03 <none> <none> kube-scheduler-k8s-master-01 1/1 Running 6 (8h ago) 2d4h 192.168.30.160 k8s-master-01 <none> <none> kube-scheduler-k8s-master-02 1/1 Running 2 (8h ago) 2d3h 192.168.30.161 k8s-master-02 <none> <none> 只要你的业务是多副本/有滚动策略，通常只是“抖一下”，不会全断；单副本服务会有短暂不可用，这是任何 CNI 迁移都绕不开的。 #部署前 root@k8s-master-01:~# ls -l /etc/cni/net.d/ total 8 -rw-r--r-- 1 root root 664 Dec 23 01:17 10-calico.conflist -rw------- 1 root root 2796 Dec 23 01:17 calico-kubeconfig #部署后 会把calico重命名 后续pod的创建会默认走cilium 所以cilium启动好后 需要把所有业务pod 都重启一遍 root@k8s-master-01:~# ls -l /etc/cni/net.d/ total 12 -rw------- 1 root root 191 Dec 23 02:03 05-cilium.conflist -rw-r--r-- 1 root root 664 Dec 23 01:17 10-calico.conflist.cilium_bak -rw------- 1 root root 2796 Dec 23 01:17 calico-kubeconfig七、Cilium组件 7.1 Cilium Agent（DaemonSet，每节点 1 个）这是 Cilium 的核心，每个节点都必须有。 主要功能： eBPF 数据面加载与维护 - 在节点内核里加载 eBPF 程序（TC/XDP、cgroup hooks 等），实现 Pod 之间转发、Service 负载均衡、网络策略等。 - 维护 Cilium 的 BPF maps（相当于内核里的高性能“路由/连接/策略表”）。 管理 Pod/Endpoint 生命周期 - 监听 K8s（Pod/Service/EndpointSlice/Node 等）变化，创建/更新 CiliumEndpoint 等资源。 - 负责把“这个 Pod 的 IP/身份/策略”下发到本节点的数据面。 Kubernetes Service 转发（L4） - 用 eBPF 做 Service 的四层负载均衡、DNAT/SNAT、session affinity 等（即使你没开 kube-proxy replacement，也会参与很多转发能力）。 NetworkPolicy 落地 - 处理 K8s NetworkPolicy / CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy，转换成 eBPF 可执行的规则。 可选能力 - Hubble（可观测）、加密（WireGuard/IPsec）、eBPF HostRouting、Egress Gateway、带宽管理等（取决于你装的时候开没开）。 一句话：cilium Pod 就是每个节点的“网络内核驱动 + 控制代理”。7.2 Envoy 代理（DaemonSet，每节点 1 个）这是 L7/Ingress/Gateway API 等功能的“七层代理执行器”。 Cilium 本身用 eBPF 擅长 L3/L4（IP/端口/连接层面的东西），但如果你要做这些就需要 Envoy： - L7 网络策略（HTTP/gRPC/Kafka 等按域名、路径、方法、Header 的策略） - Ingress / Gateway API（很多场景由 Envoy 承担实际转发） - TLS 终止/重加密、L7 可观测（看你启用的功能） 工作方式通常是： - cilium agent 用 eBPF 把某些流量“重定向”给本机的 cilium-envoy - Envoy 做完七层处理后再把流量送回去继续转发 一句话：cilium-envoy 是 Cilium 的 L7 扩展引擎。7.3 Cilium Operator（Deployment，集群级 1~2 个）这是 集群级控制面组件，不需要每节点一个。 它主要负责“跨节点/全局”的事情，比如： 1.IPAM/地址相关的集群级管理 - 你现在 ipam: kubernetes：PodCIDR 是 Kubernetes 控制器分给 Node 的，agent 按 Node 的 PodCIDR 分配 Pod IP。 - 即便如此，operator 仍会参与一些 Node/CRD 的维护工作（不同版本细节略有差异）。 2.LB-IPAM（你已经开启） - 你 config 里 enable-lb-ipam: "true"、default-lb-service-ipam: lbipam - 这意味着：当你创建 type: LoadBalancer 的 Service 时，Cilium 可以用 LB-IPAM 给它分配外部 IP（来自你配置的地址池/策略）。 - 这类“全局分配/回收”的事通常由 operator 负责协调，agent 负责在节点上落实转发。 3.管理/清理一些全局资源 - 例如身份（identity）相关、旧对象回收、状态一致性维护等（具体看你启用的功能和版本）。 一句话：cilium-operator 是 Cilium 的“集群级大管家”，负责全局资源协调。八、开启UI#最简单一把梭 helm upgrade cilium cilium/cilium -n kube-system --version 1.18.5 --reuse-values \ --set hubble.relay.enabled=true \ --set hubble.ui.enabled=true \ --set hubble.ui.service.type=NodePort 九、开启kube-proxy-free 代替集群原来的 kube-proxy用 kubeconfig 里的 server 作为 k8sServiceHost 在 master 上执行，看看你的 kubeconfig 当前指向哪个 apiserver（通常是 VIP/LB，强烈建议用 VIP/LB，不要写死单个 master IP）： root@k8s-master-01:~# kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo https://192.168.30.58:16443 helm upgrade cilium cilium/cilium -n kube-system --version 1.18.5 --reuse-values \ --set kubeProxyReplacement=true \ --set k8sServiceHost=192.168.30.58 \ --set k8sServicePort=6443 kubectl -n kube-system rollout restart ds/cilium kubectl -n kube-system rollout restart deploy/cilium-operator 验证是否开启 root@k8s-master-01:~# kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement KubeProxyReplacement: True [ens33 192.168.30.160 192.168.30.58 fe80::20c:29ff:fef6:2d (Direct Routing)]#备份 kubectl -n kube-system get ds kube-proxy -o yaml > kube-proxy.ds.yaml kubectl -n kube-system get cm kube-proxy -o yaml > kube-proxy.cm.yaml #删除 kube-proxy DaemonSet + ConfigMap（官方推荐连 CM 也删，避免 kubeadm 升级时装回来 kubectl -n kube-system delete ds kube-proxy kubectl -n kube-system delete cm kube-proxy #每个节点清理遗留 KUBE* iptables 规则（官方给的命令） 不能随便执行 如果你节点上面有特定的规则 还有就是执行后 ssh会连接不上 实在要执行先执行前面这两条 保证ssh和已建立连接永久放行 iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT iptables -I INPUT 2 -p tcp --dport 22 -j ACCEPT iptables-save | grep -v KUBE | iptables-restore十、开启Gateway api#安装 Gateway API 的 standard CRDs（否则 apiserver 里没有 Gateway/GatewayClass/HTTPRoute 这些资源） kubectl apply --server-side=true -f \ https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml #在 Cilium 里开启 Gateway API Controller helm upgrade cilium cilium/cilium -n kube-system --version 1.18.5 --reuse-values \ --set gatewayAPI.enabled=true#选择暴露方式（二选一） 有 LoadBalancer 能力（云 LB / MetalLB）：不用 hostNetwork（默认走 LB Service） 没有 LoadBalancer 能力（你这种裸金属常见）：开启 hostNetwork helm upgrade cilium cilium/cilium -n kube-system --version 1.18.5 --reuse-values \ --set gatewayAPI.enabled=true \ --set gatewayAPI.hostNetwork.enabled=true#案例 root@k8s-master-01:~# cat nginx.yaml apiVersion: v1 kind: Namespace metadata: name: webs --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx namespace: webs spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: registry.cn-guangzhou.aliyuncs.com/xingcangku/nginx:1.18 ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-svc namespace: webs spec: selector: app: nginx ports: - name: http port: 80 targetPort: 80 --- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: nginx-gw namespace: webs spec: gatewayClassName: cilium listeners: - name: http protocol: HTTP port: 8080 hostname: nginx.axzys.cn allowedRoutes: namespaces: from: Same --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: nginx-route namespace: webs spec: parentRefs: - name: nginx-gw sectionName: http hostnames: - nginx.axzys.cn rules: - matches: - path: type: PathPrefix value: / backendRefs: - name: nginx-svc port: 80 root@k8s-node-04:~# curl -H 'Host: nginx.axzys.cn' http://192.168.30.160:8080/ no healthy upstreamroot@k8s-node-04:~# curl -H 'Host: nginx.axzys.cn' http://192.168.30.160:8080/ no healthy upstreamrcurl -H 'Host: nginx.axzys.cn' http://192.168.30.160:8080/ | head30.160:8080/ | head % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 612 100 612 0 0 21315 0 --:--:-- --:--:-- --:--:-- 21857 Welcome to nginx! body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } 在浏览器中显示需要hosts文件中添加上