LVS 高可用集群监控体系搭建

一、监控体系架构（在 LVS-DR 高可用集群之上）

基础架构见《LVS_DR高可用集群实战》。本篇记录监控层如何用 Ansible 自动化搭建。

┌─────────────────────────────────────────┐ │ 监控机 prometheus01 (.91) │ │ Prometheus :9090 │ Grafana :3000 │ │ Alertmanager :9093 │ blackbox :9115 │ └───────────────────┬─────────────────────┘ │ 抓取 (scrape) ┌────────────┬───────────────┼──────────────┬──────────────┐ ▼ ▼ ▼ ▼ ▼ node-exporter blackbox探活 keepalived-exp blackbox探活 node-exporter 7台主机:9100 web/dns:9115 LVS×2:9165 LVS:22 (同上)

监控目标与采集方式：

二、Ansible 控制端与 Inventory

控制端：workstation (.21)

# /home/devops/ansible/ansible.cfg [defaults] remote_user=root inventory=/home/devops/ansible/inventory host_key_checking = False collections_path=/home/devops/ansible/mycollections roles_path=roles:/home/devops/ansible/roles

# /home/devops/ansible/inventory [web] web01 ansible_host=192.168.211.81 web02 ansible_host=192.168.211.82 ​ [dns] dns01 ansible_host=192.168.211.61 dns02 ansible_host=192.168.211.62 ​ [lvs] lb-master ansible_host=192.168.211.71 lb-backup ansible_host=192.168.211.72 ​ [nfs] nfs01 ansible_host=192.168.211.51 ​ [prometheus] prometheus01 ansible_host=192.168.211.91

软件包管理（本地存放，避免国内下载超时）

/home/devops/ansible/ ├── exporter/ │ └── node_exporter-1.11.1.linux-amd64.tar.gz ├── myapp/ │ ├── prometheus-3.5.3.linux-amd64.tar.gz │ └── grafana-10.4.13-1.x86_64.rpm ├── template/ │ ├── node_exporter.service.j2 │ ├── prometheus.service.j2 │ └── prometheus.conf.j2 ├── inventory ├── ansible.cfg ├── node_exporter.yml └── prometheus.yml

三、node-exporter 部署 Playbook（完整记录）

部署到 7 台主机，用stat+when实现幂等，handler实现配置变更自动重启。

# /home/devops/ansible/node_exporter.yml --- - name: node-exporter hosts: prometheus become: true tasks: - name: 创建监控用户 ansible.builtin.user: name: prometheus state: present ​ - name: 创建监控目录 ansible.builtin.file: path: /opt/monitor/ state: directory owner: prometheus mode: '2755' ​ - name: 检查是否已安装(目录是否存在) ansible.builtin.stat: path: /opt/monitor/node_exporter register: ne_dir ​ - name: 解压node-exporter到被控节点(仅首次) ansible.builtin.unarchive: src: /home/devops/ansible/exporter/node_exporter-1.11.1.linux-amd64.tar.gz dest: /opt/monitor/ when: not ne_dir.stat.exists ​ - name: 目录改名(幂等) ansible.builtin.command: cmd: mv /opt/monitor/node_exporter-1.11.1.linux-amd64 /opt/monitor/node_exporter creates: /opt/monitor/node_exporter ​ - name: 修改目录所有者为prometheus(递归) ansible.builtin.file: path: /opt/monitor/node_exporter owner: prometheus recurse: yes ​ - name: 使用模板创建服务文件 ansible.builtin.template: src: /home/devops/ansible/template/node_exporter.service.j2 dest: /usr/lib/systemd/system/node_exporter.service notify: reload systemd and restart node_exporter ​ - name: 启动node_exporter并设置开机启动 ansible.builtin.service: name: node_exporter.service state: started enabled: yes ​ handlers: - name: reload systemd and restart node_exporter ansible.builtin.systemd: name: node_exporter.service state: restarted daemon_reload: true

模板 node_exporter.service.j2：

[Unit] Description=node_exporter Documentation=https://prometheus.io/ After=network.target [Service] User=prometheus Group=prometheus ExecStart=/opt/monitor/node_exporter/node_exporter Restart=on-failure [Install] WantedBy=multi-user.target

幂等设计要点：stat判断二进制目录是否已存在 →when: not exists控制只解压一次；mv用creates保证已改名就跳过；模板变更通过notify触发 handler 重启。

四、Prometheus + Grafana 部署 Playbook（完整记录）

# /home/devops/ansible/prometheus.yml --- - name: prometheus hosts: prometheus01 become: true tasks: - name: 创建监控用户 ansible.builtin.user: name: prometheus state: present ​ - name: 创建监控目录 ansible.builtin.file: path: /opt/monitor/ state: directory owner: prometheus mode: '2755' ​ - name: 检查prometheus二进制是否存在(幂等) ansible.builtin.stat: path: /opt/monitor/prometheus/prometheus register: prom_bin ​ - name: 创建数据目录 ansible.builtin.file: path: /opt/monitor/prometheus/data state: directory owner: prometheus group: prometheus ​ - name: 解压prometheus到被控节点(仅首次) ansible.builtin.unarchive: src: /home/devops/ansible/myapp/prometheus-3.5.3.linux-amd64.tar.gz dest: /opt/monitor/prometheus extra_opts: ['--strip-components=1'] # 去掉顶层目录, 直接解到目标 when: not prom_bin.stat.exists ​ - name: 修改目录权限(递归) ansible.builtin.file: path: /opt/monitor/prometheus state: directory recurse: yes owner: prometheus group: prometheus ​ - name: 创建服务文件 ansible.builtin.template: src: /home/devops/ansible/template/prometheus.service.j2 dest: /usr/lib/systemd/system/prometheus.service notify: reload systemd and restart prometheus ​ - name: 创建prometheus配置文件(变更时重启) ansible.builtin.template: src: /home/devops/ansible/template/prometheus.conf.j2 dest: /opt/monitor/prometheus/prometheus.yml owner: prometheus group: prometheus notify: reload systemd and restart prometheus ​ - name: 启动prometheus ansible.builtin.service: name: prometheus.service state: started enabled: yes ​ handlers: - name: reload systemd and restart prometheus ansible.builtin.systemd: name: prometheus.service state: restarted daemon_reload: true ​ - name: grafana hosts: prometheus01 become: true tasks: - name: 复制grafana rpm到目标节点 ansible.builtin.copy: src: /home/devops/ansible/myapp/grafana-10.4.13-1.x86_64.rpm dest: /tmp/grafana.rpm ​ - name: 安装grafana ansible.builtin.dnf: name: /tmp/grafana.rpm state: present disable_gpg_check: yes # 本地rpm跳过GPG校验 ​ - name: 启动grafana并开机自启 ansible.builtin.service: name: grafana-server state: started enabled: yes

prometheus.service.j2（关键：ExecStart 单行，别拆行）：

[Unit] Description=Prometheus Server Documentation=https://prometheus.io/docs/introduction/overview/ After=network-online.target [Service] Type=simple User=prometheus Group=prometheus ExecStart=/opt/monitor/prometheus/prometheus --config.file=/opt/monitor/prometheus/prometheus.yml --storage.tsdb.path=/opt/monitor/prometheus/data --storage.tsdb.retention.time=60d --web.enable-lifecycle Restart=on-failure [Install] WantedBy=multi-user.target

五、Prometheus 抓取配置（含探活）

prometheus.conf.j2 初始版（node-exporter）：

global: scrape_interval: 15s evaluation_interval: 15s scrape_configs: - job_name: 'node-exporter' static_configs: - targets: ['192.168.211.71:9100'] labels: { instance: 'lb-master' } - targets: ['192.168.211.72:9100'] labels: { instance: 'lb-backup' } - targets: ['192.168.211.81:9100'] labels: { instance: 'web01' } - targets: ['192.168.211.82:9100'] labels: { instance: 'web02' } - targets: ['192.168.211.61:9100'] labels: { instance: 'dns01' } - targets: ['192.168.211.62:9100'] labels: { instance: 'dns02' } - targets: ['192.168.211.91:9100'] labels: { instance: 'prometheus' }

后续手工补充的探活 job（blackbox）：

- job_name: 'blackbox-web' metrics_path: /probe params: { module: [http_2xx] } static_configs: - targets: ['http://192.168.211.100/', 'http://192.168.211.81/', 'http://192.168.211.82/'] relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: 'blackbox-dns' metrics_path: /probe params: { module: [dns_udp] } static_configs: - targets: ['192.168.211.200:53', '192.168.211.61:53', '192.168.211.62:53'] # ⚠️ 注意: dns_udp 的 target 不要带 udp:// 前缀! relabel_configs: #blackbox-exporter的固定写法 - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: 'blackbox-tcp' metrics_path: /probe params: { module: [tcp_connect] } static_configs: - targets: ['192.168.211.71:22', '192.168.211.72:22'] # LVS探22不探80! relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: 127.0.0.1:9115 - job_name: 'keepalived-exporter' static_configs: - targets: ['192.168.211.71:9165'] labels: { instance: 'lb-master' } - targets: ['192.168.211.72:9165'] labels: { instance: 'lb-backup' }

六、blackbox-exporter 模块配置

# /opt/monitor/blackbox_exporter/blackbox.yml modules: http_2xx: prober: http http: preferred_ip_protocol: "ip4" tcp_connect: prober: tcp dns_udp: # 需要自己加 prober: dns dns: transport_protocol: "udp" query_name: "www.chengke.com" query_type: "A" preferred_ip_protocol: "ip4"

七、keepalived-exporter（LVS 主机 .71/.72）

# 下载 rpm (cafebazaar 有预编译包) curl -L -o keepalived-exporter.rpm \ https://github.com/cafebazaar/keepalived-exporter/releases/download/v1.7.1/keepalived-exporter_1.7.1_linux_amd64.rpm rpm -ivh keepalived-exporter.rpm # rpm 不自带 systemd 服务, 手动创建 cat > /usr/lib/systemd/system/keepalived-exporter.service <<'EOF' [Unit] Description=Keepalived Exporter After=network.target keepalived.service [Service] Type=simple User=root ExecStart=/usr/bin/keepalived-exporter --ka.json --web.listen-address=:9165 Restart=on-failure [Install] WantedBy=multi-user.target EOF systemctl enable --now keepalived-exporter

关键指标：

八、告警规则（rules 目录）

# /opt/monitor/prometheus/rules/keepalived_alerts.yml groups: - name: keepalived-alerts rules: - alert: KeepalivedDown expr: keepalived_up == 0 for: 5m labels: { severity: critical } annotations: summary: "keepalived 进程挂了: {{ $labels.instance }}" - alert: LVSHostDown expr: probe_success{job="blackbox-tcp"} == 0 for: 5m labels: { severity: critical } - alert: KeepalivedSplitBrain expr: (keepalived_vrrp_state{instance="lb-master"} == 2) and on (iname) (keepalived_vrrp_state{instance="lb-backup"} == 2) for: 1m labels: { severity: critical } - alert: HighMemoryUsage expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 90 for: 5m labels: { severity: critical }

#内存告警和主机探活失败告警

# alertmanager.yml (邮件告警) global: smtp_smarthost: 'smtp.163.com:465' smtp_from: '15320239818@163.com' smtp_auth_username: '15320239818@163.com' smtp_auth_password: 'SMTP授权码' # 163用授权码, 不是登录密码! smtp_require_tls: true route: group_by: ['alertname'] group_wait: 30s group_interval: 60s repeat_interval: 10m receiver: email receivers: - name: 'email' email_configs: - to: '收件邮箱@qq.com' send_resolved: true

九、踩坑记录（亲测）

十、最终验证与全景

# 1. 所有 target 应 up curl -s localhost:9090/api/v1/targets | python3 -m json.tool | grep health # 2. keepalived 主备 (master=2 backup=1) curl -s 'localhost:9090/api/v1/query?query=keepalived_vrrp_state' # 3. 规则全部 health=ok curl -s localhost:9090/api/v1/rules | python3 -m json.tool # 4. 邮件告警实测: 临时调低阈值触发, 收邮件后恢复

十一、一句话记忆

• 分工：Prometheus 采 / Grafana 看 / Alertmanager 喊 / exporter 传感器 / blackbox 探针 / Ansible 铺

• 幂等三件套：stat判断 +when控制 +creates防重

• 探活：probe_success 1=活 0=挂；dns 别带 udp://，LVS 探 22 不探 80

• rpm 升级必须重启服务

• 告警链路：rules → Prometheus 评估 → Alertmanager 路由 → 邮箱

十二、后续规划

1. MySQL MGR 高可用+ HAProxy（数据层高可用）

2. mysql_exporter / haproxy_exporter 纳入监控

3. 整套监控用 Ansible role 化（参数化版本、规则模板）