淘宝返利软件的可观测性架构：Prometheus与Grafana监控体系搭建-编程实验室

淘宝返利软件的可观测性架构：Prometheus与Grafana监控体系搭建

大家好，我是微赚淘客系统3.0 的研发者省赚客！

在高并发、分布式环境下，淘客系统的稳定性依赖于完善的可观测性能力。微赚淘客系统3.0 基于 Prometheus + Grafana + Spring Boot Actuator 构建了端到端监控体系，覆盖 JVM 指标、HTTP 接口性能、业务关键链路及异常告警，实现“问题可发现、根因可定位、趋势可预测”。

一、Spring Boot 应用暴露指标

首先，在juwatech.cn.rebate模块中集成 Micrometer 与 Actuator：

<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-actuator</artifactId></dependency><dependency><groupId>io.micrometer</groupId><artifactId>micrometer-registry-prometheus</artifactId></dependency>

配置application-prod.yml开启 Prometheus 端点：

management:endpoints:web:exposure:include:health,info,prometheus,metricsendpoint:prometheus:enabled:truehealth:show-details:always

启动后，访问/actuator/prometheus即可获取指标文本，例如：

http_server_requests_seconds_count{method="GET",uri="/api/commission",status="200",} 1245.0 jvm_memory_used_bytes{area="heap",id="G1 Eden Space",} 2.147e8

二、自定义业务指标埋点

针对核心业务如佣金计算、订单同步，我们通过MeterRegistry注入自定义计数器与直方图：

packagejuwatech.cn.rebate.service;importio.micrometer.core.instrument.Counter;importio.micrometer.core.instrument.MeterRegistry;importio.micrometer.core.instrument.Timer;importorg.springframework.stereotype.Service;@ServicepublicclassCommissionCalculationService{privatefinalCountercommissionSuccessCounter;privatefinalCountercommissionFailureCounter;privatefinalTimercommissionProcessTimer;publicCommissionCalculationService(MeterRegistrymeterRegistry){this.commissionSuccessCounter=Counter.builder("rebate.commission.success").description("成功计算佣金次数").register(meterRegistry);this.commissionFailureCounter=Counter.builder("rebate.commission.failure").description("佣金计算失败次数").register(meterRegistry);this.commissionProcessTimer=Timer.builder("rebate.commission.duration").description("佣金计算耗时（秒）").register(meterRegistry);}publicBigDecimalcalculate(LongorderId){returncommissionProcessTimer.recordCallable(()->{try{// 模拟佣金计算逻辑BigDecimalamount=doCalculate(orderId);commissionSuccessCounter.increment();returnamount;}catch(Exceptione){commissionFailureCounter.increment();throwe;}});}privateBigDecimaldoCalculate(LongorderId){// 实际业务逻辑returnnewBigDecimal("12.50");}}

三、Prometheus 配置服务发现

在prometheus.yml中配置基于 Kubernetes 的服务发现，自动抓取所有rebate服务实例：

scrape_configs:-job_name:'rebate-app'kubernetes_sd_configs:-role:podrelabel_configs:-source_labels:[__meta_kubernetes_pod_label_app]action:keepregex:rebate-system-source_labels:[__meta_kubernetes_pod_annotation_prometheus_io_scrape]action:keepregex:true-source_labels:[__meta_kubernetes_pod_ip]target_label:__address__replacement:$1:8080-source_labels:[__meta_kubernetes_namespace]target_label:namespace

确保 Pod 注解包含：

metadata:labels:app:rebate-systemannotations:prometheus.io/scrape:"true"

四、Grafana 仪表盘配置

导入或创建以下关键面板：

JVM 内存与 GC：使用JVM (Micrometer)官方模板（ID: 4701）

HTTP 请求延迟分布：查询语句：

histogram_quantile(0.95, rate(http_server_requests_seconds_bucket{job="rebate-app"}[5m]))

业务成功率：

rate(rebate_commission_success_total[5m]) / (rate(rebate_commission_success_total[5m]) + rate(rebate_commission_failure_total[5m]))

同时配置告警规则，例如接口错误率突增：

# alert.rules.ymlgroups:-name:rebate-alertsrules:-alert:HighCommissionErrorRateexpr:rate(rebate_commission_failure_total[5m])>0.1for:2mlabels:severity:criticalannotations:summary:"佣金计算失败率过高"description:"过去5分钟失败率超过10%，当前值：{{ $value }}"

在 Prometheus 中加载该规则：

rule_files:-"alert.rules.yml"

五、日志与链路追踪联动

虽然本文聚焦指标监控，但实际生产中需结合 Loki（日志）与 Jaeger（链路）。我们在关键方法添加 Trace ID 日志：

importorg.slf4j.Logger;importorg.slf4j.LoggerFactory;importorg.springframework.web.filter.OncePerRequestFilter;publicclassTraceIdFilterextendsOncePerRequestFilter{privatestaticfinalLoggerlog=LoggerFactory.getLogger(TraceIdFilter.class);@OverrideprotectedvoiddoFilterInternal(HttpServletRequestrequest,HttpServletResponseresponse,FilterChainfilterChain)throwsIOException,ServletException{StringtraceId=UUID.randomUUID().toString().replace("-","").substring(0,16);MDC.put("traceId",traceId);log.info("Start request: {} {}",request.getMethod(),request.getRequestURI());try{filterChain.doFilter(request,response);}finally{MDC.clear();}}}