Customize Kube-Prometheus Built-in Alerts

Kube-prometheus provides quite a few great collections of components and alerts that help us monitoring our Kubernetes cluster. I’ve used it in the production cluster for serveral months. Although the project exposes a bunch of options via _config+:: to makes it partially configurable, the scope of parameters that we can modify is still limited.

February 16, 2021 · Wi1dcard

改进 Alertmanager 的钉钉通知

多亏了 prometheus-webhook-dingtalk 项目,Alertmanager 集成钉钉消息通知得以实现。在实际使用过程中,我发现了一些可以改进的配置,例如通知消息模板可以更加清晰简洁、可以根据不同的报警优先级决定是否 Mention 某人等。

October 2, 2020 · Wi1dcard

Convert Kubernetes-Mixin Severities to OpsGenie Priorities in Prometheus

In kubernetes-mixin (a dependency of kube-prometheus-stack and prometheus-operator charts) rules, the severity label of alrets can be critical, warning, or info etc. However, OpsGenie’s priority field only accepts values like P1, P2 … P5. As a user of both of them, I would have to convert the “severity” to OpsGenie’s “priority”. For example, if a critical alert was fired, a matched P1 alert to be created in OpsGenie is expected, and warning -> P2, info -> P3, as well.

September 30, 2020 · Wi1dcard

Alertmanager 通过阿里云企业邮件 SMTP 服务发信

阿里云收购万网之后,几乎不赚钱的邮件服务还真是不维护不迭代了 — 不支持 DKIM,没法配 DMARC,就连 SMTP 也不支持 STARTTLS 587 端口。

September 24, 2020 · Wi1dcard

配置 Prometheus 服务器监控和 Grafana 看板

译者序:Prometheus 是服务器监控系统的后起之秀,可以和 Kubernetes 完美结合用于监控大量集群和应用。Grafana 是一款数据可视化看板,可指定多个数据源执行查询,将枯燥的数据转化为多维度的面板。两者均为开源项目,通过配置可实现直观强大的监控、报警、分析系统,实属运维神器。

January 9, 2019 · Wi1dcard