PrometheusGrafana实现对LinuxKubernetesNginx的监控系统

2019独角兽企业重金招聘Python工程师标准>>>

用Prometheus配合Grafana实现多用途的服务监控系统。

一. Prometheus简介

Prometheus是一个开源监控系统&＃xff0c;它前身是SoundCloud的警告工具包。从2012年开始&＃xff0c;许多公司和组织开始使用Prometheus。该项目的开发人员和用户社区非常活跃&＃xff0c;越来越多的开发人员和用户参与到该项目中。

具体介绍及架构信息参考Prometheus官网&＃xff1a;https://prometheus.io

二. Grafana介绍

grafana是用于可视化大型测量数据的开源程序&＃xff0c;他提供了强大和优雅的方式去创建、共享、浏览数据。dashboard中显示了你不同metric数据源中的数据。

具体参考Grafana官网&＃xff1a;https://grafana.com

三.实现实例

1.监控Openresty&＃xff08;Nginx&＃xff09;

1.1下载安装Openresty及nginx-module-vts监控模块

#安装编译环境 yum -y install gcc-c&＃43;&＃43; pcre pcre-devel zlib zlib-devel openssl openssl-devel#下载openresty和nginx-module-vts监控模块 cd /data/server wget https://openresty.org/download/openresty-1.13.6.2.tar.gz git clone https://github.com/vozlt/nginx-module-vts.git#解压openresty tar -zxf openresty-1.13.6.2.tar.gz#添加模块编译 ./configure --add-module&＃61;/data/server/nginx-module-vts make && make install

1.2 修改Openresty配置

不指定安装路径的话Openresty默认安装在/usr/local/openresty

进入目录/usr/local/openresty/nginx/conf 修改配置文件

cd /usr/local/openresty/nginx/conf #先备份 cp nginx.conf nginx.conf.bak vim nginx.conf

变更配置如下&＃xff1a;

主要两个配置&＃xff0c;1.指定时间同步&＃xff0c;输出日志格式。2.添加http访问路由

user nobody; worker_processes 1;events {worker_connections 1024; }http {include mime.types;default_type application/octet-stream;vhost_traffic_status_zone;log_format main &＃39;{ "&＃64;timestamp": "$time_local", &＃39;&＃39;"&＃64;fields": { &＃39;&＃39;"uri":"$request_uri",&＃39;&＃39;"url":"$uri",&＃39;&＃39;"upstream_addr":"$upstream_addr",&＃39;&＃39;"remote_addr": "$remote_addr", &＃39;&＃39;"remote_user": "$remote_user", &＃39;&＃39;"body_bytes_sent": "$body_bytes_sent", &＃39;&＃39;"host":"$host",&＃39;&＃39;"server_addr":"$server_addr",&＃39;&＃39;"request_time": "$request_time", &＃39;&＃39;"request_time":"$request_time",&＃39;&＃39;"status":"$status",&＃39;&＃39;"request": "$request", &＃39;&＃39;"request_method": "$request_method", &＃39;&＃39;"size":$body_bytes_sent,&＃39;&＃39;"upstream_time":"$upstream_response_time"&＃39;&＃39;"http_referrer": "$http_referer", &＃39;&＃39;"body_bytes_sent":"$body_bytes_sent", &＃39;&＃39;"http_x_forwarded_for": "$http_x_forwarded_for", &＃39;&＃39;"http_user_agent": "$http_user_agent" } }&＃39;;sendfile on;keepalive_timeout 65;gzip on;gzip_min_length 1k;gzip_buffers 4 16k;gzip_http_version 1.0;gzip_comp_level 3;gzip_types text/plain text/css application/x-Javascript text/xml application/xml application/xml&＃43;rss text/Javascript application/json application/Javascript;gzip_vary on;proxy_http_version 1.1;proxy_set_header Connection "";server {listen 80;server_name localhost;location / {root html;index index.html index.htm;}location /status {vhost_traffic_status_display;vhost_traffic_status_display_format html;}} }

测试配置文件是否有误

./sbin/nginx -t

通过测试后启动Openresty

./sbin/nginx

访问地址http://[ip:port]/status

可以看到如下信息&＃xff1a;

1.3配置监控转发模块

这里先使用一个Github上已经开发好的第三方监控模块nginx-vts-exporter

下载地址&＃xff1a;https://github.com/hnlq715/nginx-vts-exporter/releases

cd /data/server wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz tar -zxf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz mv nginx-vts-exporter-0.10.3.linux-amd64 nginx-vts-exporter

把nginx-vts-exporter 写成服务(以Centos7.X为例)&＃xff1a;

在/usr/lib/systemd/system/下创建启动文件

vim /usr/lib/systemd/system/nginx-vts-exporter.service

配置启动可执行文件和参数

[Unit] Description&＃61;nginx-vts-exporter After&＃61;network.target[Service] Type&＃61;simple ExecStart&＃61;/data/server/nginx-vts-exporter/nginx-vts-exporter -nginx.scrape_uri&＃61;http://127.0.0.1/status/format/json Restart&＃61;on-failure[Install] WantedBy&＃61;multi-user.target

生效配置文件并启动nginx-vts-exporter

#启动服务&＃xff0c;nginx-vts-exporter 默认监听端口为 9913 systemctl enable nginx-vts-exporter systemctl start nginx-vts-exporter systemctl status nginx-vts-exporter

1.4 配置Prometheus

配置文件路径&＃xff1a;/data/server/prometheus/prometheus.yml&＃xff0c;编辑配置文件添加如下内容&＃xff1a;

# my global config global:scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.# scrape_timeout is set to the global default (10s).# Alertmanager configuration alerting:alertmanagers:- static_configs:- targets:# - alertmanager:9093# Load rules once and periodically evaluate them according to the global &＃39;evaluation_interval&＃39;. rule_files:# - "first_rules.yml"# - "second_rules.yml"# A scrape configuration containing exactly one endpoint to scrape: # Here it&＃39;s Prometheus itself. scrape_configs:# The job name is added as a label &＃96;job&＃61;&＃96; to any timeseries scraped from this config.- job_name: &＃39;prometheus&＃39;# metrics_path defaults to &＃39;/metrics&＃39;# scheme defaults to &＃39;http&＃39;.static_configs:- targets: [&＃39;localhost:9090&＃39;]- job_name: &＃39;openresty-master&＃39;static_configs:- targets: [&＃39;localhost:9913&＃39;]

配置服务来启动

vim /usr/lib/systemd/system/prometheus.service

写入如下内容&＃xff1a;

[Unit] Description&＃61;prometheus After&＃61;network.target[Service] Type&＃61;simple ExecStart&＃61;/data/server/prometheus/prometheus --config.file&＃61;/data/server/prometheus/prometheus.yml Restart&＃61;on-failure[Install] WantedBy&＃61;multi-user.target

启动prometheus服务

#启动prometheus服务 systemctl enable prometheus systemctl start prometheus systemctl status prometheus

1.5 安装配置Grafana

下载地址&＃xff1a;https://grafana.com/grafana/download?platform&＃61;linux

按上述页面说明安装

wget https://dl.grafana.com/oss/release/grafana-6.1.3-1.x86_64.rpm yum localinstall grafana-6.1.3-1.x86_64.rpm

我比较习惯先修改一下登录密码&＃xff08;默认账号密码 admin admin&＃xff09;

先看一下安装位置

whereis grafana

发现安装在/etc/grafana 去/etc/grafana下编辑grafana.ini文件

修改[security]节点下的password为你期望的值

新版本安装后输入admin默认密码第一次进入会提示修改密码

启动Grafana服务并查看状态

systemctl start grafana-server systemctl status grafana-server

访问页面http://[ip]:3000

进入后点击设置data sources add data sources

选择Prometheus填写如下信息

使用Prometheus 2.0 Stats作为Dashboard效果如下&＃xff1a;

2.监控Linux主机

2.1监控代理Node exporter 安装

首先在prometheus官网下载 node_exporter。官网地址为&＃xff1a; https://prometheus.io/docs/introduction/overview/

#下载解压 wget https://github.com/prometheus/node_exporter/releases/download/v0.17.0/node_exporter-0.17.0.linux-amd64.tar.gz tar -zxf node_exporter-0.17.0.linux-amd64.tar.gz mv node_exporter-0.17.0.linux-amd64.tar.gz node_exporter cd node_exporter #后台启动 nohup ./node_exporter &

服务模式启动&＃xff1a;

[Unit] Description&＃61;node-exporter After&＃61;network.target[Service] Type&＃61;simple ExecStart&＃61;/data/monitor/node_exporter/node_exporter Restart&＃61;on-failure[Install] WantedBy&＃61;multi-user.target

*查询 node_exporter进程

#默认监听9100 netstat -ntlp|grep 9100 netstat -anp | grep 9100

2.2 prometheus添加代理配置

vim prometheus.yml

添加如下配置&＃xff1a;

- job_name: &＃39;tx-server&＃39;static_configs:- targets: [&＃39;94.191.16.32:9100&＃39;]

重启prometheus

systemctl restart prometheus

3.监控Kubernetes&＃xff08;k8s&＃xff09;

待补充...

附录&＃xff1a;

1.Grafana安装插件

grafana-cli plugins install [michaeldmoore-annunciator-panel] service grafana-server restart

2.command

#启动nginx服务 systemctl start nginx.service #设置开机自启动 systemctl enable nginx.service #停止开机自启动 systemctl disable nginx.service #查看服务当前状态 systemctl status nginx.service #重新启动服务 systemctl restart nginx.service #查看所有已启动的服务 systemctl list-units --type&＃61;service