elk6安装过程

2018-08-21

logstash是一个数据分析软件,主要目的是分析log日志。整一套软件可以当作一个MVC模型,logstash是controller层,Elasticsearch是一个model层,kibana是view层。

首先将数据传给logstash,它将数据进行过滤和格式化(转成JSON格式),然后传给Elasticsearch进行存储、建搜索的索引,kibana提供前端的页面再进行搜索和图表可视化,它是调用Elasticsearch的接口返回的数据进行可视化。logstash和Elasticsearch是用Java写的,kibana使用node.js框架。

1.安装java环境

apt install openjdk-8-jre-headless

2.安装elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.deb
dpkg -i elasticsearch-6.3.2.deb

2.1.安装elasticsearch-head插件

git clone https://github.com/mobz/elasticsearch-head.git  
cd elasticsearch-head  
npm install
# 如果报错 npm ERR! Failed at the phantomjs-prebuilt@2.1.16 install script ,则安装
npm install phantomjs-prebuilt@2.1.16 --ignore-scripts
npm run start

安装完head插件需要把elasticsearch的跨域访问开启,往elasticsearch.yml添加

#allow origin
http.cors.enabled: true
http.cors.allow-origin: "*"

3.安装logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.deb
dpkg -i logstash-6.3.2.deb

添加配置软链,防止警告:

ln -s /etc/logstash /usr/share/logstash/config

3.1.添加logstash配置文件

cat /etc/logstash/conf.d/logstash.conf

input {
    file {
        path => "/app/log/*.log"
        start_position => beginning
        sincedb_path => "/tmp/logstash.db"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    date {
        locale => "en"
        match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
    }
    geoip {
        source => "clientip"
    }
    if [@timestamp] < "2018-08-17" {
        drop {}
    }
}
output {
    elasticsearch {
         index => "logstash-16988.com"
         workers => 5
         hosts => "localhost:9200"
    }
}

logstash 日期转换成时间戳

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    date {
        locale => "en"
        match => ["timestamp" ,"dd/MMM/YYYY:HH:mm:ss Z"]
        target => "pubdate"
    }
    ruby {    
        code => 'event.set("pubdate",event.get("pubdate").to_i)'  
    }
}

logstash.conf 根据file内的日期过滤,例如只导10天以内的数据

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    date {
        locale => "en"
        match => ["timestamp" ,"dd/MMM/YYYY:HH:mm:ss Z"]
    }
    geoip {
        source => "clientip"
    }
    ruby {    
        code => 'if event.get("[@timestamp]").to_i < ( Time.now - 864000).to_i
            event.cancel
        end'  
    }
}

logstash 分析新浪SAE访问日志

filter {
    grok {
        match => { "message" => "%{HOSTNAME:domain} %{IPV4:clientip} %{INT:runtime} %{INT:cputime} \[%{HTTPDATE:timestamp}\] %{WORD:app} %{WORD:app_hash} %{INT:app_version} \"%{WORD:request_method} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:status} %{DATA:bytes} \"%{GREEDYDATA:referrer}\" \"%{GREEDYDATA:agent}\" *%{WORD:app_host}"}
    }   
    date {
        locale => "en"
        match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"] 
        timezone => "Asia/Shanghai"
    }   
    geoip {
        source => "clientip"
    }   
}

自动解压SAE压缩日志脚本:

#!/bin/bash
for file in *.tar.gz
do
if [[ $file != "*.tar.gz" ]]; then
	tar xf $file
	DATA=`echo $file|awk -F "_" '{print $1}'`
	NAME_1=`echo $file|awk -F "_" '{print $3}'`
	mv ${NAME_1}_log ${DATA}_${NAME_1}_log
	echo ${DATA}_${NAME_1}_log
	rm $file
fi
done

只要把该脚本放在压缩日志同级目录下,运行即可

新浪云日志说明: http://www.sinacloud.com/doc/sae/tutorial/app-management.html#ri-zhi-zhong-xin

grok规则调试工具:

https://grokconstructor.appspot.com/do/match (推荐)

http://grok.qiexun.net/

http://grokdebug.herokuapp.com

nginx对应正则

nginx日志字段定义 nginx访问字段 正则表达式
$time_local 08/Jan/2016:08:27:43 +0800 %{HTTPDATE:timestamp}
$upstream_addr 10.10.6.212:8088 %{HOSTPORT:upstream}
serveraddr:server_port 10.10.6.110:80 %{HOSTPORT:request_server}
$request_method GET %{WORD:request_method}
$uri /vvv/test/stat/index %{URIPATH:uri}
$args proptype=11&level=2&srtype=2&city=dz&region=XJ&begindate=2016-01-08&enddate=2016-01-08&apiKey=c2c959b203d669a9a21861379cb4523c&test=2 %{URIPARAM:args}
$remote_addr 10.10.6.10 %{IPV4:clientip}
$server_protocol HTTP/1.1 HTTP/%{NUMBER:httpversion}
[$http_user_agent] [Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36] [%{GREEDYDATA:agent}]
[$http_cookie] [JSESSIONID=kq3v6xi2b74j1a9yxvfvcq131] [%{GREEDYDATA:cookie}]
$http_referer http://10.10.6.110/test %{GREEDYDATA:referrer}
$host www.test.com %{HOSTNAME:domain}
$status 200 %{NUMBER:status:int}
$bytes_sent 485 %{NUMBER:body_sent:int}
$request_length 209 %{NUMBER:request_length:int}
0”$upstream_cache_status” 0”-“ 0"-"
$request_time 1.642 %{NUMBER:request_time:float}
$upstream_response_time 1.642 %{NUMBER:response_time:float}

3.2.运行logstash

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/

3.3.logstash的sincedb_path读取进度记录

/usr/share/logstash/data/plugins/inputs/file/

如果设置过软链,会读logstash.yml文件的path.data路径,则会在该目录下:

/var/lib/logstash/plugins/inputs/file/

3.4.logstash性能优化,调整参数:修改配置文件 logstash.yml

pipeline.workers: 1 
pipeline.batch.size: 125
pipeline.batch.delay: 250

命令参数:–filterworkers 或 -w 意即工作线程。Logstash 会运行多个线程。你可以用 bin/logstash -w 5 这样的方式强制 Logstash 为过滤插件运行 5 个线程。

4.安装kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-amd64.deb
dpkg -i kibana-6.3.2-amd64.deb

4.1.修改kibana监听地址

vim /etc/kibana/kibana.yml
server.host: "0.0.0.0"

其他

elasticsearch-head 通过base auth访问es

http://localhost:9100/?base_uri=http://localhost:9200&auth_user=elastic&auth_password=password

默认端口

elasticsearch集群通讯端口     :9300
elasticsearch                 :9200
elasticsearch-head            :9100
kibana                        :5601

ES操作

查看总数:

GET /logstash-*/_count?pretty

获取一条:

索引库/类型/ID

GET /{index}/{type}/{id}

添加一条:

PUT /{index}/{type}/{id}
{
  "field": "value",
  ...
}

查询:

POST /{index}/_search
{"query":{"range":{"@timestamp":{"lt":"2018-08-17"}}}}

查询删除:

POST /{index}/_delete_by_query
{"query":{"range":{"@timestamp":{"lt":"2018-08-17"}}}}

删除索引:

DEL /{index}

设置副本数量

PUT /_settings
{
    "number_of_replicas": 0
}

创建只有 一个主分片,没有副本的小索引:

PUT /my_temp_index
{
    "settings": {
        "number_of_shards" :   1,
        "number_of_replicas" : 0
    }
}

number_of_shards
每个索引的主分片数,默认值是 5 。这个配置在索引创建后不能修改。
number_of_replicas
每个主分片的副本数,默认值是 1 。对于活动的索引库,这个配置可以随时修改。

ES 配置/状态

基本索引度量标准(分片数,存储大小,内存使用情况)以及有关组成集群的当前节点的信息(数字,角色,操作系统,jvm版本,内存使用情况,cpu和已安装的插件)。

GET /_cluster/stats

节点统计信息

GET /_nodes/stats

配置优化

jvm内存建议设置为物理内存的二分之一

cat /etc/elasticsearch/jvm.options

-Xms2g
-Xmx2g