安装Logstash+ElasticSearch5+Kibana5
JDK
最新版elasticsearch-5.0.0-alpha4需要java-1.8，
首先下载jdk-1.8，下载地址：http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
解压安装：
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz
tar -zxvf jdk-8u92-linux-x64.tar.gz
mkdir /usr/local/java
mv jdk1.8.0_92 /usr/local/java/
设置环境变量：
vim ~/.bashrc 追加：
export JAVA_HOME=/usr/local/java/jdk1.8.0_92
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
验证安装版本：
root@ubuntu:~# java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
ElasticSearch5
ElasticSearch官方下载:https://www.elastic.co/downloads/elasticsearch
官方提供deb包，那就偷懒下 elasticsearch-5.0.0-alpha4.deb
安装：
dpkg -i elasticsearch-5.0.0-alpha4.deb
默认安装在/usr/share/elasticsearch 目录
启动：
cd /usr/share/elasticsearch/
mkdir config
cd bin/
./elasticsearch
错误：
如果报错：
Exception in thread "main" java.lang.RuntimeException: can not run elasticsearch as root
那么切换到普通用户
如果还报错：
Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.conf' (/usr/share/elasticsearch/config)
是因为访问权限不够：
chown malu /usr/share/elasticsearch -R
中文文档：http://learnes.net/
https://www.gitbook.com/book/looly/elasticsearch-the-definitive-guide-cn/details
Kibana5
Kibana官方下载: https://www.elastic.co/downloads/kibana
deb包下载： kibana-5.0.0-alpha4-amd64.deb
安装：
dpkg -i kibana-5.0.0-alpha4-amd64.deb
默认安装在/usr/share/kibana
启动：
cd /usr/share/kibana/bin
./kibana
中文文档：http://www.code123.cc/docs/kibana-logstash/
logstash5
官方下载：https://www.elastic.co/downloads/logstash
deb下载：logstash-5.0.0-alpha4.deb
安装：
dpkg -i logstash-5.0.0-alpha4.deb
默认安装在/usr/share/logstash
启动：
首先需要配置运行文件：
vim /etc/logstatsh/conf.d/first-pipeline.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => beginning
ignore_older => 0
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200","server-2.malu.me:9200" ]
user => "username"
password => "pass"
index => "%{module}-%{+YYYY.MM.dd}"
document_type => "%{msgtype}"
}
}
启动前可以用命令检查一下参数是否正确：
bin/logstash -f first-pipeline.conf --config.test_and_exit
然后执行：
/usr/share/logstash/bin/logstash
此时会读取logstash默认配置文件 /usr/share/logstash/config/logstash.yml
如果不存在，请从/etc/logstash/logstash.yml复制
logstash 2.3.4
下载安装环境：
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.3.4-1_all.deb
dpkg -i logstash_2.3.4-1_all.deb
apt-get update && apt-get install -y openjdk-7-jre-headless
vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/var/log/apache/access.log"
start_position => beginning
ignore_older => 0
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => "es.malu.me:6585"
user => "malu"
password => "elk"
}
}
启动：
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
离线日志采集
离线日志文件对应的默认的 $HOME/.sincedb_*文件删掉再用logstash采集
文档：
Logstash 最佳实践 http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html
redis
可以用redis来实现分布式日志管理
参考这篇文章：http://git.malu.me/Redis客户端常用命令/
配置redis_index.conf文件，直接与elasticsearch通讯：
root@ubuntu:/etc/logstash/conf.d# cat redis_index.conf
input {
redis {
host => "localhost"
port => 6379
password => "redispassword"
data_type => "list"
key => "logstash:redis"
type => "redis-input"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
配置redis_cli.conf用于读取日志：
root@ubuntu:/etc/logstash/conf.d# cat redis_cli.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => beginning
ignore_older => 0
}
}
output {
redis {
host => "localhost:6379"
password => "redispassword"
data_type => "list"
key => "logstash:redis"
}
}
然后启动：
./logstash -f /etc/logstash/conf.d/redis_index.conf
./logstash -f /etc/logstash/conf.d/redis_cli.conf --http.port 9601 #启动第二个进程会提示9600端口占用，指定其他端口就可以了
中文文档：ELKstack 中文指南
ElasticSearch5监听外网端口
一般不建议把ElasticSearch监听放外网，但本人在终端环境下，用curl实在不爽。
首先拷贝配置文件：
cp /etc/elasticsearch/elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml
vim /usr/share/elasticsearch/config/elasticsearch.yml
network.host: 192.168.1.222
http.port: 9200
discovery.zen.minimum_master_nodes: 1 #这里设为1, 设3的话会导致Kibana报错
如果直接启动ElasticSearch会报错：
initial heap size [33554432] not equal to maximum heap size [522190848]; this can cause resize pauses and prevents mlockall from locking the entire heap
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
有4个错误，我们一个个来解决：
initial heap size [33554432] not equal to maximum heap size [522190848]; this can cause resize pauses and prevents mlockall from locking the entire heap
这个错误是关于jvm内存分配的问题，需要设置的jvm配置：
cp /etc/elasticsearch/jvm.options /usr/share/elasticsearch/config/jvm.options
vim /usr/share/elasticsearch/config/jvm.options
-Xmx2g 改成 -Xmx256m ,也就是heap size [268435456] /1024/1024的值
第二个错误：
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
最大文件打开句柄数太少了，需要设置成65536：
cp /etc/security/limits.conf /etc/security/limits.conf.bak
vim /etc/security/limits.conf
malu hard nofile 65536
malu soft nofile 65536
其中malu是启动ElasticSearch的用户名
修改后重新登录malu用户，使用如下命令查看是否修改成功
[malu@localhost ~]$ ulimit -Hn
65536
第三个错误：
please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
配置文件elasticsearch.yml里设置：
discovery.zen.minimum_master_nodes: 1
第四个错误：
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
最大线程数设置过小，需要root用户来设置：
sysctl -w vm.max_map_count=262144
查看：
[root@localhost ~]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 262144
下次启动这个设置会丢失，我们把它写入配置文件：
vim /etc/sysctl.conf
vm.max_map_count=262144 #添加这一行
以上文件解决后，elasticsearch就可以正常启动了。不过由于修改了监听地址，Kibana与它的通讯地址也得对应修改：
vim /etc/kibana/kibana.yml
elasticsearch.url: "http://192.168.1.222:9200" #添加这一行
如果logstash配置了Redis与之通讯，那么同样也得修改：
output {
elasticsearch {
hosts => [ "192.168.1.222:9200" ]
}
}
文档：
elasticsearch 集群传输端口配置官方文档
ELKstack 中文指南 : https://www.gitbook.com/book/chenryn/kibana-guide-cn/details
Elasticsearch 权威指南（中文版）
官网 : https://www.elastic.co
国内几个ELK相关博客：
http://chenlinux.com/
http://xiaorui.cc/category/elasticsearch/
http://bigbo.github.io/category/technology.html
Other
高性能的PHP日志系统—SeasLog
使用 Grafana、collectd 和 InfluxDB 打造现代监控系统
