ELK-5.4.1版本的安装过程

环境

  • 系统版本 Linux version 2.6.32-696.3.1.el6.x86_64 (mockbuild@c1bl.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) )
  • java版本 openjdk version "1.8.0_131"

安装版本

  • ElasticSearch 5.4.1
  • Kibana 5.4.1
  • Logstash 5.4.1

部署

0x01 确认JDK版本及安装

  1. 依赖java的最小版本为1.8
yum install java-1.8.0-openjdk
java version

0x02 ElasticSearch安装

  1. 下载es,官网下载路径:https://www.elastic.co/downloads/elasticsearch
  2. 解压 tar xf elasticsearch-5.4.1.tar.gz
  3. 用户/目录/权限设置
  • es运行不能使用root用户,需要创建新用户
useradd es
  • 新建目录,用于es的数据和日志存储
mkdir -p /data/Logs
mkdir -p /data/Data
  • 设置目录所有者为es,否则无法访问(出现Permission denied的错误,记得设置目录权限
chown -R es:es /data/Logs
chown -R es:es /data/Data
chown -R es:es elasticsearch-5.4.1  #注意,需要设置权限
  1. 修改配置文件config/elasticsearch.yml
  • 集群名: cluster.name, 注意: 两台机器配置一致
cluster.name: inner_es_cluster
  • 节点名: node.name, 注意: 两台机器配置不同, 一台为01, 另一台为02
#第一台机器
node.name: inner_es_node_01
#第二台机器
node.name: inner_es_node_02
  • 数据路径: path.data, 为新建立的目录
path.data: /data/Data/
  • 日志路径: path.logs
path.logs: /data/Logs
  • LockMemory:
bootstrap.mlockall: true
  • 本机ip: network.host, 注意两台机器配置不同, 分贝配置为对应机器的内网ip
#第一台机器
network.host: 10.10.44.202
#第二台机器
network.host: 10.10.44.203
  • 绑定端口,注意默认http.port=9200,transport.tcp.port=9300
#配置Http访问的端口  
http.port: 9200
#配置节点之间交互的端口
transport.tcp.port: 9300
  • Discovery配置: 注意这里是两台机器内网ip+9300端口,注意这里minimum_master_nodes=2, 见最后一点防脑裂说明
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["10.10.44.202:9300", "10.10.44.203:9300"]
  1. 设置es占用内存
  • ES-5.4.1版本,在config/jvm.options中设置为4G(根据自身机器配置, 配置的内存最大不超过机器物理内存的75%. 两个变量值相等, 以获取最大的性能). 当然, 实际使用中4g可能远远不够, 这个值仅是个示例
-Xms4g
-Xmx4g
  • 启用memlock, 提升性能。修改centos配置: /etc/security/limits.conf
    加入, 注意, 示例中用户为es
es soft memlock unlimited
es hard memlock unlimited

设置完成,重启 /etc/init.d/sshd restart
如果结果是unlimited, 则无需任何处理, 直接进入下一步

ulimit -n
unlimited
  1. ES运行错误
  • 以用户es的身份运行
su es
./bin/elasticsearch   
#或者
runuser -l es -c "/data/tools/elasticsearch/bin/elasticsearch"
  • 问题:max file descriptors
ERROR: [1] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]

解决:编辑/etc/security/limits.conf, 加入

es soft nofile 204800
es hard nofile 204800
  • 问题:max threads descriptors
ERROR: [1] bootstrap checks failed
[1]: max threads descriptors [1024] for elasticsearch threads is too low, increase to at least [2048]

解决:编辑/etc/security/limits.conf, 加入

es soft nproc 2048
es hard nproc 4096
  • 问题:system call filters failed to install
ERROR: [1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

解决:编辑config/elasticsearch.yml, 加入

bootstrap.system_call_filter: false
  • 问题:max virtual memory areas vm.max_map_count
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

解决:编辑/etc/sysctl.conf,加入

vm.max_map_count=262144
然后执行sysctl -p  

或者临时设置,执行sysctl -w vm.max_map_count=262144

  • 问题:ES_HEAP_SIZE
Error: encountered environment variables that are no longer supported
Use jvm.options or ES_JAVA_OPTS to configure the JVM
ES_HEAP_SIZE=16g: set -Xms16g and -Xmx16g in jvm.options or add "-Xms16g -Xmx16g" to ES_JAVA_OPTS

解决:ElasticSearch-5.4.1版本的内存使用,在config/jvm.options中设置,如果设置了

export ES_HEAP_SIZE=4g
export ES_JAVA_OPTS="-Xms4g -Xmx4g -Xss128m"

需要将环境变量制空

unset ES_HEAP_SIZE
unset ES_JAVA_OPTS
  1. 启动测试
  • 以用户es的身份运行
./bin/elasticsearch
#或者
runuser -l es -c "/data/tools/elasticsearch/bin/elasticsearch -d" #-d代表以damon后台运行

可以看到启动日志

[2016-06-30 17:20:26,677][WARN ][bootstrap                ] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in
[2016-06-30 17:20:27,390][INFO ][node                     ] [inner_es_node_01] version[2.3.3], pid[6415], build[218bdf1/2016-05-17T15:40:04Z]
[2016-06-30 17:20:27,390][INFO ][node                     ] [inner_es_node_01] initializing ...
[2016-06-30 17:20:27,948][INFO ][plugins                  ] [inner_es_node_01] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-06-30 17:20:27,974][INFO ][env                      ] [inner_es_node_01] using [1] data paths, mounts [[/data (/dev/xvdb1)]], net usable_space [67.4gb], net total_space [98.4gb], spins? [no], types [ext3]
[2016-06-30 17:20:27,974][INFO ][env                      ] [inner_es_node_01] heap size [990.7mb], compressed ordinary object pointers [true]
[2016-06-30 17:20:29,926][INFO ][node                     ] [inner_es_node_01] initialized
[2016-06-30 17:20:29,926][INFO ][node                     ] [inner_es_node_01] starting ...
[2016-06-30 17:20:30,083][INFO ][transport                ] [inner_es_node_01] publish_address {10.0.0.1:9300}, bound_addresses {10.0.0.1:9300}
[2016-06-30 17:20:30,088][INFO ][discovery                ] [inner_es_node_01] inner_es_cluster/odmTjZRHRVaa8Zn4vTPcxA
[2016-06-30 17:21:00,091][WARN ][discovery                ] [inner_es_node_01] waited for 30s and no initial state was set by the discovery
[2016-06-30 17:21:00,099][INFO ][http                     ] [inner_es_node_01] publish_address {10.0.0.1:9200}, bound_addresses {10.0.0.1:9200}
[2016-06-30 17:21:00,099][INFO ][node                     ] [inner_es_node_01] started

等待约一分钟后, 看到如下日志代表启动成功

[2016-06-30 17:21:00,099][INFO ][node                     ] [inner_es_node_01] started

确认集群是否启动成功

curl http://10.10.44.202:9200/
{
  "name" : "inner_es_node_01",
  "cluster_name" : "inner_es_cluster",
  "version" : {
    "number" : "2.3.3",
    "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
    "build_timestamp" : "2016-05-17T15:40:04Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

确认健康状态,如果为red则需要等待一会,或者查看具体原因

curl 10.10.44.202:9200/_cluster/health?pretty
{
  "cluster_name" : "juzi-log",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
颜色 意义
green 所有主要分片和从分片都可用
yellow 所有主要分片可用,但不是所有从分片都可用
red 不是所有的主要分片都可用
  1. ElasticSearch插件的安装
    此过程可以稍后再安装
  • elasticsearch-sql-5.4.1.0
./bin/elasticsearch-plugin install https://github.com/NLPchina/elasticsearch-sql/releases/download/5.4.1.0/elasticsearch-sql-5.4.1.0.zip

一定要根据es的版本,确定安装对应插件版本
通过远程下载再安装往往特别慢,甚至安装失败,可以下载后本地安装

./bin/elasticsearch-plugin install file:///path/to/elasticsearch-sql-5.4.1.0.zip
  • 安装完成重新启动es
    es-5.x版本跟1.x/2.x有很大不一样
On elasticsearch 1.x / 2.x, visit the elasticsearch-sql web front-end:
http://localhost:9200/_plugin/sql/

On elasticsearch 5.x 需要安装site-server才可以访问查看页面

Then start the web front-end like this:
cd site-server
npm install express --save
node node-server.js 

5.x中启动node-server.js后台,访问 http://yourHost:8080即可出现通过SQL语句查询页面,端口可在site_configuration.json配置
需要注意的是node-server.js只是提供页面展示,具体查询调用的接口是POST http://localhost:9200/_sql/,需要在nginx层配置当为GET请求时访问http://yourHost:8080,POST请求访问:http://localhost:9200

  1. 参考
    ElasticSearch集群部署文档
    elasticsearch-sql
    elasticsearch-sql-wiki
    ES插件安装举例
    Elasticsearch: 权威指南 (有关于es健康状态查询说明)
    Elasticsearch权威指南(中文版)

  2. 配置说明

#cluster.name: elasticsearch             配置集群名称  
#node.name: "Franz Kafka"                配置节点名称  
#node.master: true                       配置当前节点是否具有可选为master节点的资格、  
#node.data: true                         配置当前节点是否允许存储数据  
#node.max_local_storage_nodes: 1         配置每个几点可以启动的Elasticsearch最大实例个数  
#index.number_of_shards: 5               配置索引的默认分片数  
#index.number_of_replicas: 1             配置索引的默认副本数  
#path.conf: /path/to/conf                配置文件存放目录  
#path.data: /path/to/data                配置该节点索引数据的存放目录,多个用都好分割  
#path.logs: /path/to/logs                配置日志的存放目录  
#path.plugins: /path/to/plugins          配置插件安装目录  
#network.host: 192.168.0.1               配置节点绑定的IP和与其他几点交互的IP  
#transport.tcp.port: 9300                配置节点之间交互的端口  
#http.port: 9200                         配置Http访问的端口  
#http.max_content_length: 100mb          配置可允许的文本最大长度  
#minimum_master_nodes: 2                 决定了选主需要的最少节点数, N/2+1, 两个节点即2

0x03 Kibana安装

  1. 下载kibana5.4.1版本,下载地址:https://www.elastic.co/downloads/kibana
  2. 解压: tar xf kibana-5.4.1
  3. 修改配置conf/kibana.conf
  • 绑定IP+port
server.port: 5601
server.host: "10.10.44.202"
  • 连接ElasticSearch地址
elasticsearch.url: "http://10.10.44.202:9200"
  1. 运行
[root@bogon kibana-5.4.1]# bin/kibana
  log   [07:08:09.275] [info][status][plugin:kibana@5.4.1] Status changed from uninitialized to green - Ready
  log   [07:08:09.414] [info][status][plugin:elasticsearch@5.4.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [07:08:09.470] [info][status][plugin:console@5.4.1] Status changed from uninitialized to green - Ready
  log   [07:08:09.515] [info][status][plugin:elasticsearch@5.4.1] Status changed from yellow to green - Kibana index ready
  log   [07:08:09.516] [info][status][plugin:metrics@5.4.1] Status changed from uninitialized to green - Ready
  log   [07:08:09.716] [info][status][plugin:timelion@5.4.1] Status changed from uninitialized to green - Ready
  log   [07:08:09.722] [info][listening] Server running at http://192.168.2.102:5601
  log   [07:08:09.723] [info][status][ui settings] Status changed from uninitialized to green - Ready

正常运行打印日志,如果启动失败会有红色日志打印:Status changed from uninitialized to red

  1. 访问
http://localhost:5601

推荐阅读更多精彩内容