所谓ELKK 是指 Elasticsearch, Logstash, Kibana, Kafka 这4个开源日志分析，收集, 分析展现和传输工具。

ELKK

由于 Logstash 是基于 JRuby 来实现的，所以为避免对服务器的资源占用，现在大多推荐用 FileBeat 之类的 beats 来收集 metrics

Beats + ELK

下面我们分别介绍一下这几个组件，并搭建一个数据传输管道 Metrics data pipeline:

1. ElasticSearch

基本概念

• Cluster 集群
  存储索引数据的节点的集合

• Node 节点
  一个 ElasticSearch 的运行实例, 根据 node 的 master 属性和 data 属性不同, 可以分为以下三种类型的节点

  • 主节点: node.master=true, node.data=false
  • 数据节点: node.master=false, node.data=true
  • 路由节点: node.master=false, node.data=false

• Document 文档
  被索引的信息的基本单位,它可表示一个 Json 文档

• Mapping 映射
  文档中字段的定义称为为映射

• Index 索引
  具有共同特征的数据集, 包含许多个映射

• Type 类型
  对索引的逻辑分区, 一个索引可以有多个类型, 在新的6.0之后的版本去除了这一概念

• Shards & Replicas 分片和副本
  索引文件会分别存储在一些分片(shards) 中, 并且复制到不同节点上相应的副本(Replicas)中, 从而保证了 Elasticsearch 的高可用性

安装

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz

tar xvfz elasticsearch-6.2.4.tar.gz

cd elasticsearch-6.2.4

sed -i 's/#network.host: 192.168.0.1/network.host: 10.224.77.184/g'  ./config/elasticsearch.yml 

新版本的 ElasticSearch 不允许由 root 用户启动，并且对文件句柄打开个数有所要求，所以要做些改动, 放开文件句柄数量限制，且不能以 root 用户启动

ulimit -n 262144
sysctl -w vm.max_map_count=262144

groupadd elsearch

useradd elsearch -g elsearch -p elsearch

chown -R elsearch:elsearch /opt/elasticsearch
su elsearch 
cd /opt/elasticsearch
./bin/elasticsearch -d

启动后可以直接访问http://localhost:9200/来看是否正常服务以及角色、集群名等信息：

{
    name: "node-client-001",
    cluster_name: "jiafu-es",
    cluster_uuid: "WzUQifOuRg28zlTfS33Rjg",
    version: 
    {
        number: "6.2.4",
        build_hash: "ccec39f",
        build_date: "2018-04-12T20:37:28.497551Z",
        build_snapshot: false,
        lucene_version: "7.2.1",
        minimum_wire_compatibility_version: "5.6.0",
        minimum_index_compatibility_version: "5.0.0"
    },
    tagline: "You Know, for Search"
}

以后台方式启动

./bin/elasticsearch -d

curl 'http://10.224.77.184:9200/?pretty'

如果发起一个查询

curl -X GET "http://10.224.76.179:9200/_search" -H 'Content-Type: application/json' -d'
{
    "query": {
        "query_string" : {
            "default_field" : "content",
            "query" : "this AND that OR thus"
        }
    }
}
'

使用

ElasticSearch 提供了易于使用的 REST API

1. 查询节点

$ curl -XGET 'http://10.224.77.184:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.224.76.179           32          94  12    0.47    0.26     0.23 mdi       *      czVTNso

2. 查询集群健康状态

$ curl -XGET 'http://10.224.77.184:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 51,
  "active_shards" : 51,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 51,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

3. 创建索引

curl -XPUT 'http://10.224.77.184:9200/booklib/?pretty’

4. 创建索引, 类型及文档

# curl -XPUT 'http://10.224.77.184:9200/booklib/walter/1?pretty' -d '{ "title": "posa1"}'
{
  "_index" : "booklib",
  "_type" : "walter",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}

5. 获取文档

curl -XGET 'http://10.224.77.184:9200/booklib/walter/1?pretty'

6. 查询文档

curl -XPOST 'http://10.224.77.184:9200/booklib/_search' -d '{ "query": {"match_all": {}}, "size": 3}'
{"took":13,"timed_out":false,"_shards”:
{"total":5,"successful":5,"skipped":0,"failed":0},
"hits":{"total":1,"max_score":1.0,
"hits":[{"_index":"booklib","_type":"walter","_id":"1","_score":1.0,"_source":{ "title": "posa1"}}]}}

7 .删除文档

curl -XDELETE 'http://10.224.77.184:9200/booklib/walter/1?pretty'

实例

{
  "timestamp": "2018-02-20T01:36:30.255Z",    
  "version": "1.0",
  "ip": "10.224.11.22",
  "host": "potato01.fanyamin.com",
  "service": "potato",
  "feature": "create_potato",
  "pool": "china_east",
  "properties": {
        "url": "http://potato01/api/v1.0/meetings",
        "method": "post",
        "potatoId": "12345",
        "responseCode": 200,
        "responseTimeInMs": 1000,
        "isSuccess": true,
        "errorCode": 0,
        "errorReason": ""
      
  }
}

2. LogStash

Logstash是一个开源的服务器端数据处理管道工具，它可以同时从多个源中提取数据，对其进行转换，然后将其发送到您最喜欢的目的地, 最常见的就是 ElasticSearch, 在实际应用中, 为了避免在流量高峰对 ElasticSearch 的并发请求过多, 常用 Kafka 来中转一下.

Logstash 的优点就是插件丰富, 可以满足你的大多数需求, 插件分为三类

1. Input: 输入插件
2. Output: 输出插件
3. Filter: 过滤器插件

• 安装

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.3.1.tar.gz
tar xvfz logstash-7.3.1.tar.gz
mv logstash-7.3.1 /opt/logstash
cd /opt/logstash/
vi logstash.conf
./bin/logstash -f logstash.conf &

在实际应用中，logstash 可以部署在应用服务器上，这时应该尽量少做耗费CPU 的过滤解析，将日志收集并发送到 Kafka 上就好, 配置文件 logstash.conf 示例如下

• 1）在应用服务器上安装的 logstash， 将日志文件中的日志数据收集并发送到 kafka 上

input {

  file {
    type => "potato-web-service"
    path => [ "/opt/potato/logs/potato-web.metrics*.log" ]
    add_field => [ "component", "potato-web" ]
    add_field => [ "eventtype", "metrics" ]
    start_position => "beginning"
  }

   file {
    type => "potato-scheduler-service"
    path => [ "/opt/potato/logs/potato-scheduler.metrics*.log" ]
    add_field => [ "component", "potato-scheduler" ]
    add_field => [ "eventtype", "metrics" ]
    start_position => "beginning"
  }

  file {
    type => "potato-service"
    path => [ "/opt/potato/logs//potato.metrics*.log" ]
    add_field => [ "component", "potato-service" ]
    add_field => [ "eventtype", "metrics" ]
    start_position => "beginning"
  }

}
filter {

       if [eventtype] == 'metrics'  {
        json {
            source => "message"
            target => "message"
        }
    }

}

output {
     if [eventtype] == 'metrics' {
          kafka {
            topic_id => "metrics_%{component}"
            client_id => "potato"
            retry_backoff_ms => 60000

            bootstrap_servers => "10.224.77.178:9092"

            codec => "json"
            reconnect_backoff_ms => 60000
         }
    }
    stdout {
    }
}

• 2）在日志分析服务器安装的 logstash , 将日志从 Kafka 中取出，发送到 ElaticSearch中

input {
  kafka {
    bootstrap_servers => "10.224.77.178:9092"
    topics => ["metrics_potato-scheduler" ,"metrics_potato-service","metrics_potato-web"]
    auto_offset_reset => "earliest"
    enable_auto_commit => "false"
    codec => "json"
    consumer_threads => 1
    max_poll_records => "10"

    }
}

filter {
    json {
        source => "message"
        target => "message"
        skip_on_invalid_json => true
    }
}

output {


        elasticsearch {

          hosts =>"10.224.77.176:9200"

          codec => "json"
        }

        stdout {}
}

3. Kafka

Kafka 是现在最为流行的消息队列系统, 它称自己为分布的数据流平台.

一个数据流平台具备三个关键能力:

1. 发布和订阅数据流记录，类似于消息队列或企业消息传递系统。
2. 以容错的持久方式存储数据流记录。
3. 当记录产生时处理数据流记录。

Kafka 一般用于两大类应用程序

1. 构建可在系统或应用程序之间, 可靠地获取数据的实时数据流管道
2. 构建可实时转换或处理数据的实时数据流应用程序

Kafka 的核心概念

• Kafka 运行在一个或多个可跨多个数据中心的服务器所组成的集群上
• Kafka 集群分类存储数据流的记录, 这个分类称为主题 topic.
• 每条记录包含一个 key, 一个 value, 和一个 timestamp

Kafka 应用实例

1. 先安装 Zookeeper

• 1）下载安装包

wget http://apache.org/dist/zookeeper/stable/apache-zookeeper-3.5.5-bin.tar.gz

• 2）解压

tar xvfz [apache-zookeeper-3.5.5-bin.tar.gz](http://apache.org/dist/zookeeper/stable/apache-zookeeper-3.5.5-bin.tar.gz)

• 3）创建相关目录和配置

mkdir -p /opt

mv apache-zookeeper-3.5.5-bin /opt/zookeeper
cd /opt/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
mkdir -p /opt/data/zookeeper
vi zoo.cfg

在 zoo.cfg 文件将数据目录修改为 dataDir=/opt/zookeeper/data

• 4）启动

cd zookeeper/bin/
./zkServer.sh start

安装 Kafka

1. 下载

wget http://apache.claz.org/kafka/2.3.0/kafka_2.12-2.3.0.tgz

2. 解压

tar xvfz kafka_2.12-2.3.0.tgz

3. 创建相关目录和修改配置

mv kafka_2.12-2.3.0 /opt/kafka
cd /opt/kafka/
vi config/server.properties

## 修改配置文件如下

log.dirs=/opt/logs/kafka-logs
listeners = PLAINTEXT://10.224.77.178:9092

mkdir -p /opt/logs/kafka-logs

4. 启动 Kafka

./bin/kafka-server-start.sh -daemon config/server.properties

5. 创建一个测试主题

./bin/kafka-topics.sh -create -zookeeper localhost:2181 --replication-factor 1 -partitions 1 -topic mlogs

6. 获取这个测试主题

./bin/kafka-topics.sh -list -zookeeper localhost:2181

./kafka-topics.sh --bootstrap-server 10.224.77.178:9092 --list

4. Kibana

Kibana 是一个基于 Elastic Search 的可视化工具, 通过调用 Elastic Search 的 API , 可以方便地搜索存储在 Elastic Search 的数据, 并可以绘制各种图表和仪表盘.

Kibana dashboard 的构建主要可以分为以下三部分

• Usage 用量
• Performance 性能
• Error 错误

打开 http://10.224.76.176:5601

ELK的容器化

也可以用 docker 来安装启动一个 ELK 镜像
镜像地址 https://hub.docker.com/r/sebp/elk/
详细文档见 http://elk-docker.readthedocs.io/
源文件见 https://github.com/spujadas/elk-docker

看看它的 dockerfile , 这里包括了详细的安装配置过程, 我加了点中文注释

# Dockerfile for ELK stack
# Elasticsearch, Logstash, Kibana 6.3.1

# Build with:
# docker build -t <repo-user>/elk .

# Run with: 
# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk <repo-user>/elk
# phusion/baseimage 是基于Ubuntu 针对 docker 做了一些修改, 并修复了 PID 1 僵尸进程问题
FROM phusion/baseimage
MAINTAINER Sebastien Pujadas http://pujadas.net
ENV REFRESHED_AT 2017-02-28


###############################################################################
#                                INSTALLATION
###############################################################################
# 先安装一些依赖工具, cURL, JDK 不用说了, gosu(高手) 一门新的JVM脚本语言, tzdata 是修改时区的命令
### install prerequisites (cURL, gosu, JDK, tzdata)

ENV GOSU_VERSION 1.10

ARG DEBIAN_FRONTEND=noninteractive
RUN set -x \
 && apt-get update -qq \
 && apt-get install -qqy --no-install-recommends ca-certificates curl \
 && rm -rf /var/lib/apt/lists/* \
 && curl -L -o /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture)" \
 && curl -L -o /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture).asc" \
 && export GNUPGHOME="$(mktemp -d)" \
 && gpg --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
 && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
 && rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc \
 && chmod +x /usr/local/bin/gosu \
 && gosu nobody true \
 && apt-get update -qq \
 && apt-get install -qqy openjdk-8-jdk tzdata \
 && apt-get clean \
 && set +x

# 首先都是一样, 先从官网上下载压缩包后解压缩, 添加用户及组, 创建相应目录, 并设置权限

ENV ELK_VERSION 6.3.1

### install Elasticsearch 

ENV ES_VERSION ${ELK_VERSION}
ENV ES_HOME /opt/elasticsearch
ENV ES_PACKAGE elasticsearch-${ES_VERSION}.tar.gz
ENV ES_GID 991
ENV ES_UID 991
ENV ES_PATH_CONF /etc/elasticsearch
ENV ES_PATH_BACKUP /var/backups

RUN mkdir ${ES_HOME} \
 && curl -O https://artifacts.elastic.co/downloads/elasticsearch/${ES_PACKAGE} \
 && tar xzf ${ES_PACKAGE} -C ${ES_HOME} --strip-components=1 \
 && rm -f ${ES_PACKAGE} \
 && groupadd -r elasticsearch -g ${ES_GID} \
 && useradd -r -s /usr/sbin/nologin -M -c "Elasticsearch service user" -u ${ES_UID} -g elasticsearch elasticsearch \
 && mkdir -p /var/log/elasticsearch ${ES_PATH_CONF} ${ES_PATH_CONF}/scripts /var/lib/elasticsearch ${ES_PATH_BACKUP} \
 && chown -R elasticsearch:elasticsearch ${ES_HOME} /var/log/elasticsearch /var/lib/elasticsearch ${ES_PATH_CONF} ${ES_PATH_BACKUP}
# 自启动
ADD ./elasticsearch-init /etc/init.d/elasticsearch
RUN sed -i -e 's#^ES_HOME=$#ES_HOME='$ES_HOME'#' /etc/init.d/elasticsearch \
 && chmod +x /etc/init.d/elasticsearch


### install Logstash

ENV LOGSTASH_VERSION ${ELK_VERSION}
ENV LOGSTASH_HOME /opt/logstash
ENV LOGSTASH_PACKAGE logstash-${LOGSTASH_VERSION}.tar.gz
ENV LOGSTASH_GID 992
ENV LOGSTASH_UID 992
ENV LOGSTASH_PATH_CONF /etc/logstash
ENV LOGSTASH_PATH_SETTINGS ${LOGSTASH_HOME}/config

RUN mkdir ${LOGSTASH_HOME} \
 && curl -O https://artifacts.elastic.co/downloads/logstash/${LOGSTASH_PACKAGE} \
 && tar xzf ${LOGSTASH_PACKAGE} -C ${LOGSTASH_HOME} --strip-components=1 \
 && rm -f ${LOGSTASH_PACKAGE} \
 && groupadd -r logstash -g ${LOGSTASH_GID} \
 && useradd -r -s /usr/sbin/nologin -d ${LOGSTASH_HOME} -c "Logstash service user" -u ${LOGSTASH_UID} -g logstash logstash \
 && mkdir -p /var/log/logstash ${LOGSTASH_PATH_CONF}/conf.d \
 && chown -R logstash:logstash ${LOGSTASH_HOME} /var/log/logstash ${LOGSTASH_PATH_CONF}

ADD ./logstash-init /etc/init.d/logstash
RUN sed -i -e 's#^LS_HOME=$#LS_HOME='$LOGSTASH_HOME'#' /etc/init.d/logstash \
 && chmod +x /etc/init.d/logstash


### install Kibana

ENV KIBANA_VERSION ${ELK_VERSION}
ENV KIBANA_HOME /opt/kibana
ENV KIBANA_PACKAGE kibana-${KIBANA_VERSION}-linux-x86_64.tar.gz
ENV KIBANA_GID 993
ENV KIBANA_UID 993

RUN mkdir ${KIBANA_HOME} \
 && curl -O https://artifacts.elastic.co/downloads/kibana/${KIBANA_PACKAGE} \
 && tar xzf ${KIBANA_PACKAGE} -C ${KIBANA_HOME} --strip-components=1 \
 && rm -f ${KIBANA_PACKAGE} \
 && groupadd -r kibana -g ${KIBANA_GID} \
 && useradd -r -s /usr/sbin/nologin -d ${KIBANA_HOME} -c "Kibana service user" -u ${KIBANA_UID} -g kibana kibana \
 && mkdir -p /var/log/kibana \
 && chown -R kibana:kibana ${KIBANA_HOME} /var/log/kibana

ADD ./kibana-init /etc/init.d/kibana
RUN sed -i -e 's#^KIBANA_HOME=$#KIBANA_HOME='$KIBANA_HOME'#' /etc/init.d/kibana \
 && chmod +x /etc/init.d/kibana


###############################################################################
#                               CONFIGURATION
###############################################################################
# 配置比较麻烦, 预先在`  
### configure Elasticsearch

ADD ./elasticsearch.yml ${ES_PATH_CONF}/elasticsearch.yml
ADD ./elasticsearch-default /etc/default/elasticsearch
RUN cp ${ES_HOME}/config/log4j2.properties ${ES_HOME}/config/jvm.options \
    ${ES_PATH_CONF} \
 && chown -R elasticsearch:elasticsearch ${ES_PATH_CONF} \
 && chmod -R +r ${ES_PATH_CONF}

### configure Logstash

# certs/keys for Beats and Lumberjack input
RUN mkdir -p /etc/pki/tls/certs && mkdir /etc/pki/tls/private
ADD ./logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt
ADD ./logstash-beats.key /etc/pki/tls/private/logstash-beats.key

# filters
ADD ./02-beats-input.conf ${LOGSTASH_PATH_CONF}/conf.d/02-beats-input.conf
ADD ./10-syslog.conf ${LOGSTASH_PATH_CONF}/conf.d/10-syslog.conf
ADD ./11-nginx.conf ${LOGSTASH_PATH_CONF}/conf.d/11-nginx.conf
ADD ./30-output.conf ${LOGSTASH_PATH_CONF}/conf.d/30-output.conf

# patterns
ADD ./nginx.pattern ${LOGSTASH_HOME}/patterns/nginx
RUN chown -R logstash:logstash ${LOGSTASH_HOME}/patterns

# Fix permissions
RUN chmod -R +r ${LOGSTASH_PATH_CONF}

### configure logrotate

ADD ./elasticsearch-logrotate /etc/logrotate.d/elasticsearch
ADD ./logstash-logrotate /etc/logrotate.d/logstash
ADD ./kibana-logrotate /etc/logrotate.d/kibana
RUN chmod 644 /etc/logrotate.d/elasticsearch \
 && chmod 644 /etc/logrotate.d/logstash \
 && chmod 644 /etc/logrotate.d/kibana


### configure Kibana

ADD ./kibana.yml ${KIBANA_HOME}/config/kibana.yml


###############################################################################
#                                   START
###############################################################################

ADD ./start.sh /usr/local/bin/start.sh
RUN chmod +x /usr/local/bin/start.sh

EXPOSE 5601 9200 9300 5044
VOLUME /var/lib/elasticsearch

CMD [ "/usr/local/bin/start.sh" ]

用法

sudo docker pull sebp/elk
sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

启动之后, 可以看到

也可以用 docker-compose 来启动

$vi docker-compose.yml 

elk:
  image: sebp/elk
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"

参考资料

• ElasticSearch 官方文档
• ELK document
• Docker PID1 僵尸进程问题