Flume：安装、部署、及flume的案例

一、 Flume安装

1）准备
       jdk:openjdk8
       hadoop:hadoop-2.6.0-cdh5.14.0
       在cloudera官网下载cdh5版本的flume安装包

$ tar -zxvf flume-ng-1.6.0-cdh5.14.0.tar.gz -C /opt/cloudera/  # 解压到/opt/cloudera目录下
$ cd /opt/cloudera
$ sudo mv flume-ng-1.6.0-cdh5.14.0 flume       # 将文件夹名改flume       
$ sudo chown -R hadoop ./flume            # 修改文件权限

二、修改配置文件

$ cp conf/flume-env.sh.template conf/flume-env.sh
$ sudo vim conf/flume-env.sh

在配置文件flume-env.sh中，修改JAVA_HOME路径：

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64  #换成你的java路径

三、验证安装是否成功

$ cd /opt/cloudera/flume
$ ./bin/flume-ng version

结果显示如下，则证明安装成功：

Flume 1.6.0-cdh5.14.0
Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
Revision: 939f6066f6056e7e8042292647eb5032628ac5a0
Compiled by jenkins on Sat Jan  6 13:37:47 PST 2018
From source with checksum c350591306508e3cdd4942fb1debefc9

四、 Flume小例子(Netcat )

任务介绍：NetCat Source：监听一个指定的网络端口，即只要应用程序向这个端口里面写数据，这个source组件就可以获取到信息。

①.创建你自己的属性文件，基于现有的模板。

$ cp conf/flume-conf.properties.template conf/flume.conf

②.修改conf/flume.conf文件


文件内容如下：
# Name the components on this agent

a1.sources = r1

a1.sinks = k1

a1.channels = c1
# Describe configure the source


a1.sources.r1.type = netcat

a1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

# Describe the sink

a1.sinks.k1.type = logger
# Use a channel which buffers events in memory

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactionCapacity = 100

# 通过channel将source与sink连接起来

a1.sources.r1.channels = c1

a1.sinks.k1.channel = c1

五、 ubuntu安装telnet

1. 首先更新一下

$ sudo apt-get update

2. 安装

$ sudo apt-get install xinetd telnetd

3.修改/etc/xinetd.conf，并加入以下内容：

# Simple configuration file for xinetd 
# Some defaults, and include /etc/xinetd.d/ 
defaults 
{ 
# Please note that you need a log_type line to be able to use log_on_success 
# and log_on_failure. The default is the following : 
# log_type = SYSLOG daemon info 
instances = 60 
log_type = SYSLOG authpriv 
log_on_success = HOST PID 
log_on_failure = HOST 
cps = 25 30 
} 
includedir /etc/xinetd.d

4.修改/etc/xinetd.d/telnet并加入以下内容：

# default: on 
# description: The telnet server serves telnet sessions;it uses 
# unencrypted username/password pairs for authentication. 
service telnet 
{ 
disable = no 
flags = REUSE 
socket_type = stream 
wait = no 
user = root 
server = /usr/sbin/in.telnetd 
log_on_failure += USERID 
}

5.重启机器或重启网络服务

### 1. 重启系统
$ sudo reboot
### 2. 重启网络服务
$ sudo /etc/init.d/xinetd restart

6.测试telnet是否安装成功

### 1. win10下使用cmd窗口执行如下命令：
$ telnet ip地址

六、启动agent的shell操作：

$ cd /opt/cloudera/flume
$ ./bin/flume-ng agent -n a1 -c conf -f conf/flume.conf -Dflume.root.logger=DEBUG,console

结果显示如下,则证明运行成功：

2018-08-13 19:35:41,475 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: []
2018-08-13 19:35:41,476 (conf-file-poller-0) [WARN - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:135)] No configuration found for this host:a1
2018-08-13 19:35:41,492 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

七、使用telnet发送消息

### 查看当前开启的端口号
$ netstat -tnlp

结果显示如下：

tcp6       0      0 127.0.0.1:44444         :::*                    LISTEN      2124/java

### 监听端口号本机44444
$ telnet localhost 44444