Hadoop 搭建Flume环境

1551627192148_.pic.jpg

环境

  • CentOS 6.8 64位 1核 2GB

  • JDK 1.7.0_75 64 位

  • Hadoop 1.1.2

  • Flume 1.5.2

安装Flume

  • 下载 flume 安装包
$ wget http://archive.apache.org/dist/flume/1.5.2/apache-flume-1.5.2-bin.tar.gz
  • 解压 flume 安装包
$ tar -xzf apache-flume-1.5.2-bin.tar.gz
$ mv apache-flume-1.5.2-bin /app/flume-1.5.2

解压后,将 apache-flume-1.5.2-bin 移动到 /app 目录下,并命名为 flume-1.5.2。

  • 修改 /etc/profile 文件
$ sudo vi /etc/profile
  • 添加 flume 环境变量
export FLUME_HOME=/app/flume-1.5.2
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=$PATH:$FLUME_HOME/bin
  • 保存生效
$ source /etc/profile

配置Flume

  • 进入 /app/flume-1.5.2/conf 目录
$ cd /app/flume-1.5.2/conf
  • 复制 flume-env.sh.template 命名为 flume-env.sh
$ cp flume-env.sh.template flume-env.sh
  • 修改 flume-env.sh 文件
JAVA_HOME= /app/lib/jdk1.7.0_55
JAVA_OPTS="-Xms100m -Xmx200m -Dcom.sun.management.jmxremote"

修改以上两项配置。

验证配置完成

  • 复制 flume-conf.properties.template 命名为 flume-conf.properties
$ cp flume-conf.properties.template flume-conf.properties
  • 修改 flume-conf.properties 文件
# The configuration file needs to define the sources, the channels and the sinks.
# Sources, channels and sinks are defined per agent, in this case called 'a1'
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# For each one of the sources, the type is defined
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

#The channel can be defined as follows.
a1.sources.r1.channels = c1
# Each sink's type must be defined
a1.sinks.k1.type = logger

#Specify the channel the sink should use
a1.sinks.k1.channel = c1

# Each channel's type is defined.
a1.channels.c1.type = memory
# Other config values specific to each type of channel(sink or source)
# can be defined as well
# In this case, it specifies the capacity of the memory channel
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
  • 在 /app/flume-1.5.2/conf 目录执行
$ ./bin/flume-ng agent --conf ./conf/ --conf-file  ./conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console
  • 再打开一个终端,执行
$ curl localhost:44444
OK

推荐阅读更多精彩内容