hadoop系列安装小记

字数 900阅读 38

原文3年多前发表在私人站点,现迁移到简书

当时装的是5.1.0,现在最新的版本是5.4.2,因为有在线业务使用,所以暂时不升级。

cdh

独立下载hadoop各个组件再安装比较繁琐(hdfs+yarn+hbsae+zk+hive),没有选好版本可能会冲突,CDH的版本都是选定好的,安装和升级文档齐全,非常方便

安装前配置

官方流程 大致分一下3个步骤:

配置yum源

wget http://archive.cloudera.com/cdh5/one-click-install/redhat/5/x86_64/cloudera-cdh-5-0.x86_64.rpm

sudo yum --nogpgcheck localinstall cloudera-cdh-5-0.x86_64.rpm  #安装rpm,会加一个clouder的yum源:

yum clean all 、 yum makecache # 重新构建yum缓存

sudo rpm --import http://archive.cloudera.com/cdh5/redhat/5/x86_64/cdh/RPM-GPG-KEY-cloudera #导入GPG验证的key

* 可能的问题:

1.运行yum的可能遇到错误:

It's possible that the above module doesn't match the current version of Python, which is:2.7.3 (default, May 19

2014, 15:04:50) [GCC 4.1.2 20080704 (Red Hat 4.1.2-46)]

需要修改yum的python依赖版本:

修改文件: vim /usr/bin/yum

修改头#!/usr/bin/python  => #!/usr/bin/python2.4

2.找不到host命令,需要装下bind-utils:yum install bind-utils

安装jdk

yum -y install unzip

curl -L -s get.jenv.io | bash

source /home/admin/.jenv/bin/jenv-init.sh

jenv install java 1.7.0_45

jdk通过USER账号安装,cdh系列的需要在自己的特定账号下执行,比如hdfs账号,所以会出现找不到JAVA_HOME的问题,解决方法:

  • 在/etc/sudoers 配置:Defaults env_keep+=JAVA_HOME

  • 设置ROOT下的JAVA_HOME指向USER。。,需要修改USER为可执行权限

  • 还有另一个方法,是在/etc/default/bigtop-utils 下配置javahome(chmod 755 /home/USER)


export JAVA_HOME=/home/USER/.jenv/candidates/java/current

chmod 755 /home/USER/

HDFS

安装和配置

NameNode、Client


sudo yum install hadoop-hdfs-namenode

sudo yum install hadoop-client

安装DataNode


在DataNode机器上执行:

sudo yum install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce

设置hdfs config文件到自己的目录下

sudo cp -r /etc/hadoop/conf.empty /etc/hadoop/conf.my_cluster

sudo /usr/sbin/alternatives --install /etc/hadoop/conf hadoop-conf /etc/hadoop/conf.my_cluster 50

sudo /usr/sbin/alternatives --set hadoop-conf /etc/hadoop/conf.my_cluster

sudo chmod -R 777 /etc/hadoop/conf.my_cluster

(alternatives --config java好像无效)

创建数据目录(用户组hdfs:hdfs 权限700):

datanode:sudo mkdir -p /data/hadoop/hdfs/dn

sudo chown -R hdfs:hdfs /data/hadoop

hadoop-env.sh

hadoop默认为namenode、datanode都是1G的内存:

export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Xmx3072m -verbose:gc -Xloggc:/var/log/hadoop-hdfs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -Xmx2048m -verbose:gc -Xloggc:/var/log/hadoop-hdfs/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

core-site.xml

<property>
<!-- namenode地址和端口 -->
 <name>fs.defaultFS</name>
 <value>hdfs://cdhhadoop1:8020</value>
</property>
<!-- 回收站,默认保留一天 -->
<property>
 <name>fs.trash.interval</name>
 <value>1440</value>
</property>
<property>
 <name>fs.trash.checkpoint.interval</name>
 <value>0</value>
</property>
<!-- 配置Snappy压缩 -->
 <property>
  <name>io.compression.codecs</name>
 <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>

配置hdfs-site.xml

<!-- 超级用户 -->
<property>
   <name>dfs.permissions.superusergroup</name>
   <value>admin</value>
</property>
<!-- hdfs副本 -->
<property>
   <name>dfs.replication</name>
   <value>2</value>
</property>
<!-- dfs.namenode.name.dir 作为namenode存放元信息的目录,如果设置多个则会有一个拷贝,可以在另外一台机器上搭一个NFS共享目录,作为备份 ->
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/data/hadoop/hdfs/nn</value>
  </property>
  <property>
     <name>dfs.datanode.data.dir</name>
     <value>/data/hadoop/hdfs/dn</value>
  </property>
  • 其他配置:

1.如果datanode的目录有一个写失败,DataNode就会停止,这样这个DataNode上的所有目录的副本都会增加,如果要避免这种情况,可以设置容忍失败的目录个数

2.可以配置负载均衡,默认的分配策略是随机的,可以配置一个策略比如磁盘大小

3.没有配置web hdfs

启动

  • 格式化namenode
sudo -u hdfs hdfs namenode -format
日志文件目录:/var/log/hadoop-hdfs
  • 启动namenode
sudo service hadoop-hdfs-namenode start
  • 启动datanode
sudo service hadoop-hdfs-datanode start
  • 初始化

hdfs运行以后,推荐在hdfs上创建tmp目录,并设置权限:

$ sudo -u hdfs hadoop fs -mkdir /tmp
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp

测试

http://localhost:50070/dfshealth.html#tab-overview

简单的测试只要执行下hadoop fs命令即可,如果要测试读写性能,要等mapreduce装好


【写性能测试】

hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar  TestDFSIO -write -nrFiles 10 -fileSize 1000

我们集群的一次测试结果:

----- TestDFSIO ----- : write

          Date & time: Sun Jul 13 21:40:41 CST 2014

      Number of files: 10

Total MBytes processed: 10000.0(总共10个文件,每个1G)

    Throughput mb/sec: 6.452312250618132(总大小/Map总时间)

Average IO rate mb/sec: 6.50354528427124

IO rate std deviation: 0.6099282285067701

Test exec time sec: 197.818(整体执行时间)

Throughput是总大小文/每个Map时间之和,如果算并发吞吐量的话,可以乘以Map数量,详细解读可以参考:Benchmarking and Stress Testing an Hadoop Cluster With TeraSort, TestDFSIO & Co


【读性能测试】

hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar  TestDFSIO -read -nrFiles 10 -fileSize 1000

20:38:21 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read

20:38:21 INFO fs.TestDFSIO:  Date & time: Tue Jul 15 20:38:21 CST 2014

20:38:21 INFO fs.TestDFSIO:        Number of files: 10

20:38:21 INFO fs.TestDFSIO: Total MBytes processed: 10000.0

20:38:21 INFO fs.TestDFSIO:      Throughput mb/sec: 16.79261125104954

20:38:21 INFO fs.TestDFSIO: Average IO rate mb/sec: 16.829221725463867

20:38:21 INFO fs.TestDFSIO:  IO rate std deviation: 0.8154139285912413

20:38:21 INFO fs.TestDFSIO:    Test exec time sec: 84.614

测试结果以后需要清理测试结果

hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-test.jar  TestDFSIO -clean

在windows看客户端下测试Hdfs,需要到

https://github.com/srccodes/hadoop-common-2.2.0-bin 下载并替换hadoopHome下的bin目录

YARN

安装和配置


sudo yum install hadoop-yarn-resourcemanager

sudo yum install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver

sudo mkdir -p /data/yarn/local

sudo mkdir -p /data/yarn/logs

sudo chown -R yarn:yarn /data/yarn

hadoop fs -mkdir -p /user/history

hadoop fs -chmod -R 1777 /user/history

hadoop fs -chown mapred:hadoop /user/history

yarn-site.xml

<configuration>
   <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>cdhhadoop1</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>cdhhadoop1:8088</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>
  <property>
    <description>List of directories to store localized files in.</description>
    <name>yarn.nodemanager.local-dirs</name>
    <value>file:///data/yarn/local</value>
  </property>
  <property>
    <description>Where to store container logs.</description>
    <name>yarn.nodemanager.log-dirs</name>
    <value>file:///data/yarn/logs</value>
  </property>
  <property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>hdfs:///log/yarn/apps</value>
  </property>
  <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
     <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
     </value>
  </property>

启动

  • 端口
resourceManager 8088/cluster
nodeManager  8042/node
JobHistory  19888/jobhistory
Name:http://localhost:8088/cluster/nodes
Node1:http://localhost:8042/node
Node2:http://localhost:8043/node

测试

通过hadoop自带的randowmwriter测试下:
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar randomwriter out

21:14:34 INFO mapreduce.Job: Job job_1405247153654_0004 completed successfully
21:14:34 INFO mapreduce.Job: Counters: 33

        File System Counters

                FILE: Number of bytes read=0

                FILE: Number of bytes written=1772230

                FILE: Number of read operations=0

                FILE: Number of large read operations=0

                FILE: Number of write operations=0

                HDFS: Number of bytes read=2350

                HDFS: Number of bytes written=21545727074(写入20G)

                HDFS: Number of read operations=80

                HDFS: Number of large read operations=0

                HDFS: Number of write operations=40

        Job Counters

                Killed map tasks=10

                Launched map tasks=30

                Other local map tasks=30

                Total time spent by all maps in occupied slots (ms)=7247472

                Total time spent by all reduces in occupied slots (ms)=0

                Total time spent by all map tasks (ms)=7247472

                Total vcore-seconds taken by all map tasks=7247472

                Total megabyte-seconds taken by all map tasks=7421411328

        Map-Reduce Framework

                Map input records=20

                Map output records=2043801

                Input split bytes=2350

                Spilled Records=0

                Failed Shuffles=0

                Merged Map outputs=0

                GC time elapsed (ms)=8157

                CPU time spent (ms)=641440

                Physical memory (bytes) snapshot=2889732096

                Virtual memory (bytes) snapshot=14388494336

                Total committed heap usage (bytes)=2371878912

        org.apache.hadoop.examples.RandomWriter$Counters

                BYTES_WRITTEN=21475013178

                RECORDS_WRITTEN=2043801

        File Input Format Counters

                Bytes Read=0

        File Output Format Counters

                Bytes Written=21545727074

The job took 604 seconds.

ZK

安装和配置

安装


sudo yum install zookeeper

sudo yum install zookeeper-server

拷贝配置


sudo cp -r /etc/zookeeper/conf.dist /etc/zookeeper/conf.my_cluster

sudo alternatives --verbose --install /etc/zookeeper/conf zookeeper-conf /etc/zookeeper/conf.my_cluster 50

sudo alternatives --set zookeeper-conf /etc/zookeeper/conf.my_cluster

修改配置文件并在节点间同步


/etc/zookeeper/conf.my_cluster/zoo.cfg

server.1=A:2888:3888

server.2=B:2888:3888

server.3=C:2888:3888

创建数据目录


mkdir -p /data/zookeeper

chown -R zookeeper:zookeeper /data/zookeeper

启动


启动日志在/var/log/zookeeper

在A运行 :

$ service zookeeper-server init --myid=1

$ service zookeeper-server start

在B运行

$ service zookeeper-server init --myid=2

$ service zookeeper-server start

在C运行

$ service zookeeper-server init --myid=3

$ service zookeeper-server start

测试


zookeeper-client -server A:2181

zookeeper-client -server B:2181

目录列表: ls /

创建目录: create /test "empty"

HBase

安装和配置

安装


所有机器上: sudo yum install hbase

NameNode:sudo yum install hbase-master

DataNode: sudo yum install hbase-regionserver

拷贝自己的配置文件


sudo cp -r /etc/hbase/conf.dist /etc/hbase/conf.my_cluster

sudo alternatives --verbose --install /etc/hbase/conf hbase-conf /etc/hbase/conf.my_cluster 50

sudo alternatives --set hbase-conf /etc/hbase/conf.my_cluster

修改最大文件数限制


避免Too many open files(/etc/security/limits.conf)

hdfs  -      nofile  32768

hbase -      nofile  32768

阿里云机器默认已经是65535,所以不做修改

hdfs DataNode会限制打开的文件数( /etc/hadoop/conf/hdfs-site.xml)

  dfs.datanode.max.xcievers

  65535

创建目录


hadoop fs -mkdir /hbase

hadoop fs -chown hbase /hbase

hbase-site.xml

<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <name>hbase.rootdir</name>
  <value>hdfs://myhost:8020/hbase</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
    <value>A,B,C</value>
</property>
<!--关闭checksum-->
<property>
    <name>hbase.regionserver.checksum.verify</name>
    <value>false</value>
    <description>
        If set to  true, HBase will read data and then verify checksums  for
        hfile blocks. Checksum verification inside HDFS will be switched off.
        If the hbase-checksum verification fails, then it will  switch back to
        using HDFS checksums.
    </description>
  </property>
<property>
    <name>hbase.hstore.checksum.algorithm</name>
    <value>NULL</value>
    <description>
      Name of an algorithm that is used to compute checksums. Possible values
      are NULL, CRC32, CRC32C.
    </description>
  </property>

启动


service hbase-master start

service hbase-regionserver start

测试


60010是master的端口 http://localhost:60010/master-status?filter=all

60030是regionServer的端口

测试hbase集群是否支持snappy:

hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://namenode:8020/benchmarks/hbase snappy

通过hbase shell访问hbase

Hive

安装和配置

安装hive/metastore/hieveserver


sudo yum install -y hive

sudo yum install -y hive-metastore

sudo yum install -y hive-server2

mysql-connector-java.jar


在metastore的机器,把mysql-connector-java.jar放到/usr/lib/hive/lib/目录下

java堆配置

我们配置的是3G


官方文档有误,实际配置文件是:/etc/hive/conf/hive-env.sh

if [ "$SERVICE" = "hiveserver2或者metastore" ]; then

  export HADOOP_OPTS="${HADOOP_OPTS} -Xmx3072m -Xms1024m -Xloggc:/var/log/hive/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps"

fi

export HADOOP_HEAPSIZE=512

metastore配置(配置文件:hive-site.xml)

参考

metastore 配置hdfs


先初始化下hdfs得配置,再从namenode把最新的配置拷过来:scp /etc/hadoop/conf.my_cluster/hdfs-site.xml /etc/hadoop/conf.my_cluster/core-site.xml host:/etc/hadoop/conf.my_cluster/

hiveserver2配置(配置文件:/etc/hive/conf/hive-site.xml)

主要是配置metastore地址,zk地址


  hive.support.concurrency

  Enable Hive's Table Lock Manager Service

  true

  hive.zookeeper.quorum

  Zookeeper quorum used by Hive's Table Lock Manager

  A,B,C

  hive.metastore.local

  false

  hive.metastore.uris

  thrift://xxxxx:9083

启动


sudo /sbin/service hive-metastore start

sudo /sbin/service hive-server2 start

测试

  • 1./usr/lib/hive/bin/beeline

  • 2.!connect jdbc:hive2://localhost:10000 username password org.apache.hive.jdbc.HiveDriver

    或者: !connect jdbc:hive2://10.241.52.161:10000 username password org.apache.hive.jdbc.HiveDriver

  • 3.show tables;

hive服务端日志在:/var/log/hive

hive shell日志在/tmp/admin/hive.log,之前有个配置错误引起的异常,一直没找到日志,原来路径是在/etc/hive/conf下的log4j配置的

参考

推荐阅读更多精彩内容