Hadoop分布式安装

目的

3台虚拟机实现1个namenode,2个datanode的hadoop集群。

环境

硬件:MAC * 1;Parallels Desktop虚拟机*3:CentOS-7

使用ssh客户端操作3台机器

JDK版本

[keon@centos-linux bin]$ /usr/jdk1.8.0_101/bin/java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

HADOOP版本

[keon@centos-linux bin]$ ./hdfs version
Hadoop 2.7.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41
Compiled by jenkins on 2016-01-26T00:08Z
Compiled with protoc 2.5.0
From source with checksum d0fda26633fa762bff87ec759ebe689c
This command was run using /usr/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar

配置过程

1.修改host

[keon@centos-linux bin]$ vim /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.211.55.6 h1
10.211.55.7 h3
10.211.55.8 h2

2.配置免密码

在h1生成密匙

[keon@centos-linux bin]$ ssh-keygen -t rsa
[keon@centos-linux ~]$ ls .ssh/
id_rsa id_rsa.pub known_hosts

在h2,h3拷贝公匙

scp keon@h1:/home/keon/.ssh/id_rsa.pub /home/keon/.ssh/authorized_keys

在h1验证是否免密码成功

[keon@centos-linux ~]$ ssh h2
Last login: Tue Aug 2 10:34:14 2016 from h1
[keon@centos-linux ~]$ exit
logout
Connection to h2 closed.
[keon@centos-linux ~]$ ssh h3
Last login: Tue Aug 2 10:34:25 2016 from h1

3.修改配置文件

[keon@centos-linux hadoop-2.7.2]$ pwd
/usr/hadoop-2.7.2

(1)etc/hadoop/hadoop-env.sh

[keon@centos-linux hadoop-2.7.2]$ vim etc/hadoop/hadoop-env.sh

修改java环境地址

export JAVA_HOME=/usr/jdk1.8.0_101

(2)etc/hadoop/core-site.xml

<configuration>
<property>

<name>fs.default.name</name>
<value>hdfs://h1:9000/</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop-2.7.2/tmp</value>
</property>
</configuration>

(3)etc/hadoop/hdfs-site.xml

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/hadoop-2.7.2/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/hadoop-2.7.2/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>

(4)etc/hadoop/mapred-site.xml

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

修改完后,推送到每一个节点

启动测试

1.格式化namenode

[keon@centos-linux hadoop-2.7.2]$ ./bin/hdfs namenode -format

16/08/02 11:25:45 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = h1/10.211.55.6
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.2
STARTUP_MSG: classpath = /usr/hadoop-2.7.2/etc
.
.
.
16/08/02 11:25:47 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/08/02 11:25:47 INFO util.GSet: VM type = 64-bit
16/08/02 11:25:47 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/08/02 11:25:47 INFO util.GSet: capacity = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /usr/hadoop-2.7.2/namenode ? (Y or N) y
16/08/02 11:25:49 INFO namenode.FSImage: Allocated new BlockPoolId: BP-566776757-10.211.55.6-1470108349176
16/08/02 11:25:49 INFO common.Storage: Storage directory /usr/hadoop-2.7.2/namenode has been successfully formatted.
16/08/02 11:25:49 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/08/02 11:25:49 INFO util.ExitUtil: Exiting with status 0
16/08/02 11:25:49 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at h1/10.211.55.6
************************************************************/

2.启动服务
启动hdfs

./sbin/start-dfs.sh

Starting namenodes on [h1]
h1: starting namenode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-namenode-centos-linux.shared.out
h3: starting datanode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-datanode-centos-linux.shared.out
h2: starting datanode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-datanode-centos-linux.shared.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.7.2/logs/hadoop-keon-secondarynamenode-centos-linux.shared.out

启动yarn

./sbin/start-yarn.sh

starting yarn daemons
starting resourcemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-resourcemanager-centos-linux.shared.out
h3: starting nodemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-nodemanager-centos-linux.shared.out
h2: starting nodemanager, logging to /usr/hadoop-2.7.2/logs/yarn-keon-nodemanager-centos-linux.shared.out

3.验证服务
h1

[keon@centos-linux hadoop-2.7.2]$ /usr/jdk1.8.0_101/bin/jps
2930 Jps
1442 SecondaryNameNode
1045 NameNode
2443 ResourceManager

h2

[keon@centos-linux hadoop-2.7.2]$ /usr/jdk1.8.0_101/bin/jps
7703 DataNode
29240 Jps
8286 NodeManager

h3

[root@centos-linux keon]# /usr/jdk1.8.0_101/bin/jps
11584 NodeManager
12273 Jps
10346 DataNode

相关进程已启动

推荐阅读更多精彩内容

  • 一、系统参数配置优化 1、系统内核参数优化配置 修改文件/etc/sysctl.conf,添加如下配置,然后执行s...
    张伟科阅读 3,166评论 0 14
  • 之前的有点忘记了,这里在云笔记拿出来再玩玩.看不懂的可以留言 大家可以尝试下Ambari来配置Hadoop的相关环...
    HT_Jonson阅读 2,621评论 0 50
  • 1分布式集群规划 2 配置 2.1 机器hostname及IP,以机器131为例 关闭防火墙 修改hostname...
    aduo阅读 193评论 0 0
  • 1. 前言 断断续续写了一段时间的设计模式,终于把经典的23种设计模式全写完了。下面对这些设计模式总结一下。 2....
    java高并发阅读 3,475评论 2 78
  • 姓名:王琦 公司:南京浩乐科技有限公司 【日精进第67天】 一、【学~勤学】 ①日常课诵 《大学开篇》《六项精进》...
    王琦精进打卡阅读 22评论 0 0
  • 18年买了ipad 和苹果 开始用电脑绘制 68元下载了软件 让我一度痴迷其中 儿子会来捣乱 干脆让他加入一同玩
    河马爸爸ELLISON阅读 375评论 4 5
  • 春夜喜雨 唐·杜甫 好雨知时节,当春乃发生。 随风潜入夜,润物细无声。 野径云...
    不忘初心662阅读 86评论 0 3
  • 刘曦_阅读 264评论 1 0