本文记录从初始系统安装Hadoop单机版的具体过程。
环境
| 系统环境 | 版本 |
|---|---|
| 操作系统 | Ubuntu 18.04 |
| Java | openjdk version "1.8.0_212" |
| Hadoop | 3.2.0 |
1. 更新APT镜像源
添加apt阿里镜像源，加速资源的下载速度，参考官方文档
- 编辑
/etc/apt/sources.list.d/aliyun.list
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
- 执行更新
sudo apt update
2. 安装相关工具
Java JDK推荐使用jdk8，jdk9以上默认禁用了许多javax接口，会导致nodemanager和resourceManager启动失败。
sudo apt install -y openjdk-8-jdk
sudo apt install -y git
sudo apt install -y vim
sudo apt install -y curl
sudo apt install -y zsh
sudo apt install -y openssh-server
3. 安装Zsh（个人爱好）
sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
chsh -s /bin/zsh
4. 新建Hadoop用户
sudo adduser hduser
5. 配置SSH权限（Hadoop用户）
su hduser
ssh-keygen -t rsa
cat ~/.sshid_rsa.pub >> ~/.sshauthorized_keys
测试
ssh localhost
6. 下载Hadoop3.2.0
其余版本链接参考清华大学开源软件镜像站
wget [https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-3.2.0.tar.gz](https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/stable2/hadoop-3.2.0.tar.gz)
7. 修改Hadoop配置文件
为了方便，首先编辑~/.bashrc，配置用户环境变量
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_HOME=/home/hduser/hadoop-family/hadoop-3.2.0
export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar
export PATH=${PATH}:${HADOOP_HOME}/bin/:${HADOOP_HOME}/sbin/
- 配置
hadoop-3.2.0/etc/hadoop/hadoop-env.sh
# The java implementation to use. By default, this environment
export JAVA_HOME=${JAVA_HOME}
- 配置
hadoop-3.2.0/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp</value>
<description>Abase for other temporary directories.</description>
</property>
</configuration>
- 配置
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>localhost:50090</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hduser/hadoop-family/hadoop-3.2.0/file-tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hduser/hadoop-family/hadoop-3.2.0/file-tmp/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
- 配置
hadoop-3.2.0/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/home/hduser/hadoop-family/hadoop-3.2.0/hdfs-tmp</value>
</property>
</configuration>
- 配置
hadoop-3.2.0/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>localhost:8088</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>localhost:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>localhost:8032</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>localhost:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>localhost:8033</value>
</property>
</configuration>
8. 格式化Hadoop文件系统
hdfs namenode -format
9. 启动Hadoop
Tips:
若出现DataNode未启动的现象，原因可能是由于hdfs-tmp/dfs/name/current/VERSION和hdfs-tmp/dfs/name/current/VERSION中的clusterID不匹配。
可以清空hdfs-tmp，并重新执行hadoop-3.2.0/bin/hdfs namenode -format。
或将hdfs-tmp/dfs/name/current/VERSION的clusterID复制进hdfs-tmp/dfs/name/current/VERSION。
start-all.sh
访问Hadoop
命令行执行jps可以看到：
命令行执行hdfs dfsadmin -report可以看到：
浏览器访问http://localhost:9870 可以看到:
浏览器访问http://localhost:8088 可以看到:
