Hadoop 搭建Sqoop环境

1411625392886_.pic_hd.jpg

环境

  • CentOS 6.8 64位 1核 2GB

  • JDK 1.7.0_75 64 位

  • Hadoop 1.1.2

  • Sqoop 1.4.5

安装Sqoop

  • 下载 sqoop 安装包
$ wget http://archive.apache.org/dist/sqoop/1.4.5/sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz
  • 解压 sqoop 安装包
$ tar -xzf sqoop-1.4.5.bin__hadoop-1.0.0.tar.gz
$ mv sqoop-1.4.5.bin__hadoop-1.0.0 /app/sqoop-1.4.5

解压后,将 sqoop-1.4.5.bin__hadoop-1.0.0 移动到 /app 目录下。

  • 修改 /etc/profile 文件
$ sudo vi /etc/profile
  • 添加 sqoop 环境变量
export SQOOP_HOME=/app/sqoop-1.4.5
export PATH=$PATH:$SQOOP_HOME/bin
  • 保存生效
$ source /etc/profile
  • 验证安装成功
$ sqoop version
Warning: $HADOOP_HOME is deprecated.

21/07/03 21:01:56 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
Sqoop 1.4.5
git commit id 5b34accaca7de251fc91161733f906af2eddbe83
Compiled by abe on Fri Aug  1 11:15:29 PDT 2014

配置Sqoop

  • 将 mysql-connector-java-5.1.22-bin.jar 复制到 /app/sqoop-1.4.5/lib 目录下。
$ cp /home/yohann/mysql-connector-java-5.1.22-bin.jar /app/sqoop-1.4.5/lib/
  • 进入 /app/sqoop-1.4.5/bin 目录
$ cd /app/sqoop-1.4.5/bin
  • 修改 configure-sqoop 文件
#if [ ! -d "${HBASE_HOME}" ]; then
#  echo "Warning: $HBASE_HOME does not exist! HBase imports will fail."
#  echo 'Please set $HBASE_HOME to the root of your HBase installation.'
#fi

## Moved to be a runtime check in sqoop.
#if [ ! -d "${HCAT_HOME}" ]; then
#  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
#  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
#fi

#if [ ! -d "${ACCUMULO_HOME}" ]; then
#  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
#fi
#if [ ! -d "${ZOOKEEPER_HOME}" ]; then
#  echo "Warning: $ZOOKEEPER_HOME does not exist! Accumulo imports will fail."
#  echo 'Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.'
#fi

注释掉 HBase 和 Zookeeper 等检查。

  • 进入 /app/sqoop-1.4.5/conf 目录
$ cd /app/sqoop-1.4.5/conf
  • 复制 sqoop-env-template.sh 命名为 sqoop-env.sh
$ cp sqoop-env-template.sh sqoop-env.sh
  • 修改 sqoop-env.sh 文件
export HADOOP_COMMON_HOME=/app/hadoop-1.1.2
export HADOOP_MAPRED_HOME=/app/hadoop-1.1.2

设置 hadoop 运行程序所在路径。

  • 保存生效
$ source /etc/profile
  • 验证配置完成
sqoop help