title: sqoop2 配置
tags: sqoop2,环境配置,1.99.7,官网
grammar_cjkRuby: true
1 Server installation
2 解压到 /usr/tools 文件夹下
tar -zxvf sqoop-1.99.7-bin-hadoop200.tar.gz -C /usr/tools
3 Hadoop dependencies
If the environment $HADOOP_HOME is set, Sqoop will usee the following locations: $HADOOP_HOME/share/hadoop/common, $HADOOP_HOME/share/hadoop/hdfs, $HADOOP_HOME/share/hadoop/mapreduce and $HADOOP_HOME/share/hadoop/yarn.
如果你已经在/etc/profile中配置了 $HADOOP_HOME那你将不需要额外配置以上参数
4 Hadoop configuration
core-site.xml 添加
<property>
<name>hadoop.proxyuser.sqoop2.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.sqoop2.groups</name>
<value>*</value>
</property>
yarn-site.xml 添加
如果，不加上面的属性，会产生什么问题呢？可能在数据迁移的过程中出现“Aggregation is not enabled”的问题，导致数据迁移失败
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
配置完后需要重新启动hadoop
5 Third party jars 第三方数据库驱动配置
mkdir /usr/tools/sqoop-1.99.7-bin-hadoop200/extralib
cp mysql-jdbc*.jar /usr/tools/sqoop-1.99.7-bin-hadoop200/extralib
6 Configuring PATH 修改/etc/profile
#sqoop
export SQOOP_HOME=/usr/tools/sqoop-1.99.7-bin-hadoop200
export SQOOP_SERVER_EXTRA_LIB=$SQOOP_HOME/extralib/
export PATH=$PATH:$SQOOP_HOME/bin
不要忘了source 生效
7 Configuring Server 服务器参数配置
路径/usr/tools/sqoop-1.99.7-bin-hadoop200 下面的sqoop.properties文件
143 # Hadoop configuration directory
144 org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/hadoop-2.6.4/etc/hadoop
145
元数据数据库，换成mysql需要在这个地方进行配置。暂时不用修改使用系统自带的derby数据库
# JDBC repository provider configuration
org.apache.sqoop.repository.jdbc.handler=org.apache.sqoop.repository.derby.DerbyRepositoryHandler
org.apache.sqoop.repository.jdbc.transaction.isolation=READ_COMMITTED
org.apache.sqoop.repository.jdbc.maximum.connections=10
org.apache.sqoop.repository.jdbc.url=jdbc:derby:@BASEDIR@/repository/db;create=true
org.apache.sqoop.repository.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
org.apache.sqoop.repository.jdbc.user=sa
org.apache.sqoop.repository.jdbc.password=
为了将日志输出到/usr/tools/sqoop-1.99.7-bin-hadoop200/logs
进入末行模式 将@LOGDIR@全部替换为../logs
:% s/@LOGDIR@/..\/logs/g
8 验证 sqoop2-tool verify
[root@master conf]# sqoop2-tool verify
Setting conf dir: /usr/tools/sqoop-1.99.7-bin-hadoop200/bin/../conf
Sqoop home directory: /usr/tools/sqoop-1.99.7-bin-hadoop200
Sqoop tool executor:
Version: 1.99.7
Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
13 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/tools/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.
9 启动服务 sqoop2-server start
[root@master logs]# sqoop2-server start
Setting conf dir: /usr/tools/sqoop-1.99.7-bin-hadoop200/bin/../conf
Sqoop home directory: /usr/tools/sqoop-1.99.7-bin-hadoop200
Starting the Sqoop2 server...
0 [main] INFO org.apache.sqoop.core.SqoopServer - Initializing Sqoop server.
13 [main] INFO org.apache.sqoop.core.PropertiesConfigurationProvider - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hadoop-2.6.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/tools/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Sqoop2 server started.
[root@master logs]# jps
7088 Jps
4982 ResourceManager
4664 NameNode
7069 SqoopJettyServer
4813 SecondaryNameNode
相关链接
