001.Azkaban-3.x 源码编译

软件准备

Azkaban源码包，下载地址：https://github.com/azkaban/azkaban/releases
我这里使用3.76.0版本

环境准备

JDK-1.8+
Git：yum install git -y

(1) 解压源码包

# 解压后的目录结构如下：
[hadoop@beh07 azkaban-3.76.0]$ ll
total 92
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-core
drwxrwxr-x 4 hadoop hadoop    52 May 31 06:26 az-crypto
drwxrwxr-x 5 hadoop hadoop    86 May 31 06:26 az-examples
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-exec-util
drwxrwxr-x 3 hadoop hadoop    54 May 31 06:26 az-flow-trigger-dependency-plugin
drwxrwxr-x 3 hadoop hadoop    53 May 31 06:26 az-flow-trigger-dependency-type
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-hadoop-jobtype-plugin
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-hdfs-viewer
-rw-rw-r-- 1 hadoop hadoop 21925 May 31 06:26 az-intellij-style.xml
drwxrwxr-x 4 hadoop hadoop    49 May 31 06:26 az-jobsummary
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-common
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-db
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-exec-server
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 azkaban-hadoop-security-plugin
drwxrwxr-x 3 hadoop hadoop    55 May 31 06:26 azkaban-solo-server
drwxrwxr-x 3 hadoop hadoop    54 May 31 06:26 azkaban-spi
drwxrwxr-x 3 hadoop hadoop   100 May 31 06:26 azkaban-web-server
drwxrwxr-x 3 hadoop hadoop    37 May 31 06:26 az-reportal
-rw-rw-r-- 1 hadoop hadoop 10672 May 31 06:26 build.gradle
-rw-rw-r-- 1 hadoop hadoop  6409 May 31 06:26 CONTRIBUTING.md
drwxrwxr-x 3 hadoop hadoop  4096 May 31 06:26 docs
drwxrwxr-x 3 hadoop hadoop    21 May 31 06:26 gradle
-rw-rw-r-- 1 hadoop hadoop  1488 May 31 06:26 gradle.properties
-rwxrwxr-x 1 hadoop hadoop  5296 May 31 06:26 gradlew
-rw-rw-r-- 1 hadoop hadoop  2260 May 31 06:26 gradlew.bat
-rw-rw-r-- 1 hadoop hadoop 11358 May 31 06:26 LICENSE
-rw-rw-r-- 1 hadoop hadoop  2359 May 31 06:26 NOTICE
-rw-rw-r-- 1 hadoop hadoop  2406 May 31 06:26 README.md
-rw-rw-r-- 1 hadoop hadoop    31 May 31 06:26 requirements.txt
-rw-rw-r-- 1 hadoop hadoop  1170 May 31 06:26 settings.gradle
drwxrwxr-x 6 hadoop hadoop   124 May 31 06:26 test
drwxrwxr-x 2 hadoop hadoop    78 May 31 06:26 tools

(2) 优化编译速度

查看azkaban-3.76.0/gradle/wrapper/gradle-wrapper.properties文件：

distributionUrl=https\://services.gradle.org/distributions/gradle-4.6-all.zip

可以看到依赖的gradle为4.6版本，从https://gradle.org/releases/下载对应版本的gradle的zip包，放到azkaban-3.76.0/gradle/wrapper目录下。

然后修改gradle-wrapper.properties文件的最后一行，改为：

#distributionUrl=https\://services.gradle.org/distributions/gradle-4.6-all.zip
distributionUrl=gradle-4.6-all.zip

(3) 编译源码

在解压目录下有一个可执行文件gradlew，进行以下操作：

step 1

这一步是执行时间最长的，耐心等待

[hadoop@beh07 azkaban-3.76.0]$ ./gradlew build
Downloading file:/opt/beh/core/azkaban-3.76.0/gradle/wrapper/gradle-4.6-all.zip
......

# 当看到BUILD SUCCESSFUL的时候，你会感觉自己真是鸿运当头
BUILD SUCCESSFUL in 5m 6s
105 actionable tasks: 96 executed, 9 from cache

可能遇到的报错1：

Could not determine the dependencies of task ':az-flow-trigger-dependency-type:kafka-event-trigger:fatJar'.
> Could not resolve all files for configuration ':az-flow-trigger-dependency-type:kafka-event-trigger:compile'.
   > Could not download avro-tools.jar (org.apache.avro:avro-tools:1.8.1)
      > Could not get resource 'https://repo.maven.apache.org/maven2/org/apache/avro/avro-tools/1.8.1/avro-tools-1.8.1.jar'.
         > Read timed out
   > Could not download netty.jar (io.netty:netty:3.10.5.Final)
      > Could not get resource 'https://repo.maven.apache.org/maven2/io/netty/netty/3.10.5.Final/netty-3.10.5.Final.jar'.
         > Connection reset

解决办法：重新执行./gradlew build，直到所需的jar包下载完成

可能遇到的报错2：

> Could not resolve net.jpountz.lz4:lz4:1.2.0.
  Required by:
      project :az-hadoop-jobtype-plugin > org.apache.spark:spark-core_2.10:1.4.0
   > Skipped due to earlier error

解决办法：重新执行./gradlew build，直到所需的jar包下载完成

step 2

[hadoop@beh07 azkaban-3.76.0]$ ./gradlew clean
Parallel execution with configuration on demand is an incubating feature.

BUILD SUCCESSFUL in 2s
19 actionable tasks: 19 executed

step 3

[hadoop@beh07 azkaban-3.76.0]$ ./gradlew installDist
Parallel execution with configuration on demand is an incubating feature.

> Task :azkaban-web-server:npm_install 
added 39 packages in 0.835s


BUILD SUCCESSFUL in 11s
53 actionable tasks: 38 executed, 14 from cache, 1 up-to-date

step 4

[hadoop@beh07 azkaban-3.76.0]$ ./gradlew test
Parallel execution with configuration on demand is an incubating feature.
......
BUILD SUCCESSFUL in 1m 59s
68 actionable tasks: 24 executed, 13 from cache, 31 up-to-date

如果遇到报错的情况，多执行几次

step 5

[hadoop@beh07 azkaban-3.76.0]$ ./gradlew build -x test
Parallel execution with configuration on demand is an incubating feature.
......
BUILD SUCCESSFUL in 22s
74 actionable tasks: 31 executed, 1 from cache, 42 up-to-date

执行完以上5步之后，我们就编译好了源码，需要的安装包也已经打包好了，去以下目录中找：

azkaban-3.76.0/azkaban-db/build/distributions
azkaban-3.76.0/azkaban-exec-server/build/distributions
azkaban-3.76.0/azkaban-hadoop-security-plugin/build/distributions
azkaban-3.76.0/azkaban-solo-server/build/distributions
azkaban-3.76.0/azkaban-web-server/build/distributions

找到的.tar.gz文件分别是：

azkaban-db-0.1.0-SNAPSHOT.tar.gz
azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz
azkaban-hadoop-security-plugin-0.1.0-SNAPSHOT.tar.gz
azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz
azkaban-web-server-0.1.0-SNAPSHOT.tar.gz

这样，一份新鲜的azkaban安装包就已经出炉了！

(6) 编译 execute-as-user.c 文件

这个文件后面会使用到，这里也编译一下，提前准备好。

# 找到这个文件的位置
[hadoop@beh07 azkaban-3.76.0]$ find . -name execute-as-user.c 
./az-exec-util/src/main/c/execute-as-user.c

# 把文件复制到源码包的根目录下，好找一点
[hadoop@beh07 azkaban-3.76.0]$ cp ./az-exec-util/src/main/c/execute-as-user.c ./

# 编译这个文件，并改名为execute-as-user
[hadoop@beh07 azkaban-3.76.0]$ gcc execute-as-user.c -o execute-as-user

# 检查一下，看看是否有execute-as-user文件产生
[hadoop@beh07 azkaban-3.76.0]$ ll | grep execute-as-user
-rwxrwxr-x 1 hadoop hadoop 13616 Aug 23 10:57 execute-as-user
-rw-rw-r-- 1 hadoop hadoop  3976 Aug 23 10:53 execute-as-user.c

然后把execute-as-user这个文件保存好，我们以后会用的到。