feat(tdp-builder): new tdp-builder based on apache hadoop 3.3.6 builder image

[SteBaum]

The previeuous tdp-builder uses maven 3.6.0 which is too old for ranger 2.6.0 which requires at least maven 3.6.3. And even with a maven update a newly built tdp-builder could not build ranger-2.6.0 as it threw an error.

This new version of tdp builder is based on the official Apache image for building of Hadoop 3.4.1. All TDP-stack 2.0 components can be compiled with this image. However the TDP-stack 1.1 can not be compiled since maven throws an error for Hadoop 3.1.1.

[jsanc525]

I have been able to build all of the following successfully:

zookeeper: 3.8.4-basic
hadoop: 3.3.6-1.0
tez: 0.10.4-basic
spark: 3.5.3-0.0
hive: 4.0.0-0.0
hbase: 2.5.10-basic
knox: 2.0.0-0.0
ranger: 2.6.0-1.0
phoenix: 5.2.1-0.0
phoenix-queryserver: 6.0.0-2.0

Please let me know if there are others that I should check .
Tomorrow I'll work on deploying a cluster with the built components just to make sure everything works as expected, but otherwise everything seems to be working well .
thanks for fixing this @SteBaum !

[SteBaum]

I have been able to build all of the following successfully:

zookeeper: 3.8.4-basic hadoop: 3.3.6-1.0 tez: 0.10.4-basic spark: 3.5.3-0.0 hive: 4.0.0-0.0 hbase: 2.5.10-basic knox: 2.0.0-0.0 ranger: 2.6.0-1.0 phoenix: 5.2.1-0.0 phoenix-queryserver: 6.0.0-2.0

Please let me know if there are others that I should check . Tomorrow I'll work on deploying a cluster with the built components just to make sure everything works as expected, but otherwise everything seems to be working well . thanks for fixing this @SteBaum !

Thanks @jsanc525 ! Yes there is also iceberg, it is the only component built with gradle. An iceberg repository has been created in TOSIT with the compilation command in the README.md. It should'nt take too long.

[jsanc525]

@SteBaum I was able to build Iceberg this morning with no issues !

[SteBaum]

I have changed the tdp-builder and have taken as base image the apache hadoop 3.3.6 image.

The reason is there were 9% differences when doing a package diff between the Hadoop tarball build with the old builder and this one.

These changes have been made compared to the added features of the old builder:

• Maven 3.6.0 has been changed with the version 3.9.9 so that Tez 0.10.4 and Ranger 2.6.0 can be built. It does not make a change for the other components.
• libpam0g-dev has been removed as it made the packaging of ranger-2.6.0-1.0-usersync` fail and this dependency is not included in the apache ranger 2.6.0 builder image.
• Python2 packages have been removed since no component in the TDP 2 stack uses it.
• Node 16 has been removed since Jupyter extensions are build with tdp-builder-python now.

Ant has been maintained but could be removed as I do not see any component built with it.

All TDP stack 2 components have been successfully built and deployed in a cluster with this new builder.

[jsanc525]

All of the components built correctly . I was able to get a cluster started with no issues, and all per-requisite tests passed .