[8.19] (backport #18733) Respect ARCH env var when downloading JDK via gradle

[mergify]

Release notes

Ship the correct bundled JDK for all logstash artifacts.

What does this PR do?

With the refactor to make gradle the entry point to every CI task #18471 (and to not have any cyclical dependencies) the copyJdk task ended up satisfying the dependency for all tasks for which it was needed. Previously, rake would call back in to gradle and continuously download and delete JDKs. This resulted in a bug where a single JDK was shipped for ALL artifacts.

In practice, CI for building artifacts is split across:

1. x86 linux VM builder linux-x86_64 tar, darwin-x86_64 tar, windows-x86_64 zip, amd64 deb, x86_64 rpm: with x86 JDK (linux ok, windows/mac broken)
2. x86 linux VM builder linux-aarch64 tar, darwin-aarch64 tar, arm64 deb, aarch64 rpm: with x86 JDK (broken)
3. x86 linux VM builder x86 docker images (ok)
4. arm linux VM builder: arm docker images (ok)

The combination of selecting the architecture and operating system were overly complex and made assumptions about the host os/arch.

New approach
Now we just download all the JDKs for a specific arch up front and then package the correct one based on the target artifact (NOT the host). The copyAllJdks task uses getTargetPlatforms() which reads the ARCH environment variable:

ARCH=x86_64 downloads linux-x64, windows-x64, darwin-x64 JDKs
ARCH=aarch64downloads linux-aarch64, darwin-aarch64 JDKs

Each platform gets its own download task (downloadJdk_linux_aarch64) and output folder (jdk-linux-aarch64/). The rake packaging scripts select the correct JDK folder based on the artifact being built.

Why is it important/What is the impact to the user?

Arm/aarch64/windows/mac users can now run logstash installed from .rpm, .tar, .deb artifacts.

Validation steps:

I checked out a VM we use as a builder and build artifacts for all platforms for both architecture sets:

The following script is used to check JDKs

bash-4.4$ cat jdk-validate.sh
#!/bin/bash
for f in logstash-*.tar.gz; do
  [[ $f == *no-jdk* ]] && continue
  echo "$f:"; tar -xzOf "$f" --wildcards '*/jdk*/bin/java' 2>/dev/null | file -
done
for f in logstash-*.zip; do
  [[ $f == *no-jdk* ]] && continue
  echo "$f:"; unzip -p "$f" '*/jdk/bin/java.exe' 2>/dev/null | file -
done
for f in *.deb; do
  [[ $f == *no-jdk* ]] && continue
  echo "$f:"; ar p "$f" data.tar.gz | tar -xzOf - --wildcards '*/jdk/bin/java' | file -
done
for f in *.rpm; do
  [[ $f == *no-jdk* ]] && continue
  echo "$f:"; rpm2cpio "$f" | cpio -i --to-stdout '*/jdk/bin/java' 2>/dev/null | file -
done

For x86_64 ARCH=x86_64 ./gradlew artifactAll:
The build dir contains

Dockerfile-full		      jdk-validate.sh				    logstash-9.4.0-SNAPSHOT-no-jdk.zip
Dockerfile-observability-sre  libs					    logstash-9.4.0-SNAPSHOT-windows-x86_64.zip
Dockerfile-oss		      logstash-9.4.0-SNAPSHOT-amd64.deb		    logstash-9.4.0-SNAPSHOT-x86_64.rpm
Dockerfile-wolfi	      logstash-9.4.0-SNAPSHOT-darwin-x86_64.tar.gz  plugin_aliases_hashed.yml
fpm			      logstash-9.4.0-SNAPSHOT-linux-x86_64.tar.gz   reports
jdk-21.0.9-darwin-x64.tar.gz  logstash-9.4.0-SNAPSHOT-no-jdk.deb	    tmp
jdk-21.0.9-linux-x64.tar.gz   logstash-9.4.0-SNAPSHOT-no-jdk.rpm
jdk-21.0.9-windows-x64.zip    logstash-9.4.0-SNAPSHOT-no-jdk.tar.gz

The validate script results:

bash-4.4$ ./jdk-validate.sh
logstash-9.4.0-SNAPSHOT-darwin-x86_64.tar.gz:
/dev/stdin: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>
logstash-9.4.0-SNAPSHOT-linux-x86_64.tar.gz:
/dev/stdin: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
logstash-9.4.0-SNAPSHOT-windows-x86_64.zip:
/dev/stdin: PE32+ executable (console) x86-64, for MS Windows
logstash-9.4.0-SNAPSHOT-amd64.deb:
/dev/stdin: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped
logstash-9.4.0-SNAPSHOT-x86_64.rpm:
/dev/stdin: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

For arm ARCH=aarch64 ./gradlew artifactAll:
The build dir contains

Dockerfile-full			  jdk-validate.sh				 logstash-9.4.0-SNAPSHOT-no-jdk.rpm
Dockerfile-observability-sre	  libs						 logstash-9.4.0-SNAPSHOT-no-jdk.tar.gz
Dockerfile-oss			  logstash-9.4.0-SNAPSHOT-aarch64.rpm		 logstash-9.4.0-SNAPSHOT-no-jdk.zip
Dockerfile-wolfi		  logstash-9.4.0-SNAPSHOT-arm64.deb		 plugin_aliases_hashed.yml
fpm				  logstash-9.4.0-SNAPSHOT-darwin-aarch64.tar.gz  tmp
jdk-21.0.9-darwin-aarch64.tar.gz  logstash-9.4.0-SNAPSHOT-linux-aarch64.tar.gz
jdk-21.0.9-linux-aarch64.tar.gz   logstash-9.4.0-SNAPSHOT-no-jdk.deb

The validate results:

bash-4.4$ ./jdk-validate.sh
logstash-9.4.0-SNAPSHOT-darwin-aarch64.tar.gz:
/dev/stdin: Mach-O 64-bit arm64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE>
logstash-9.4.0-SNAPSHOT-linux-aarch64.tar.gz:
/dev/stdin: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, not stripped
logstash-9.4.0-SNAPSHOT-arm64.deb:
/dev/stdin: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, not stripped
logstash-9.4.0-SNAPSHOT-aarch64.rpm:
/dev/stdin: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, not stripped

Similarly I build container artifacts (which we have better test coverage for) and validated those worked for arm (we have a ton of test coverage in CI for x86).

Closes #18728

---

This is an automatic backport of pull request #18733 done by Mergify.

[mergify]

Cherry-pick of 7f1ca8e has failed:

On branch mergify/bp/8.19/pr-18733
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 7f1ca8eea.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   .buildkite/aarch64_pipeline.yml
	modified:   .buildkite/scripts/dra/generatesteps.py
	new file:   .buildkite/scripts/exhaustive-tests/build-on-x86.sh
	new file:   .buildkite/scripts/exhaustive-tests/test-on-arm.sh
	deleted:    buildSrc/src/main/groovy/org/logstash/gradle/tooling/ExtractBundledJdkVersion.groovy
	modified:   buildSrc/src/main/groovy/org/logstash/gradle/tooling/ToolingUtils.groovy
	deleted:    buildSrc/src/test/groovy/org/logstash/gradle/tooling/ExtractBundledJdkVersionTest.groovy
	modified:   rakelib/artifacts.rake

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   build.gradle

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[donoghuc]

aarch64 tests https://buildkite.com/elastic/logstash-aarch64-pipeline/builds/628

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: f5014d5

History

• 💔 Build #4291 failed c8897de

cc @donoghuc

[donoghuc]

New aarch64 tests are green