Skip to content

Conversation

@MisterRaindrop
Copy link
Collaborator


Change logs

Cloudberry PXF Integration Testing Framework

This is a Docker-based integration testing framework for Cloudberry Database and PXF (Platform Extension Framework)
connector.

Key Features:

  • Unified Architecture: Ubuntu 22.04 base images for consistent environment
  • Multi-container Setup:
    • singlecluster: Hadoop ecosystem (HDFS, YARN)
    • Cloudberry Database + PXF connector
  • Automated CI/CD: GitHub Actions pipeline with build, test, and artifact collection

Major Improvements:

  • Enhanced Workflows: Improved GitHub Actions with better error handling and reporting
  • Test Automation: Comprehensive test suite with detailed result reporting
  • Configuration Management: Resolved PXF profile conflicts and configuration issues

Local Development Workflow:

First stage:
git clone cloudberry-pxf and cloudberry in the same directory. They need to be mounted to Docker.
then cd cloudberry-pxf and run command.

docker compose -f concourse/docker/pxf-cbdb-dev/ubuntu/docker-compose.yml down -v
          
docker compose -f concourse/docker/pxf-cbdb-dev/ubuntu/docker-compose.yml build

docker compose -f concourse/docker/pxf-cbdb-dev/ubuntu/docker-compose.yml up -d

docker exec pxf-cbdb-dev bash -lc \
"cd /home/gpadmin/workspace/cloudberry-pxf/concourse/docker/pxf-cbdb-dev/ubuntu && ./script/entrypoint.sh"

docker exec pxf-cbdb-dev bash -lc \
"cd /home/gpadmin/workspace/cloudberry-pxf/concourse/docker/pxf-cbdb-dev/ubuntu && ./script/run_tests.sh"

Contributor's checklist

Here are some reminders before you submit your pull request:

  • Make sure that your Pull Request has a clear title and commit message. You can take the Git commit template as a reference.
  • Sign the Contributor License Agreement as prompted for your first-time contribution (One-time setup).
  • Learn the code contribution and doc contribution guides for better collaboration.
  • List your communications in the GitHub Issues or Discussions (if has or needed).
  • Feel free to ask for the cloudberrydb team to help review and approve.

MisterRaindrop and others added 6 commits December 18, 2025 19:02
- Configure S3/MinIO integration with PXF servers
- Add Parquet profile support in pxf-profiles.xml
- Update cloud access test cases to use MinIO protocol
- Configure Gradle and Go to use aliyun mirrors for faster builds
- Improve Docker entrypoint with MinIO deployment
- Update test utilities to support protocol switching (s3/minio/hdfs)
- Fix OutOfMemory test server configuration
- Enhance CI workflow scripts and build process
- Add detailed test reporting with statistics
- Fix permission and disk space issues
- Optimize Java memory and add debug logging
- Update entrypoint for database initialization
- Add Kerberos docker entrypoint to bootstrap secure singlecluster + PXF,
- Improve run_tests.sh env/test runner (MinIO/S3A setup, gpupgrade helpers, hooks)
- Harden automation/regress comparisons and expected outputs to reduce noise/flakes
- Update pxf-pre/post-gpupgrade scripts to work with Cloudberry env
@MisterRaindrop
Copy link
Collaborator Author

@Mulily0513 Hi, I've merged your code. Please review it when you have time.

MisterRaindrop and others added 2 commits December 19, 2025 10:50
Co-authored-by: Dianjin Wang <wangdianjin@gmail.com>
Co-authored-by: Dianjin Wang <wangdianjin@gmail.com>
@tuhaihe tuhaihe force-pushed the liuxiaoyu/merge_ci_test_4 branch from 2c0aea6 to 43957e2 Compare December 19, 2025 07:09
@ostinru
Copy link
Collaborator

ostinru commented Dec 22, 2025

Hi @MisterRaindrop! I have played with this PR. Looks really good!
My minor improvements here: ostinru#8 (It seems that it is not possible to create PR on top of other PR, so I don't know how to transfer my commits to this PR)

@tuhaihe
Copy link
Member

tuhaihe commented Dec 22, 2025

Hi @MisterRaindrop! I have played with this PR. Looks really good! My minor improvements here: ostinru#8 (It seems that it is not possible to create PR on top of other PR, so I don't know how to transfer my commits to this PR)

Hey @ostinru could you list the commit you want to cherry-pick to this PR? I can help do this.

You mean from 0933af0 to 68e5a53?

@my-ship-it my-ship-it merged commit 7e7f496 into apache:merge-with-upstream Dec 22, 2025
18 of 20 checks passed
@tuhaihe
Copy link
Member

tuhaihe commented Dec 22, 2025

Hi @ostinru you can rebase your fix based on the latest work branch. It's good to create a PR now.

@ostinru
Copy link
Collaborator

ostinru commented Dec 22, 2025

Hi @MisterRaindrop! I have played with this PR. Looks really good! My minor improvements here: ostinru#8 (It seems that it is not possible to create PR on top of other PR, so I don't know how to transfer my commits to this PR)

Hey @ostinru could you list the commit you want to cherry-pick to this PR? I can help do this.

You mean from 0933af0 to 68e5a53?

Thanks!
I have created new PR. Skipped on commit - firstly I should check it better before submiting to upstream (this repo).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants