Proposal: Migrate E2E tests to Ginkgo framework

[IrvingMg]

This issue proposes migrating the E2E tests to the Ginkgo testing framework to make the tests more robust, as discussed in #200 and #207.

This proposal describes a high-level plan to migrate the urunc E2E tests to the Ginkgo/Gomega testing framework in order to improve test structure, reliability, and failure diagnostics. It explains the expected test structure, migration steps, and CI changes needed for a smooth transition. The plan was prepared with the help of AI and is meant to guide the overall direction rather than define every detail.

Feedback and suggestions are very welcome.

Full Integration Plan (click to expand)

Ginkgo Integration Plan for urunc E2E Tests

1. Ginkgo Basics

Ginkgo is a testing framework widely used by Kubernetes and other CNCF projects. Gomega is the assertion library that pairs with Ginkgo.

Here's an example of a basic Ginkgo test structure:

var _ = Describe("Ctr", func() {
    BeforeEach(func() {
        // Shared setup - runs before each test
    })

    AfterEach(func() {
        // Shared cleanup - runs after each test
    })

    Context("Qemu", func() {
        It("runs hello-world unikernel", func() {
            // Test logic
            Expect(err).NotTo(HaveOccurred())
        })

        It("runs redis unikernel", func() {
            // Another test
        })
    })

    Context("Firecracker", func() {
        It("runs nginx unikernel", func() {
            // Test logic
        })
    })
})

• Describe: Top-level container (e.g., "Ctr", "Nerdctl")
• Context: Groups related tests (e.g., by VMM)
• BeforeEach / AfterEach: Shared setup and cleanup
• It: Individual test case
• DescribeTable / Entry: Table-driven tests (shown in next section)

2. DescribeTable for Table-Driven Tests

Our current arrays in test_cases.go (e.g., nerdctlTestCases()) map naturally to DescribeTable:

var _ = Describe("Nerdctl", func() {
    var tool testTool

    AfterEach(func() {
        // Log capture on failure
        if CurrentSpecReport().Failed() {
            logs, _ := tool.logContainer()
            AddReportEntry("container-logs", logs)
        }
        // Cleanup
        tool.stopContainer()
        tool.rmContainer()
    })

    DescribeTable("unikernel containers",
        func(args containerTestArgs) {
            tool = newNerdctlTool(args)

            output, err := tool.runContainer(false)
            Expect(err).NotTo(HaveOccurred())

            if args.TestFunc != nil {
                err = args.TestFunc(tool)
                Expect(err).NotTo(HaveOccurred())
            } else {
                Expect(output).To(ContainSubstring(args.ExpectOut))
            }
        },
        Entry("Hvt-rumprun-hello", containerTestArgs{
            Image:     "harbor.nbfc.io/nubificus/urunc/hello-hvt-rumprun:latest",
            Name:      "Hvt-rumprun-capture-hello",
            Devmapper: true,
            ExpectOut: "Hello world",
        }),
        Entry("Qemu-unikraft-ping", containerTestArgs{
            Image:    "harbor.nbfc.io/nubificus/urunc/redis-qemu-unikraft-initrd:latest",
            Name:     "Qemu-unikraft-ping-redis",
            TestFunc: pingTest,
        }),
        // ... more entries
    )
})

Benefits:

• Each Entry generates a separate test that can pass/fail independently
• AfterEach handles cleanup and log capture for all entries
• Existing containerTestArgs struct works directly as entry parameters

3. Full Suite Structure Example

Here's what a complete Ginkgo suite file would look like, organizing tests by VMM:

// nerdctl_suite_test.go
package urunce2etesting

import (
    "testing"
    . "github.com/onsi/ginkgo/v2"
    . "github.com/onsi/gomega"
)

func TestNerdctl(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Nerdctl Suite")
}

var _ = BeforeSuite(func() {
    // One-time setup - e.g., pull images
})

var _ = AfterSuite(func() {
    // One-time cleanup
})

var _ = Describe("Nerdctl", func() {
    Context("Qemu", func() {
        DescribeTable("unikernels", testUnikernel,
            Entry("unikraft-ping", pingArgs, Label("qemu", "unikraft")),
            Entry("rumprun-hello", helloArgs, Label("qemu", "rumprun")),
        )
    })

    Context("Firecracker", func() {
        DescribeTable("unikernels", testUnikernel,
            Entry("rumprun-hello", fcHelloArgs, Label("firecracker")),
        )
    })

    Context("Hvt", func() {
        It("runs rumprun hello", func() {
            // Individual test using It
            Expect(err).NotTo(HaveOccurred())
        })
    })
})

4. Features for Robust Testing

Eventually - Replace time.Sleep with polling:

// Instead of: time.Sleep(3 * time.Second)
Eventually(func() error {
    return pingTest(tool)
}, 30*time.Second, 1*time.Second).Should(Succeed())

FlakeAttempts - Automatic retry for flaky tests:

It("should respond to ping", FlakeAttempts(3), func() {
    // Test will retry up to 3 times on failure
})

Automatic log capture on failure:

AfterEach(func() {
    if CurrentSpecReport().Failed() {
        logs, err := tool.logContainer()
        if err == nil {
            AddReportEntry("container-logs", logs)
        }
    }
})

Labels - Filter tests by VMM, unikernel type, or tool:

Entry("Qemu-unikraft-ping", args, Label("qemu", "unikraft", "network"))

// Run only qemu tests: ginkgo --label-filter="qemu"
// Run network tests: ginkgo --label-filter="network"

SpecTimeout - Per-test timeouts:

It("should complete within 60s", SpecTimeout(60*time.Second), func() {
    // Test implementation
})

Parallel execution - Speed up test runs:

# Run specs in parallel within a suite
ginkgo -p ./tests/e2e/...

5. Files to Refactor

Current File	Changes
tests/e2e/common.go	Keep as-is - testTool interface and helper functions remain unchanged
tests/e2e/test_functions.go	Keep as-is - test functions (pingTest, seccompTest, etc.) are reusable
tests/e2e/tests_skeleton.go	Remove - runTest() orchestration replaced by BeforeEach/AfterEach
tests/e2e/test_cases.go	Keep as-is - containerTestArgs arrays become DescribeTable entries
tests/e2e/e2e_test.go	Remove - currently contains all test functions, will be split into separate suite files
tests/e2e/setup_test.go	Adapt - TestMain image pulling logic moves to Ginkgo's BeforeSuite/AfterSuite

New files to create:

New File	Purpose
tests/e2e/ctr_suite_test.go	Ginkgo test file for ctr
tests/e2e/nerdctl_suite_test.go	Ginkgo test file for nerdctl
tests/e2e/crictl_suite_test.go	Ginkgo test file for crictl
tests/e2e/docker_suite_test.go	Ginkgo test file for docker

6. Reusing Existing Test Functions

The current test functions in test_functions.go (pingTest, seccompTest, userGroupTest, namespaceTest, blockMountTest, httpStaticNetTest) can be reused directly - they already accept testTool and return error:

// Existing function signature works with Ginkgo
func pingTest(tool testTool) error {
    // ... existing implementation
}

// Usage in Ginkgo
It("should respond to ping", func() {
    err := pingTest(tool)
    Expect(err).NotTo(HaveOccurred())
})

// Or with Eventually for async waiting
It("should respond to ping", func() {
    Eventually(func() error {
        return pingTest(tool)
    }, 30*time.Second).Should(Succeed())
})

7. CI Integration

Current CI setup (vm_test.yml):

• Tests run via matrix strategy: test: ["test_ctr","test_nerdctl","test_crictl","test_docker"]
• Invoked via Makefile: make ${{ matrix.test }}
• Already has failure log capture via journalctl

Changes needed for Ginkgo:

• Install Ginkgo CLI: go install github.com/onsi/ginkgo/v2/ginkgo@latest
• Update Makefile targets to use ginkgo command instead of go test
• Optionally enable JUnit reports (--junit-report) for better CI visibility

8. Migration Plan

Recommended order: ctr → nerdctl → crictl → docker

Phase	Tool	Rationale
1	ctr	Set up Ginkgo suite infrastructure, migrate simplest tests (output matching only, no network, no pod lifecycle)
2	nerdctl	Most comprehensive test coverage, validates all test functions work with Ginkgo
3	crictl	Adds pod lifecycle complexity (createPod/stopPod/rmPod)
4	docker	Similar structure to nerdctl, straightforward after nerdctl is done
5	Cleanup	Remove old infrastructure (e2e_test.go, tests_skeleton.go), update CI

Each phase can be a separate PR for incremental review.

[IrvingMg]

@cmainas Here is an example of a log file from a project using Ginkgo for E2E tests:

https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/8723/pull-kueue-test-e2e-main-1-35/2013986068071714816/build-log.txt

The file contains logs from Docker and other components, but near the end you can see the Ginkgo test logs. You can look for the following line, which indicates where the Ginkgo logs start:

Running Suite: End To End Suite: kindest/node:v1.35.0 - /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster

Ginkgo also allows generating reports in formats such as JSON or XML. Here is an example of a JSON report:

https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/8723/pull-kueue-test-e2e-main-1-35/2013986068071714816/artifacts/run-test-e2e-singlecluster-1.35.0/e2e.json