Test Case Candidates

[tcooper]

Here is a rough list of potential test case candidates we should be able to knock out pretty easily.

The vast majority of these have been successfully completed, we just haven't documented the process and/or defined how to track that status for each release candidate and/or full release.

Suggest more in this discussion issue by supplying your own lists in comment. I can copy/paste them into this list as they come in.

These would only be checked off as done where a Test Case document has been merged into the repository after completing review.

We'll want/need to create cross reference to a requirements document and implement proper test case naming hierarchy (this is in work). Once those two pieces are defined and we have documentation describing them we can turn this list into a document that will require PRs to update.

Anaconda

boot ISO

[ ] Successful install of minimal from boot ISO (network) with tier0
[ ] Successful install of minimal from boot ISO (network) with auto selected mirror
[ ] Successful install of server from boot ISO (network) with tier0
[ ] Successful install of server from boot ISO (network) with auto selected mirror
[ ] Successful install of graphical server from boot ISO (network) with tier0
[ ] Successful install of graphical server from boot ISO (network) with auto selected mirror
[ ] Successful install of workstation from boot ISO (network) with tier0
[ ] Successful install of workstation from boot ISO (network) with auto selected mirror

minimal ISO

[ ] Successful install of minimal from minimal ISO (no network)

DVD ISO

[ ] Successful install of minimal from DVD ISO (no network)
[ ] Successful install of server from DVD ISO (network) with tier0
[ ] Successful install of graphical server from DVD ISO (network) with tier0
[ ] Successful install of workstation from DVD ISO (network) with tier0

EPEL as extra repository

[ ] Successful install of workstation with XFCE from DVD ISO (network) with tier0 and extra repository (EPEL)

Alternate Language Installs

[ ] Successful install of minimal from DVD ISO (no network) in French
[ ] Successful install of minimal from DVD ISO (no network) in German
[ ] Successful install of minimal from DVD ISO (no network) in Spanish

migrate2rocky.sh

[ ] Successful migration of vagrant base box for each of centos/rhel/ol/alma
[ ] Successful migrate of vagrant base box with EPEL enabled
[ ] Successful blocked migration from Rocky
[ ] Successful blocked migration with extra repositories defined
[ ] Successful migration with EFI boot
[ ] Successful migration with BIOS boot
[ ] Successful migration with MBR
[ ] Successful migration with GPT
[ ] Successful migration with RAID
[ ] Successful migration with LVM on RAID
[ ] Successful migration with MULTI boot grub2 configuration

Applications

[ ] Successful install / test of TBD precompiled application
[ ] Successful build, install and test of TBD source

[grayeul]

What is the significance of the 'tier0' references above? Does this mean using a dl.rockylinux.org URL instead of the mirrorlist ? Or what?

[grayeul]

Additional item for Install test (not sure exactly where this could/should be tested) - is
[ ] Verify ability to install from a pre-configured kickstart file.

And (different category):
[ ] Verify full RedHat debranding (i.e. search for names?)

[tcooper]

@grayeul Yes, tier0 is dl.rockylinux.org.

Interesting that you added pre-configured kickstart... automated installs as part of this testing would be from pre-configured kickstart files or coded for some automated UI verification test (ie. OpenQA style). Kickstart is easier (I do this now) while OpenQA style would take more time to build/validate. We probably will do both.

For debranding yes... the upstream packages are patched every time there is an update. With the knowledge that is in git.rockylinux.org about that patching we can write tests that verify debranding was done for every package update and that no new branded content was added to the packages that would require updates to the patches.

[lumarel]

As of the additions of todays meeting:

• Successful installs for different languages

And for later with OpenQA available it should also be possible to:

• Short application tests (smoketest)

[alexhaydock]

One thing I'd add to the list if possible is to test successful installs that make use of the Anaconda OpenSCAP plugin, just to make sure the popular hardening policies that make use of the scrap-security-guide content still work. That can all be automated via Kickstart quite easily though.

[tcooper]

@alexhaydock a quick glance (and search) through os-autoinst-distri-fedora suggests they don't have tests for security profiles (does Fedora even has security profiles?) and this will need to be developed de-novo for Rocky. I think it's reasonable to add this but it might have to wait until we've modified a few of the existing tests and gotten a handle on openQA before taking this on.

[alexhaydock]

@tcooper They do actually have OSPP and PCI-DSS profiles, but Fedora is generally too fast moving for any of the big compliance giants like ANSSI, CIS and DISA STIG to apply. I'm a little surprised they don't have tests for it at all, but I suppose they consider it less important to their audience than it would be for a RHEL or RHEL-derivative's audience, which I generally agree with.

I'm looking forward to see where we can get with OpenQA since it's looking very useful! When the time comes, I'm happy enough to have a go at writing the test cases for the Anaconda compliance tooling.