Unit test coverage

[dijidiji]

Description

Continuing on from #110. This works for me without any changes from last trimester, so I'm looking for some additional testing so I can add these instructions to the unit testing guide before this gets properly peer reviewed.

1. Switch to this branch: git checkout unit-test-coverage
2. Follow the unit testing guide to make sure you can build and run tests
3. Linux/WSL users install lcov via your package manager e.g., apt install lcov (note that you may need to sudo this command)
   macOS users install gcovr via homebrew brew install gcovr
4. Go to the CMake Tools settings in VS Code, switch to the Workspace tab, and change the following settings:
   a. Coverage Info Files:
   Click Add Item and add ${workSpaceFolder}/projects/cmake/build/lcov.info
   b. Linux/WSL users:
   Pre Run Coverage Target: lcov-init-coverage
   Post Run Coverage Target: lcov-capture-coverage
   macOS users:
   Post Run Coverage Target: gcovr-capture-coverage
5. Rebuild the test project, then run tests with coverage with this button:
   
6. After the tests run VS Code should update with the coverage info like this:
   

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

• Generates coverage reports on WSL (Ubuntu 22.04)
• Generates coverage reports on macOS X (Ventura 13.7.5, Intel)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code in hard-to-understand areas
• I have made corresponding changes to the documentation (see Update unit testing guide with code coverage ThothTech-Documentation-Website#146)
• My changes generate no new warnings
• I have requested a review from ... on the Pull Request

[sam-stajnko]

Hi @dijidiji ,
I've looked over this PR and despite my rather limited knowledge of CMake, the changes to the CMakeLists and CMakePresets files look good and I can't find any faults.

The only thing is after pulling this branch onto my local machine and trying to test the coverage functionality in VS Code, I can't seem to get it working. I have followed all of the steps you provided, built the project using the CMake tools extension and ran the tests in the testing pane using the 'Run tests with Coverage' button but for some reason it only shows whether the tests passed or not (No coverage information or visualization). Is there a VS Code extension required for showing the coverage information?

Another thing you might want to change with your documentation is that you have misspelled the workspace alias when specifying the location of the coverage info file in the CMake Tools settings:

${workSpaceFolder}/projects/cmake/build/lcov.info

${workspaceFolder}/projects/cmake/build/lcov.info

I will attach screenshots of my current configuration. Note that I am running Ubuntu (Not WSL).

CMake Tools

CMake Tools Workspace Settings

Terminal (While running tests)

Results (No Coverage)

lcov.log

The information from the lcov log file leads me to believe that coverage information is being generated but for some reason I can't see it in VS Code like your attached screenshots.

Please respond and clarify whether what I'm experiencing is normal or not.
I would have approved this PR now but I'm just a bit uncertain about the results I'm getting.

REgards,
Sam Stajnko.

[dijidiji]

@sam-stajnko Thanks for catching that typo in the lcov.info path, fixed!

With regards to the coverage info not showing up, can you re-run and confirm if the build process finishes fully? In your screenshot it stopped at [build] Capture coverage data with lcov, which makes me wonder if the CMake Tools extension hung up early. The full output should exit with code 0 if it's successful:

[proc] Executing command: /usr/bin/cmake --build /home/diji/SIT374-378/splashkit-core/projects/cmake/build --parallel 14 --target lcov-capture-coverage --
[build] Capture coverage data with lcov
[build] Built target lcov-capture-coverage
[driver] Build completed: 00:00:15.683
[build] Build finished with exit code 0

[sam-stajnko]

@dijidiji

After the tests have been ran (at around 15 seconds), it stays in a hung state until around 60 seconds, then the terminal clears itself and CMake Tools re-configures itself for some reason. For this reason, I couldn't screenshot the exact moment the build process finished and it returned an exit code. I don't know what's going on to be honest...

I will give it another go in a second.

[sam-stajnko]

@dijidiji
I cleared the bin and build directories and rebuilt the project from scratch.

I then navigated to the testing pane of VS Code and pressed the 'Run tests with Coverage' button.

After this screenshot, it hung until 66.9 seconds had elapsed and then it closed the testing tab automatically:

Going back to the testing tab, there is no sign of coverage information:

These screenshots probably won't help much but I really can't think of what the issue is. Is there an extension I need to install for VS Code?

[dijidiji]

@dijidiji

After the tests have been ran (at around 15 seconds), it stays in a hung state until around 60 seconds, then the terminal clears itself and CMake Tools re-configures itself for some reason. For this reason, I couldn't screenshot the exact moment the build process finished and it returned an exit code. I don't know what's going on to be honest...

I will give it another go in a second.

I think this might be a problem with the extension, what version of CMake Tools do you have installed?

[sam-stajnko]

@dijidiji

I have CMake Tools v1.21.36 which is probably the latest version since I reinstalled it today.

[dijidiji]

Ah yep, got the same issue when updating to 1.21.36, as well as the pre-release version. @sam-stajnko could you check if it works if you downgrade to 1.20.53 (click the arrow next to Uninstall and then Install Specific Version)?

[sam-stajnko]

@dijidiji
Yep that did the trick! It seems to be working now.

[sam-stajnko]

I have reviewed the changes made to CMakeLists and CMakePresets and can't find any faults. After pulling this PR branch onto my local machine and testing, I have found that after downgrading the CMake Tools extension to the previous version, it seems to work as intended with coverage information being generated in the Testing panel of VS Code.

Something you might want to do in the future is make a note somewhere about the issues encountered with the latest version of CMake Tools so that no-one else experiences the same troubles.

But apart from that, I have no issues with approving this PR. Nice work Daniel.

Regards,
SAm Stajnko

[dijidiji]

Something you might want to do in the future is make a note somewhere about the issues encountered with the latest version of CMake Tools so that no-one else experiences the same troubles.

Good call, updated the related documentation with a note regarding the version issue

[rory-cd]

General Information

Type of Change:

• New feature

The ability to measure test coverage is a very useful tool both for Thoth Tech team members and leadership, as it identifies gaps in coverage, and provides a metric to measure progress. Happy to approve with minor suggestions for improvement.

Code Quality

• Repository: The Pull Request is made to the correct repository
• Readability: The code is easy to read and follow, with comments included to help understand the code
• Maintainability: This code could easily be maintained or extended in the future

Functionality

• Correctness: The code meets the requirements of the task
• Impact on Existing Functionality: Impact on existing functionality has been considered and tested

Testing

Tested in Windows/WSL:

• Generated HTML report via command line
• Ran tests using the CMake Tools VSCode Extension v1.20.53, as described by the previous reviewer (and in the updated documentation)

Tests ran successfully, and results seem accurate. There is a slight difference between the results in the report vs the extension. This is likely due to quirks of VSCode vs genhtml (for example rounding differences, inclusions, etc.), but is worth noting.

HTML report:

VSCode result:

Documentation

• Documentation: Both inline and applicable external documentation are updated and clear. CONTRIBUTING.md has been updated to include test coverage instructions, but only for MacOS.

Pull Request Details

• PR Description: The problem being solved is clearly described
• Checklist Completion: All relevant checklist items have been reviewed and completed

Suggested Improvements

• CONTRIBUTING.md currently only mentions test coverage for MacOS. A test coverage section for the other operating systems would be beneficial. However, test coverage is not a key aspect of contributing, so this is not crucial.
• Test results vary slightly depending on usage of console commands or CMake Tools extension. It would be best if this was addressed to ensure consistency, or mentioned as expected behaviour in the documentation.