TRG 11 - Testing and Quality (TRG Suggestion)#1396
TRG 11 - Testing and Quality (TRG Suggestion)#1396ds-hzimmer wants to merge 2 commits intoeclipse-tractusx:mainfrom
Conversation
Moved code coverage TRG position to a new section TRG-11. Adapted and reworded suggestions and removed threshold requirement
Fixed anchor linking
ds-hzimmer
changed the title
ds-hzimmer
requested review from
a team,
MonikaJacobsen,
lgblaumeiser,
matbmoser,
stephanbcbauer and
ther3sa
lgblaumeiser
left a comment
lgblaumeiser
left a comment
There was a problem hiding this comment.
I have to admit, I do not get the reason for this TRG. It is missing:
- A reason on why this TRG is needed
- Concrete guidelines on what is expected
- Requires an external visible impact, the other TRGs are all concerned with general expectations an end user can expect so that he can adopt the project easier. I do not see, how this trg adds such a quality, as the adoptors user experience is not influenced by the code coverage of our products, but by the quality experience of the software. Code coverage is an internal view for a developer group, if this helps them to improve the quality experience.
In this form, the TRG imho does not make sense, sorry, to be so direct.
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| | Status | Created | Post-History | |
There was a problem hiding this comment.
Change logs are always latest change on top!
|
|
||
| ## Why | ||
|
|
||
| Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability. |
There was a problem hiding this comment.
That is not a goal, it is a statement that you want that, but what do we want to achieve with this? Actually, I think, that is quite a good question, right? I do not see an overall why that can be easily expressed, to be honest.
There was a problem hiding this comment.
| Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability. | |
| Goal: To increase visibility in the software quality of the released products, we use test coverage as a metric. It is reported during the release. |
That's how I understood the common sense. Will we put it into the release check and / or the release notes on eclipse tractus-x?
|
|
||
| 1.1. **Code Coverage** | ||
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. |
There was a problem hiding this comment.
Please use SHOULD or MUST or MAY for expressing things that have to be done!
And is it should or must? And what does report mean?
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. | ||
| - Code coverage measurement and reporting should be automated and integrated into the applicable CI/CD pipelines as part of the release process checks. | ||
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | ||
| - However no fixed minimum code coverage threshold (percentage of line coverage) for new code or the existing code base is currently enforced that would automatically block builds or releases if it falls below such a value. |
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. | ||
| - Code coverage measurement and reporting should be automated and integrated into the applicable CI/CD pipelines as part of the release process checks. | ||
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). |
There was a problem hiding this comment.
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | |
| - An Eclipse Tractus-X software project **MAY** use a line coverage of 80% as is the recommendation of the Tractus-X Test and Release Management, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). |
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | ||
| - However no fixed minimum code coverage threshold (percentage of line coverage) for new code or the existing code base is currently enforced that would automatically block builds or releases if it falls below such a value. | ||
| - Thus project leads respectively the committers of a project define, document and enforce acceptable coverage levels and any project-specific Quality Gates themselves. | ||
| - Per default, code coverage should be calculated using both unit tests and integration tests combined, unless a project follows a different documented approach. |
There was a problem hiding this comment.
Again this is a SHOULD
There was a problem hiding this comment.
Were still in the minimum requirements. I would opt for a MUST and maybe only on unit tests
| - Is code only used for logging, metrics, or monitoring. | ||
| - Depend on platform-specifics, for example hardware that cannot be simulated in a test. | ||
|
|
||
| Extensive or critical deliberate exclusions from the code coverage measurement should be reviewed, documented and approved by the project's committers. |
There was a problem hiding this comment.
This is redundant, they have to be documented, this implies a review by the committers
|
|
||
| ### 3. Quality Assurance | ||
|
|
||
| 3.1. **Code Review Requirements** |
There was a problem hiding this comment.
This section is basically redundant.
|
|
||
| ### 2. Analysis and Reporting | ||
|
|
||
| 2.1. **Tools for Code Coverage Reporting** |
There was a problem hiding this comment.
This section does not add much value, either you express a requirement or you give a concrete guideline, but this is neither.
There was a problem hiding this comment.
+1
Here I would prefer how we finally want to report it: in the release checks and / or the release notes of eclipse-tractus x.
The current content, I would see at the example with a note, that we currently use SonarQube
| - During review, the project's committers may also accept changes that reduce code coverage below this defined threshold value if they explicitly decide this. | ||
| - As a general guidance, as outlined above new code is recommended to reach at least 80.0% coverage (SonarQube default Quality Gate). This is not mandatory. | ||
|
|
||
| 3.2. **Risk Analysis** |
There was a problem hiding this comment.
That does not add value. In general, the committers of a Tractus-X project are responsible for their quality. They decide on the real existing quality whether to release, not because of a metric.
stephanbcbauer
left a comment
stephanbcbauer
left a comment
There was a problem hiding this comment.
Nothing to add additional to @lgblaumeiser comments, just my stupid question about projects/products
|
|
||
| 1.1. **Code Coverage** | ||
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. |
There was a problem hiding this comment.
Never thought about it, but are we talking about projects within Eclipse Tractus-X? Or products?
There was a problem hiding this comment.
That is a good point, I also stumbled over this, as far as I remember. In general Eclipse Tractus-X is an Open Source project, and contains lots of repos. We added this level of product to identify lead repositories which might have dependent repos.
There was a problem hiding this comment.
I added a stetement for that above. I think product is correct. I noticed that during adding check for release guideline check automation. If product is in the .tractusx, then it should apply.
But you've got a point with lead repositories. This TRGs scope is frontend and backend code coverage. Not charts. e.g. Portal lead repo contains charts only. Thus, the coverage reporting must apply to the dependent repos, too.
Proposal:
All Eclipse Tractus-X products (product in metadata file .tractusx) MUST measure and report the code coverage quality metric for backend and frontend code contained in the dependent repositories of the leadingRepository.
Note: this may be also in the point above stating the overall applicability
tom-rm-meyer-ISST
left a comment
tom-rm-meyer-ISST
left a comment
There was a problem hiding this comment.
Thanks for raising!
Please differentiate the applicable repository type (product incl. all repositories) and the content (frontend & backend). I would not even go for unit test or integration test criteria. It may be sufficient to have a testing strategy documented in the arc (what would be an improvement because it eases the work for other contributors).
|
|
||
| ## Why | ||
|
|
||
| Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability. |
There was a problem hiding this comment.
| Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability. | |
| Goal: To increase visibility in the software quality of the released products, we use test coverage as a metric. It is reported during the release. |
That's how I understood the common sense. Will we put it into the release check and / or the release notes on eclipse tractus-x?
|
|
||
| Goal: To ensure that all released software components measure test coverage of their code, as a common metric to track and improve quality and reliability. | ||
|
|
||
| This guideline applies to all software components and projects that are part of the Eclipse Tractus-X release process. |
There was a problem hiding this comment.
| This guideline applies to all software components and projects that are part of the Eclipse Tractus-X release process. | |
| This guideline applies to all software components in Eclipse Tractus-X (`product` in metadata file `.tractusx`). |
Doesn't make sense to apply to types support, special.
|
|
||
| 1.1. **Code Coverage** | ||
|
|
||
| - All Eclipse Tractus-X software projects should measure and report code coverage as a quality metric. |
There was a problem hiding this comment.
I added a stetement for that above. I think product is correct. I noticed that during adding check for release guideline check automation. If product is in the .tractusx, then it should apply.
But you've got a point with lead repositories. This TRGs scope is frontend and backend code coverage. Not charts. e.g. Portal lead repo contains charts only. Thus, the coverage reporting must apply to the dependent repos, too.
Proposal:
All Eclipse Tractus-X products (product in metadata file .tractusx) MUST measure and report the code coverage quality metric for backend and frontend code contained in the dependent repositories of the leadingRepository.
Note: this may be also in the point above stating the overall applicability
| - Eclipse Tractus-X Test and Release Management recommend using 80.0% line coverage for new code as an orientation value, based on the default Eclipse Tractus-X SonarQube [Quality Gate conditions](https://sonarcloud.io/organizations/eclipse-tractusx/quality_gates/show/AWBzEoq-FTEFvoJcI01C). | ||
| - However no fixed minimum code coverage threshold (percentage of line coverage) for new code or the existing code base is currently enforced that would automatically block builds or releases if it falls below such a value. | ||
| - Thus project leads respectively the committers of a project define, document and enforce acceptable coverage levels and any project-specific Quality Gates themselves. | ||
| - Per default, code coverage should be calculated using both unit tests and integration tests combined, unless a project follows a different documented approach. |
There was a problem hiding this comment.
Were still in the minimum requirements. I would opt for a MUST and maybe only on unit tests
|
|
||
| ### 2. Analysis and Reporting | ||
|
|
||
| 2.1. **Tools for Code Coverage Reporting** |
There was a problem hiding this comment.
+1
Here I would prefer how we finally want to report it: in the release checks and / or the release notes of eclipse-tractus x.
The current content, I would see at the example with a note, that we currently use SonarQube
|
|
||
| #### 4.2. **GitHub Workflow Integration Code Example** {#github-workflow-example} | ||
|
|
||
| An example of a GitHub workflow integration using Java and Apache Maven can be found below. |
There was a problem hiding this comment.
Do we have a good example for a frontend project, too? That would be very helpful.
5 participants
Suggestion for a Tractus-X Release Guideline (TRG) for "Testing and Quality" in a new section TRG-11, with an initial TRG 11.01 - Code Coverage
Revised from previous pull request (closed) due to some past reviewers no longer being active in the project, adaptations to incorporate their objections (reworded definition, removed threshold requirement), and move needed to a new section TRG-11 is now already occupied by the new KITs section.
Description
Tractus-X Release Guidelines (TRGs): A new TRG section for "Testing and Quality" should be created, and this TRG is a first suggestion in this new section.
The goal is to ensure that all released Eclipse Tractus-X software components meet sufficient test coverage to guarantee quality, stability, and reliability. Measurement and reporting should be performed by all Tractus-X projects. A minimum quality threshold is suggested by Test and Release Management based on default Eclipse Tractus-X settings, but not required or automatically enforced.
The coverage should be measured and monitored using established Open Source tools. For reporting of the measured values, Eclipse Tractus-X already provides a SonarQube Cloud installation. https://sonarcloud.io/organizations/eclipse-tractusx/projects
Other tools to measure code coverage and report it in the build pipeline are also possible if decided by a project. Using either alternative tools or other SonarQube installations should also be the method if the code of a Tractus-X repository is branched to a new repository outside of the Eclipse Tractus-X project, and thus a team would not have access to this SonarCloud organization.
Pull request created by issue:
eclipse-tractusx/sig-release#970
Pre-review checks
Please ensure to do as many of the following checks as possible, before asking for committer review:
[x] DEPENDENCIES are up-to-date. Dash license tool. Committers can open IP issues for restricted libs.
[x] Copyright and license header are present on all affected files