[Improvement] include test evidences

[calado-merkle]

It would be good to introduce the visualization of test evidence, such as screenshots or videos, especially in the case of failed tests.
This feature, along with the error message, would allow for a possible identification of candidates for the root cause of the error without needing to perform a detailed analysis.

This evidence can be generated in the failed keyword or in the teardown of the failed test, whether it is registered or not.

[timdegroot1996]

Hi @calado-merkle thanks for opening an issue!

There are already 2 features in place that might be what you are looking for:

1. There is a "Test Messages" graph that displays the most occuring error messages. You can even change how this graph works if you want to use placeholders, check out the option "-m" or "--messageconfig" that is detailed here: https://github.com/timdegroot1996/robotframework-dashboard?tab=readme-ov-file#usage
2. There is also something in place where you can directly link your log.html files to your dashboard. This will enable you to click on runs/suites/tests to directly open the related log file where the suite/test will also immediately open. This can be enabled with under "-u true" or "--uselogs true" and detailed configuration instructions can be found here: https://github.com/timdegroot1996/robotframework-dashboard?tab=readme-ov-file#usage

If neither of these can satisfy the need of visualising your test evidence can you give me some example of how this should work? A mockup or a little bit more detail of what it would look like in the dashboard would be ideal! Because for me right now it is a little bit vague what you want to be displayed.

Some thoughts that I get from your description:

• A separate tab with screenshots, videos, messages or whatever that is available, linked to the tests?
• Some way to get the details of a failed test in the dashboard (although this already feels a lot like the functionality I describe above with 2. where you just directly open the log file).

Kind regards,
Tim

[timdegroot1996]

@calado-merkle Do you have an update on my response above?

[calado-merkle]

Hi, Tim
Yes, I was thinking about having it in a separate tab, where it would show the screenshots, videos, etc. from the tests that fail, and not just the error messages.
You’ll probably need to look for the “capture page screenshot” or another registered keyword that happens immediately before the failure.
The idea is to have an area focused solely on the failed tests, showing the error messages and the evidence captured by the registered keywords.

[timdegroot1996]

Can you elaborate on how this would be different and an improvement when looking at the linking of logs that I described above in point 2?

Because to me it feels like test evidence should be available to you in the log.html file and if I were to build something similar in robotdashboard while linking is possible and the log.html has all the required details already it seems like I am rebuilding robot functionality which I want to prevent.

[calado-merkle]

What I’m suggesting is different from the native functionality of RFW.

Currently, when a test fails, an error message is displayed, and there may or may not be associated evidence (such as screenshots, videos, etc.). This evidence is located within the keyword (KW) that fails, inside the test case hierarchy, which can be expanded or collapsed.

What I propose is to have, for example, a table where:

• One column lists the failed test cases,
• Another column shows the corresponding error messages.
  When clicking on an error message, a modal window could open displaying the related evidence.
• It could even include the test case tags, since bugs are usually linked to those tags.

This section would therefore aggregate only the failures and their associated evidence, making it much easier to analyze them.

[timdegroot1996]

I think your idea of a more in depth analysis of failed testcases sounds good. I will try to implement this in some way in the Test section.

I still think that the evidence should be located in the log.html files and I don't think it is a good idea to "double" that on the dashboard side as well. However what is possible is that clicking on the error messages opens the respective test in the log.html so you can instantly go to the failed test. (Similar to how it already works with clicking the graphs/tests/suites if you use the --uselogs True flag).

The only question I have is what I should do with multiple runs in the table you are describing. Because in your description you only mention 1 column per test for message/tags etc. How should this work with multiple runs as the dashboard usually contains history of multiple runs.

✅ Test Run Summary

Test Name	Run #1	Run #2	Run #3	Tags
LoginFlowTest	🟢 Pass	🔴 Fail – timeout	🟢 Pass	auth, ui, smoke
PaymentGatewayTest	🟢 Pass	🟢 Pass	⚠️ Error – API 500	payments, api, regression
UserProfileUpdateTest	🔴 Fail – validation	🟢 Pass	🟢 Pass	profile, db, sanity

---

Legend:
🟢 Pass – test succeeded
🔴 Fail – test failed
⚠️ Error – infrastructure or unexpected error

Above is a suggestion of how I think it could be implemented. Is this in the direction you are looking for?

[calado-merkle]

Actually, I was thinking of having only a single run, which means just one set of files (output.xml, log.html, and report.html).
Having multiple runs would imply having n sets of these files.

In your approach, where clicking redirects to the corresponding HTML report for each test, you would only need to map the file sets by run. However, that’s not exactly what I had in mind.

My intention was to have a single run and display all failed tests, their messages, and evidence directly in your report, without needing to open multiple files.
The goal is to centralize everything within your dashboard, and only open the log.html file for deeper analysis when necessary.

The idea of keeping a history of test execution trends is great, but I believe that falls more under the responsibility of CI tools, which are meant to store historical data.
Your tool would serve best as a dashboard focused on a single execution.

I’m not sure what your current plan is, what’s your view on this?

[timdegroot1996]

@calado-merkle, sorry for the late response. The last couple of months have been quite busy.

I've thought about your goal a bit more but I still feel that this is not the direction that I want to take with robotdashboard.
For me the goal of the tool is not "just" looking at 1 run but looking at multiple runs and finding trends across those runs.

Storing the actual details of the test runs into robotdashboard feels like a duplication of the log.html file which I want to prevent. As my goal is aggregating multiple run results and finding trends that way.

If you still feel like there is a way to improve the dashboard by including some form of test evidence that could help find trends across multiple runs I'm open to that suggestion, so please let me know if you have suggestions for that.

If that is not the case I would propose closing this issue.
Let me know what you think, and thanks for taking the time to discuss this topic!