[Feature Request] Track exception occurrences accros test runs

[zied-klouz-lb]

Hi,

Appreciate if there would be a dedicated metrics for Exceptions.

Motivation: keep track of occurred exceptions, even when already handled by TRY/EXCEPT.

Suggestions:

• Table for all occurred exceptions
• Bar graph for exceptions (similar to messages graph) for all displayed runs
• Same bar graph for latest run

[timdegroot1996]

Thanks for creating an issue. I will look into the possibilities with the output.xml file and update you when I know more!

[timdegroot1996]

@zied-klouz-lb I'm looking into this right now and so far I tried the following scenario:

*** Test Cases ***
Exceptional Test
    TRY
        Some Keyword With An Exception
    EXCEPT    I'm a failure of a keyword
        Log    message=Caught Exception
    END


*** Keywords ***
Some Keyword With An Exception
    Fail    msg=I'm a failure of a keyword

I'm assuming in above usecase you would be interested in the "I'm a failure of a keyword" message as that is the "exception" that is occuring. Am I right in assuming so? And I'm also wondering if generally failing keywords like below should be handled in the solution you are aiming for. Think about stuff like this:

${status} Run Keyword And Return Status Fail msg=I failed

Should "I failed" in this scenario also be an "exception" that should be in the list of data that you are requesting?

Apart from that I found the following when looking at the result visitor api on a keyword level:

print(keyword): Fail    msg=I'm a failure of a keyword
print(dir(keyword)): ['BREAK', 'CONTINUE', 'ELSE', 'ELSE_IF', 'ERROR', 'EXCEPT', 'FAIL', 'FINALLY', 'FOR', 'GROUP', 'IF', 'IF_ELSE_ROOT', 'ITERATION', 'KEYWORD', 'KEYWORD_TYPES', 'MESSAGE', 'NOT_RUN', 'NOT_SET', 'PASS', 'RETURN', 'SETUP', 'SKIP', 'SUITE', 'TASK', 'TEARDOWN', 'TEST', 'TRY', 'TRY_EXCEPT_ROOT', 'VAR', 'WHILE', '__annotations__', '__bool__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_datetime_to_timestr', '_elapsed_time', '_elapsed_time_from_children', '_end_time', '_get_id', '_include_in_repr', '_repr_format', '_setter__body', '_setter__tags', '_setup', '_start_time', '_teardown', '_timestr_to_datetime', 'args', 'assign', 
'body', 'body_class', 'config', 'copy', 'deepcopy', 'doc', 'elapsed_time', 'elapsedtime', 'end_time', 'endtime', 'failed', 'from_dict', 'from_json', 'full_name', 'has_setup', 'has_teardown', 'id', 'kwname', 'libname', 'message', 'messages', 'name', 'not_run', 'owner', 'parent', 'passed', 'repr_args', 'setup', 'skipped', 'source_name', 'sourcename', 'start_time', 'starttime', 'status', 'tags', 'teardown', 'timeout', 'to_dict', 'to_json', 'type', 'visit']       
print(keyword.failed): True
print(keyword.body): [robot.result.Message(message="I'm a failure of a keyword", level='FAIL')]
print(keyword.message): I'm a failure of a keyword

print(keyword): Some Keyword With An Exception
print(dir(keyword)): ** same huge list as above ** 
print(keyword.failed): True
print(keyword.body): [robot.result.Keyword(name='Fail', args=("msg=I'm a failure of a keyword",), assign=())]
print(keyword.message): I'm a failure of a keyword

print(keyword): Log    message=Caught Exception
print(dir(keyword)): ** same huge list as above **
print(keyword.failed): (empty)
print(keyword.body): [robot.result.Message(message='Caught Exception', level='INFO')]
print(keyword.message): (empty)

As it appears there are no statistics for "caught in exception" but I can check that on a more detailed level in #robotframework in slack, if that is necessary.

So my actual questions for you are the following:

1. Do you want to track "all" failing keywords or only the ones that are in try/except clauses?
2. What part of the error would you want to see visualized? Keyword name + message?

Let me know your thoughts!
Tim

[timdegroot1996]

@zied-klouz-lb Do you have an update on my comment above?