PHP Fatal Error gives partial suite result, with no easy hint of what's going on

[jplambert]

Hi,

We are using FitNesse with the phpslim runner and integrated it with Jenkins.

Sometimes the PHP fixtures gets a Fatal Error for many possible reasons. (an easy to reproduce case would be a syntax error in the PHP)

That kills the PHP process and stops the Slim runner. This is kind of obvious when running the same suite manually: the FitNesse output is clearly missing something, and we can have a look at the "Output captured" and see the Fatal Error. However, when running by Jenkins, we only see that a handful of tests has run instead of the complete suite. (like Jenkins says there is now 3 tests instead of 6, but other than that there is nothing warning that it went horribly wrong)

For now our FitNesse practices are still young and our base of tests is small, so we detect immediately when that happens. However, that can easilly give the feeling that FitNesse is not reliable and I can imagine that when the test suite will be much bigger that it could go unnoticed during several days.

Do you think it would be possible:

• To catch such Fatal Error to make a single test fail but not crash the whole Suite?
• Alternatively (maybe better but not sure it is possible given the integration between FitNesse and Jenkins) make Jenkins consider the build as Failed in that case instead of considering the provided results of the test suite as if nothing special happened.

What do you think?

Thanks in advance,

JP

[ggramlich]

Hi JP,

it has been about 4 years, since I last seriously used PHP.

Have you tried playing around with the set_error_handler in https://github.com/ggramlich/phpslim/blob/master/PhpSlim/PhpSlim/StatementExecutor.php

You can give it a second argument for the error types, even though this will probably not work for syntax errors.
Also, I just saw, that constructor calls don't use the special error handling. You might want to add it there, too.

This might be a step towards your first suggestion.
Does this help a little?

Gregor

[jplambert]

Thanks Gregor,

We'll have a look at this suggestion. However, looking at the PHP doc, it seems that the default value for this extra parameter will already catch anything, so providing a value should not help.

Thanks anyway for pointing out the interesting part in the code. I'll let you know how our experiments goes.

JP

[etienneCharignon]

Hi JP,

Did you manage to find something ? I'm getting in the same problem.

Etienne

[jplambert]

Hi Etienne,

I don't remember exactly, I believe that we used bigger timeout values. You can have a look at my fork: https://github.com/jplambert/phpslim

JP