memory leak test fails when coverage is used

[naylor-b]

'''
File "/home/swryan/dev/OpenMDAO/openmdao/core/tests/test_setup_memleak.py", line 62, in test_setup_memleak
self.assertLess(mem_diff, MAX_MEM_DIFF_KB,
AssertionError: 611.1328125 not less than 200 : Memory leak in setup(): 611.1 KiB difference between 10 and 100 iter runs
'''

While looking into how to stop() coverage for the duration of the test, Steve found this code where it appears the same collector is on the stack twice so it gets popped then immediately resumes. Could be a place to start looking...

Also, he said it happens with n=1 and not n=2.

    def stop(self) -> None:
        """Stop collecting trace information."""
        assert self._collectors
        if self._collectors[-1] is not self:
            print("self._collectors:")
            for c in self._collectors:
                print(f"  {c!r}\n{c.origin}")
        assert self._collectors[-1] is self, (
            f"Expected current collector to be {self!r}, but it's {self._collectors[-1]!r}"
        )

        self.pause()

        # Remove this Collector from the stack, and resume the one underneath
        # (if any).
        self._collectors.pop()
        if self._collectors:
            self._collectors[-1].resume()

[swryan]

For context, the idea was to implement a decorator to disable coverage for the test:

def nocoverage():
    """
    Disable coverage for a test case.
    """
    def decorator(testcase):
        testcase.__no_coverage__ = True
        return testcase

    return decorator


@nocoverage()
class TestSetupMemLeak(unittest.TestCase):
    """ Test for memory leaks when calling setup() multiple times """

with the corresponding change in test.py:

@@ -287,7 +287,11 @@ def run(self, queue=None):
                sys.stdout = outstream
                sys.stderr = errstream

                # handle "no coverage" request
                if hasattr(parent, '__no_coverage__') and parent.__no_coverage__:
                    stop_coverage()
                else:
                    start_coverage()

After doing this I found that the stop_coverage() call was not always turning it off. This led me to the aforementioned code, where I could see that the stop call was immediately resuming coverage due to the extra entry in the stack.