Xcode 9 compatibility

[DogeWowzie]

Are there any known compatibility issues with Xcode 9 and Cedar? So far I am experiencing strange side effects like looping unit tests and unfoldable spec scopes. Codebase is Objective C.

[tjarratt]

Hey @DogeWowzie - this is the first I've heard of it. Unfortunately we don't have a lot of teams within Pivotal that are using Cedar much these days - possibly none at all. In short, we don't find out that there are problems and that Xcode and Cedar are no longer best friends until someone files a helpful issue, like yourself :)

Have you investigated at all yet ? Are you sure that a small test case illustrates the same problem as your codebase does ? If you create a bare-bones minimal test suite with a single test case, do you see the same behavior ?

During the next few weeks I'll try to dedicate some of my own time to look into this, but I'm a parent, have several days I'll be out of the office on vacation here in France, running interviews and managing a team in my office, so I can't promise a fix on any given timeline. That said, I love helping first-time contributors to open source projects and I'd love to help you out if you're interested in digging in. Let me know how I can best help you, if you'd like :)

[DogeWowzie]

Hey @tjarratt, thank you for your reply.

Cedar tests in our project just work fine with Xcode 8, we tried to isolate the problem by deactivating "suspicious" spec classes, but the "looping problem" always occurs when the test target contains a certain number of specs, that's why we think it might be a general problem due to the recent XCTest changes of Xcode 9.

Furthermore, we feel like Cedar tends to be behave inconstantly in Xcode 9, some specs classes are only succeeding while being focussed. I appreciate your effort in having a look into this.

Thank you very much in advance :)

[tjarratt]

Regarding "cold folding" - I've opened up a separate issue to discuss that. Apparently apple broke it when they introduced a new editor in Xcode 9.

[tjarratt]

@DogeWowzie have you tried to create a new, standalone project and see if you can reproduce the same behavior ? I've been running Cedar's specs today in Xcode 9.2 and I haven't yet been able to reproduce this yet.

The thing that would be the most beneficial to me is having a standalone project (ideally something I could put on my machine and debug) so that we have a realistic scenario to explore. I'd say the chances are pretty slim of me being able to fix the bug alone, unless I happen to come across some tests in Cedar's test suite that happen to reproduce the issue.

[tjarratt]

@DogeWowzie I've tried a few times to replicate this with Xcode 9.3, comparing my results with Xcode 8.3.3 and haven't seen any cases where specs have been running repeatedly - even for some of our larger specs where we have something like 200 examples inside a single spec file.

Have you found out anything new ? Ideally, in order to resolve this, I would want to be able to dig into a codebase that consistently reproduces this problem that I can run on my machine. If you can upload a stand-alone example of this as a repo on github that would allow me to make a lot more progress.

Barring that I'd love to know some other details that might help identify the problem:

• What exact version of Xcode are you running ?
• What simulator do you use ? (device and OS version)
• How do you run your tests ?
• Do you know if you are using a test bundle, or a "spec suite" (the answer to the question above might identify this)
• Is there anything special about the configuration for your test target ?
• Do your tests link against any interesting frameworks ?
• Can you tell me the threshold for the number of tests when the "Looping problem" occurs ?
• What kinds of things are your tests doing ? Are they testing UIViewController subclasses ? Are they testing simple value objects ? Are they testing the network ? You can test any number of things with Cedar, and it would be curious to know if there's any sort of pattern here...

[DogeWowzie]

Hey @tjarratt, sorry for the late reply, we took a closer look into our "looping" test target. We have three targets with specs, only one loops. This problem appeared since we updated to Xcode 9, we are currently running Xcode 9.31 with iOS 11.3 simulators.

Turns out specs are looping due to our excessive usage of spy_on in that test target. Now our tests don't loop anymore when we perform a "clean build folder" with Xcode in advance. Everything seems fine now, but we were kind of confused by the fact that everything works just fine with Xcode 8.

[tjarratt]

Oh wow, that's really interesting. spy_on and friends are some dark magic, for sure - I've seen a number of interesting bugs related to Cedar spies throughout the years, especially when you throw Swift into the mix.

I wish I had thought of cleaning the build folder. When strange things start to happen, that's usually my initial instinct. My best guess to explain this behavior is that perhaps there were some changes in XCTest between Xcode 8 and 9 and Xcode somehow managed to load multiple versions of the same spec that were compiled separately, but somehow loadable into the same process after you upgraded, if it was using the same build directory. Normally this changes between releases, but who knows in this case.

Are there any more issues to investigate, or do you feel we could close this issue for now ?

[DogeWowzie]

Hey @tjarratt, I feel like closing this issue for now. Please note that we refactored all the specs (removed the spy_ons) and then managed to prevent the specs from looping by performing "clean build folder" afterwards. If we don't do that, looping does still occur, but the spec target itself feels more stable. Thank you very much for your patience :) Cheers