Failed test does not store output - or wrong error message

[JoseAlcerreca]

java.lang.AssertionError: "MatchesActivityScreenshot" failed to match reference image. 18 pixels differ (8.6805556E-4 %)
Output written to: /storage/emulated/0/Download/screenshots/com.google.samples.apps.nowinandroid.feature.topic.test/MatchesActivityScreenshot.png

However the screenshots directory is empty, This might be a feature and the only bug is in the error message.

Using emulator API 30 and 28.

[sergio-sastre]

In my case ,the tests are failing.
I cloned this project, and tried to record the screenshot on an emulator running API 31, but got the following message:

Unfortunately I could not get it working on any emulator

[rharter]

It looks like that's still trying to compare. The flag to records recently changed from recordScreenshots to dropshots.record, so please make sure you're using the right one.

[sergio-sastre]

It looks like that's still trying to compare. The flag to records recently changed from recordScreenshots to dropshots.record, so please make sure you're using the right one.

It happens when executing ./gradlew :dropshots:connectedAndroidTest -Pdropshots.record. Looks like it ignores the flag when executing the task?

[rharter]

Ahh, I see what's happening, @sergio-sastre. The Dropshots runtime library in this project doesn't use the dropshots gradle plugin since the plugin adds the dependency, and that would end up running in circles. I'm in the process of getting better documentation about how to update things for that project in #18.

[rharter]

@JoseAlcerreca for the original issue, are you saying that the /storage/emulated/0/Download/screenshots/ directory on your test device is empty? I've seen a few issues with the Files app on the emulator not refreshing, but that error message is printed after the image is written to device, so I wouldn't expect to see that unless the write completed.

[Areyana]

actual issue, screenshots folder still empty

[JayShortway]

I noticed this too. After running a failing test, the output / diff images are:

1. pulled to the host machine into the configured resultsDir, which is <path_to_your_project>/<module_name>/build/outputs/test-results/ by default, and then
2. cleared from the device by the ClearScreenshotsTask.

Step 2 is what makes the error message confusing, because if you go to look in that folder it's empty. So either the error message is incorrect or the fact that ClearScreenshotsTask runs is a bug.

Workaround
If you want to leave the diff images on the device, you can work around Step 2 by excluding the ClearScreenshotsTask like so:

./gradlew :<module_name>:connectedDebugAndroidTest --exclude-task :<module_name>:clearDebugAndroidTestScreenshots

(or equivalent for your build variant.)

[renetik]

In my case I had to run:
./gradlew task

find name of task:
clearQaAndroidTestScreenshots

and run it like this:
./gradlew connectedCheck --exclude-task clearQaAndroidTestScreenshots

Its definitelly a bug.