Compatibility with Gradle Managed Devices

[saket]

Dropshots doesn't seem to be able to generate snapshots when tests are run on managed devices. This can be reproduced in the sample app. Any ideas what might be happening?

[saket]

An official sample project from Google has some answers:

https://github.com/android/testing-samples/blob/dc7a21f08125c4293e5b970dd42ae31e1c6ae933/ui/espresso/ScreenshotSample/app/src/androidTest/java/com/example/android/testing/espresso/screenshotsample/ScreenshotTest.kt#L21

[CodyEngel]

Are there any updates on this? It looks like the screenshots location for Gradle Managed Devices will differ from where Dropshots stores screenshots BUT from the linked Google project it does look like screenshots work.

Would it be possible to configure the gradle plugin to check for a different output folder to retrieve the screenshots?

[ajesh-n]

Based on the answer here
Storing the result in Android/media/<Bundle Identifier>/additional_test_output will fix this problem

 private fun writeImage(name: String, filePath: String?, screenshot: Bitmap): String {
    val externalStorageDir = Environment.getExternalStoragePublicDirectory("Android")
    val screenFolder = File(externalStorageDir, "/media/${targetContext.packageName}/additional_test_output/screenshots".appendPath(filePath))
    if (!screenFolder.exists() && !screenFolder.mkdirs()) {
      throw IllegalStateException("Unable to create screenshot storage directory.")
    }

    val file = File(screenFolder, "${name.replace(" ", "_")}.png")
    file.outputStream().use {
      screenshot.compress(Bitmap.CompressFormat.PNG, 100, it)
    }
    return file.absolutePath
  }

[rharter]

I've been doing some investigation here and I think I've got a plan to complete support for Gradle Managed Devices, while also fixing several other issues and cleaning up some of the plugin's artifact collection. I'm documenting my thoughts and plan here for feedback and review.

---

While the location of additional test data is correct (for the most part), access to that is actually provided by a PlatformTestStorage class. Similar to InstrumentationRegistry, there is a PlatformTestStorageRegistry that the Android test runner uses to configure this.

PlatformTestStorage is important, because there are other places that test output can go. For instance, test output can be sent to a service that might stream it over the network (I think services like Firebase Test Lab might use something like this, though I haven't been able to validate that).

For Dropshots, my plan is to use the registered PlatformTestStorage to write both reference and diff images, which will allow much more flexibility in artifact collection.

Ideally we'd be able to also use this service for reference screenshot input, but the input streams are currently not supported for Gradle, therefore we'll stick with the existing addition of reference images to the assets source set.

This change means that the record task no longer needs to connect to the device via ADB, as it can simply copy the reference images from the additional test output directory into the correct screenshot directory.

Speaking of "correct screenshot directory", this change would also allow for support of storage of reference images for multiple devices. You could have different versions and configurations of test devices configured as Gradle Managed Devices and run Dropshots tests across all of them without issue. In order to support this, I'm planning to update the reference image directory structure to include the device name, with "connected" serving as the default name for connected emulators (which don't have names).

In the future, we could extend this to match connected devices based on some other configuration attribute (i.e. serial?, emulator id?) via a plugin extension, but that will be a problem for another day.

The updated reference image directory structure would look something like this:

src/androidTest/screenshots/connected/...
src/androidTest/screenshots/pixel5api31/...

In order for this to work, I need to be able to pass information from a gradle task to the Dropshots runtime. I'd love to add instrumentation arguments that can be read from InstrumentationRegistry.getArguments(), but as far as I can tell the additional instrumentation arguments are only read from the Gradle project's extension properties and are not extensible.

To work around that I plan to use the same PlatformTestStorage, but creating a Gradle task that will write a config file to the additional test output directory which the runtime can read. This should allow me to pass configuration details, like whether Dropshots is recording screenshots (so assertions don't fail the build), and the device name that reference images should be read from, without breaking configuration cache or requiring a rebuild of the application.

To summarize, here are the steps I'm planning to support this feature and fix many other issues.

• Update Dropshots runtime to write reference and diff images using PlatformTestStorage
• Update record task to copy reference images from additional test output
• Update plugin to write test run config to additional test output directory

[saket]

Sounds good, thanks for the progress update!

[JoseAlcerreca]

@rharter would it help if we met with the GMD folks?

[sergio-sastre]

Android-Testify already suppors GMD, so I leave the link to the corresponding PRs in case it helps 😊
https://github.com/ndtp/android-testify/issues?q=is%3Aissue+is%3Aclosed+GMD

@JoseAlcerreca
Also related to GMD. Although it supports sharding, it does not seem to work with screenshot testing. I created an issue related to that with reproducible example:
android/testing-samples#502

GMD noticeably speeds up test execution, fixing this test sharding issue would be a great extra boost 😉

[rharter]

Thanks, everyone. I've got a work in progress on my work machine that I'm getting finished up to push for review. I'm hoping to get that moving in the coming week, as things seem to be progressing pretty well. I'll mention here when I have something to share.

[rharter]

I've just created #82 in draft form. This is what I'm thinking of, though I'd love some feedback about how this is looking. I think it provides the basis for Gradle Managed Device support, along with things like Firebase Test Lab support, but there are still a couple of things to clear up.

@JoseAlcerreca, if your offer of connection with the GMD folks is still on the table, I have some open questions about things like additional test input, contributing to instrumentation arguments, and the preferred way to access managed test device configuration from a plugin.

As for the draft PR, I'll work to test more thoroughly with this, but I'd also welcome any thoughts or feedback.

[tomczyn]

Any update on this? Or maybe there's some hacky way to reconfigure it locally until it's properly fixed?

[rharter]

Unfortunately not. The primary problem that I've run into is that the reference screenshots are compiled into the test APK, but with GMD that APK can be distributed to multiple devices, and there's no way to pass runtime test configuration information (vai instrumentation arguments or additional test input, which isn't currently supported) for each device, so the runtime library doesn't have a way to know which device screenshots it needs to compare against.

There might be a path to single configuration support for GMD, but IMHO the primary benefit is being able to easily test on multiple configurations.