Run in Firebase Test Lab

[an-gerasimenko]

Looks like the TestParameterInjector is unsupported by the Firebase Test Lab, at least all tests, that use the injector don't run in my case.
Isn't it?

[nymanjens]

I had never heard of the existence of this project. If it supports Java JUnit4 or JUnit5 tests with arbitrary dependencies, then I don't see why you shouldn't be able to use TestParameterInjector there.

FYI: There is an open feature request to support RobolectricTestRunner: #5

[Alton09]

I've found that this actually does work @an-gerasimenko . The issue on our team was with the usage of Flank to run our Firebase Test Lab tests. But if you use the gcloud cli directly to run FTL tests, this library works as expected.

[handstandsam]

I've been working a way to fully enumerate the tests so that it can be executed by test method and not test class. That enables sharding at the test level.

What I ended up with that seems to be working (mostly) so far:

• Reading the DEX files from the APKs, finding all classes annotated with RunWith and TestParameterInjector, generating Java code for all @Test methods to represent the structure of these classes for all fields annotated with @TestParameter and tests methods (including their arguments). Additionally I have to go through each enum that is referenced and generate stub java code for that as well since so TestParameterInjector can enumerate. Then, I take this generated project with each test containing a special custom @TestRule to capture the ran method name. I execute that project and end up with a full list of test names.

Implementation Note: I'm using the package-private com.google.testing.junit.testparameterinjector.TestMethodProcessorList.createNewParameterizedProcessors() .calculateTestInfos(testMethod, testClass) method from the library to compute it the exact same way it will do it at runtime.

This ends up giving me all the test methods run, and then I can run each test by specific name.

---

Other approaches considered:

• Using the package-private method in a KAPT or KSP plugin to generate some other representation of the test names.
• Use libraries to programatically create the classes using Java Reflection via ByteBuddy/JavaAssist to create the class in memory and then run the package-private method to compute it. It ended up not working as I expected though.

The final solution (which is working in some cases) isn't great, but it does work for Java & Kotlin and only needs the final APKs. NOTE: This solution will not work for anything other than @TestParameter and @TestParameters annotations because other providers are computed at runtime and cannot be determined by static analysis of the Dex files.

[handstandsam]

I've made progress on this tool I'm developing to solve this problem. It takes in appApk and testApk files, and returns a full list of test names. I do this by doing analysis on the DEX files, generate a shell of the tests classes in Java and compile them in-memory into a separate classloader. At that point I can use the com.google.testing.junit.testparameterinjector.TestMethodProcessorList.createNewParameterizedProcessors() .calculateTestInfos(testMethod, testClass) solution. It works for @TestParameter for String, enums, int and double. There will definitely be limits to the complexity this will handle, but will allow me to support a way of doing this analysis. Additionally this is only for JUnit4 usage to solve this Android issue. Note: The solution I have could do static code analysis and create the same model, but could hit limits for resolving enums.

@nymanjens - Is this crazy, or is there a better deterministic way to compute the test names statically? Related: Is there any future vision of making test names calculated statically for a subset of use cases?

I think this will be good for these more simple cases. If it continues to work, I'll try to get it open sourced so that someone could leverage it.

[nymanjens]

or is there a better deterministic way to compute the test names statically?

JUnit4 calculates the test names during the construction of the test runner, so it's pretty much static already, in the sense that there cannot be any dependencies on the test instance.

So if you want to make a static method that lists the calculated test methods, you can do:

TestParameterInjector testRunner = new TestParameterInjector(testClass);
testRunner.computeTestMethods()

(the snippet above uses the fact that I just made and released this change (5ea6810) 5 days ago, which exposes computeTestMethods() as public, along with methodBlock() and methodInvoker())

I'm not sure if the above is useful because I have never worked with DEX files or Firebase. It does feel similar to #5 , which now managed to implement, only relying on the three public methods mentioned above. We should be open sourcing that implementation in the next weeks hopefully, and I'll update this thread when we do.

At that point I can use the com.google.testing.junit.testparameterinjector.TestMethodProcessorList.createNewParameterizedProcessors() .calculateTestInfos(testMethod, testClass) solution

On this option, TestMethodProcessorList and TestInfo is deliberately package-private, and I occasionally have to rearrange things in order to keep the codebase somewhat healthy. We go to great efforts to avoid breaking changes with the public functionality, but we cannot do the same for package-private classes, as it would basically mean no refactoring is possible anymore. So I would advice against relying on these.

[handstandsam]

Thank you @nymanjens . I used used the public methods you suggested and got it working!

Here are the steps I use to compute the test names from the static APKs:

1. Use org.smali:dexlib2 to read the DEX files from both the app and test APKs
2. Iterate through all defined classes in the DEX files.
3. Find classes with the annotation @RunWith(TestParameterInjector.class)
4. Ensure there are @Test methods in that class.
5. Transform these test classes using TestParameterInjector to a data class IR model.
6. For each field, constructor parameter, or method parameter, determine if it is an ENUM to create a full list of used ENUMs (by resolving that type from the DEX files).
7. Map these ENUM classes into the data class IR model.
8. Taking the IR models, generate Java Code for each of the tests and enums.
9. Compile java code in-process and load it into a new isolated classloader (with the following JARs on the classpath: junit, testparameterinjector and guava).
10. Run the TestParameterInjector library on the tests in the isolated classloader to determine test methods names.
11. Return the list of parameterized test methods.

---

This doesn't support ALL functionality of TestParameterInjector, but it does cover a large surface area and the key features my team is looking for.

If all goes well, I plan on open sourcing the solution for others to use.

[handstandsam]

If you're using Kotlin, Burst 2.0 looks like just what you need to run on Test Lab! https://github.com/cashapp/burst

This is NOT the same solution I came up with above ☝ for TestParameterInjector, but has the same requirements of needing to compute all names via static analysis.

Burst 2.0 uses a Kotlin Compiler Plugin to generate the other test methods. This means there is a deterministic cartesian product at compile-time.

I'll be trying out Burst 2.0, and if it works, there really isn't a need to keep my solution. 🎉