Android App Testing 101

Testing is an important part of a software development lifecycle. However in this blog post, I will not be going through why you should be writing tests. Chances are, if you're reading this, you are probably interested in writing test already. Here, I will give you a broader view of testing frameworks, tools in Android so that you can start getting your feet wet today.

Scope of testing

Generally speaking, there are three scopes of testing;

Unit tests - Testing only a unit within your app; for example, testing the expected value from a function given an input, sometimes with the help of mocks or fakes.

Integration tests - Test the integration between multiple units; for example, testing whether clicking on a button actually makes an intended API call and update the state correctly based on the response.

End to end tests - Test a large part of your app, maybe a user flow that spans across multiple screens.

You might have seen the following diagram before and this is true for Android as well. The higher in the pyramid it gets the more resource consuming to write and execute. Therefore, Unit tests are the fastest and easiest to write which means you want more of these in your code. Reversibly, end to end tests are hardest to write and took longest to execute. So, you'd want less of these.

Unit Tests

Since Android uses Java as the main language, it's not surprising that it uses JUnit for its unit testing framework. Officially, JUnit 4 is the supported version. But however, there are third party plugins that allows you to write and execute tests written in JUnit 5 as well. Personally, I'd recommend using JUnit 5 just because writing parameterized tests and the ability to group and organize test is something that I appreciate. Do note that there are some limitations to it and not all features might work.

JUnit works for pure java files only. Accordingly, for codes that interact with Android API such as Activity, Fragment etc, you will have to write a higher level test that confirms your integration with these framework APIs. As stated above, integration tests are less efficient than unit tests and hence you'd want them less. To do so, separate your business logic out of Android components. This would allow you to have more test coverage just by writing unit tests alone.

Integration Tests

Integration tests covers the integration between two or more units. In Android domain, this is most often the integration with network, database or the system frameworks. The part I want to focus here is the last one. These integration with system frameworks includes accessing resources (strings, colors, assets etc), rendering UI on the system, executing services and workers and so on. Traditionally this used to be done on a real device but Android development has evolved to the point where we can now run this in JVM with the help of AndroidJUnit runner, which is part of boarder AndroidX testing libraries. This is also known as instrumentation test in Android.  An example of these tests would be to check whether you get the correct string for correct locale. You can also use this framework to write UI tests but I prefer UI tests in a device or an emulator because I can debug easily if I can see the interaction physically.

End to end test

E2E, short for end-to-end test, runs on a device or an emulator by emulating what the user would click and these tests covers an entire user flow from beginning to end. These tests need to be as close as possible to an production environment, and often run on a test environment that replicates production. In Android, these tests are written with the help of Espresso testing framework. If you're using Kotlin, you can make Espresso API easier with Kakao library. These test will be in the category of white box testing where you know the internal of your app.

Since we're talking about white box testing, it is also worth to mention writing end to end test in a black box manner. If you want these black box tests with first party support, look into Android's UIAutomator API. There are also popular third party testing framework like Appium and Selenium that a lot of companies rely on.

These tests run on physical device or an emulator and this raises the question of which device to use. Trying to test on the lowest supported API level, screen size and latest one is enough for most cases. If you can afford, you can look into Firebase Test Lab or AWS Device farm to run on different variants. These services allow you to commission devices and scale up your testing easily.

Mocking

Mocking means replacing an actual implementation with something you expect it to do. Unless you're writing for a full end-to-end test, you might have to mock your components. It could be as simple as just extending your abstraction interface and returning something from it.

interface UserRepo {
    fun getUsername(): String?
}

fun testUserName() {
    val mockUserRepo = object : UserRepo {
        override fun getUsername(): String? {
            return "Darth Vader"
        }
    }

    assertEquals("expected", doSomething(mockUserRepo))
}

But of course doing this for every test will be painful, so we rely on libraries for this. Mockito and MockK is good contender for mocking with my personal preference more on MockK.

There are also debates among mock vs fakes. Essentially the difference is fake has an actual working implementation that takes shortcut to get it working. For example, you can use a fake database that lives in memory for test instead of one that persist as a file. My personal recommendation is if you are writing a higher level test such as an integration or end-to-end test, it might be better to use a fake but I feel like fakes in unit tests is just not worth it.

Manual Testing

In addition to having automated tests, you should also regularly run manual testing. This allow you to uncover areas that cannot be automated like usability and  accessibility. Sometimes this can be in the form of exploratory testing where you play around with your app to see if there's any area that you can improve in term of experience. You could even do this with an actual user or someone who never seen your app before and see what's their behaviors. You might be thinking in term of developer or product perspective but your user might not be thinking the same with you, so it's always a good idea to run these kinds of tests sometimes.

Pitfalls

For those who are new to testing, it's easy to fall into one of these pitfalls. Do note that these are not just specific to Android but testing in general so you can apply the same principle everywhere.

Testing very specific implementation

Sometimes you want to test every details of a function to tiny detail. For example, if I have a function calculcateTen  that is supposed to return number 10, the implementation could be 5 * 2 , 4+6, 15-5. Here, you should be testing the end result of the function, not making sure it's doing the specific implementation. Trying to test specific details make the test very brittle and make the code harder to refactor. This could be debated that testing very specifically give you the benefits of having a robust test. No matter what, balancing it out and identifying areas to test is a skill you will pick up by honing your instincts as you keep writing tests.

Testing implementation of libraries

You should not be testing the functionalities of third party libraries. The library owner should have tested it themselves already, and you should only be the testing the integration between them. For example, if the library functionality is "If you give me an apple, I will give you an apple juice", you can test your functions to make sure you pass an apple to the library and you can assume it will returns apple juice and not anything else.

Getting hung up on test coverage

Test coverage is an indicator to make sure you have the optimal amount of tests. The problem with getting too hung up on this metric is that people will just hack around the test to get it high. I've seen teams that enforce 100% test coverage and they would literally just execute the code with no assertions just to get to hundred percent. Always have a reminder in your head that having great test coverage is great but it should not be the sole metric that determines how well your tests are.

I didn't go very deep into each topic here because I want this post to be introductory. I hope this give you a starting point but if there's any specific topic, drop me an email or dm me on twitter anytime, and I'd be more than happy to write up a follow up post.