Learn the differences of Test Doubles

Introduction

This is Day 15 of the Merpay Tech Openness Month 2022.

Hi, I’m @smoriwaki, an Android software engineer in Merpay Android Team.
We use a lot of Test Doubles such as Stubs, Mocks and Spies in our usual development.
We are currently working on replacing the architecture of Mercari App with Jetpack Compose, and it is recommended to use Fakes in the project as a trial approach to reduce stub implementations for each test.
To learn the difference between each type of test double, I wrote an article and presented it at an internal lighting talk. This is an edited version of it.

What is Test Doubles

Test Doubles are an object that you replace a real object for testing purposes, like a stunt double for an actor in a movie.
There are 4 types + 1 optional type of Test Doubles, Stubs, Mocks, Spies, Fakes and Dummies.
They were originally defined by Gerard Meszaros in xUnit Test Patterns. (Test Double – xUnit Patterns.com)
There are several definitions depending on the source, but in this article, I’d like to use the definition by Martin Fowler’s article.
It is based on xUnit Test Patterns, and simpler and easier to understand I think.

Name	Description
Stub	Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.
Mock	Mocks are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don’t expect and are checked during verification to ensure they got all the calls they were expecting.
Spy	Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
Fake	Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an InMemoryTestDatabase is a good example).
Dummy	Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.

How to use Test Doubles

Let’s check each of them, comparing them with own test class and a mock framework for Kotlin, MockK.
For example, I’d like to test the GetItemListUseCase class.
Let’s try to replace ItemRepository with Test Doubles.

class GetItemListUseCase(
    private val repository: ItemRepository,
) {
    operator fun invoke(): Result<List<Item>> {
        return repository.getItemList()
    }
}

interface ItemRepository {
    fun getItemList(): Result<List<Item>>
    fun getItem(itemId: String): Result<Item>
}

open class ItemRepositoryImpl(
    private val service: ItemService,
) : ItemRepository {
    override fun getItemList(): Result<List<Item>> {
        return service.getItemList()
    }

    override fun getItem(itemId: String): Result<Item> {
        if (itemId.isBlank()) {
            return Result.failure(IllegalArgumentException())
        }
        return service.getItem(itemId)
    }
}

Stub

Stubs return a canned value.
Therefore, other Test Doubles also have a part of a Stub feature.

Using own test class

class StubItemRepository(
    private val itemList: List<Item> = emptyList(),
    private val item: Item = Item(),
) : ItemRepository {
    override fun getItemList(): Result<List<Item>> {
        return Result.success(itemList)
    }

    override fun getItem(itemId: String): Result<Item> {
        return Result.success(item)
    }
}

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `invoke return itemList`() {
        val itemList = listOf(
            Item("item1"),
            Item("item2")
        )
        val repository = StubItemRepository(itemList)
        useCase = GetItemListUseCase(repository)

        val result = useCase.invoke()
        Truth.assertThat(result.isSuccess).isTrue()
        Truth.assertThat(result.getOrNull()).isEqualTo(itemList)
    }
}

Using MockK

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `invoke return itemList`() {
        val itemList = listOf(
            Item("item1"),
            Item("item2")
        )
        val repository = mockk<ItemRepository> {
            every { getItemList() } returns Result.success(itemList)
        }
        useCase = GetItemListUseCase(repository)

        val result = useCase.invoke()
        Truth.assertThat(result.isSuccess).isTrue()
        Truth.assertThat(result.getOrNull()).isEqualTo(itemList)
    }
}

※This code might not be useful as a real-world test since it only verifies the value of a Stub.

Mock

Mocks are similar to Stubs, but Mocks can check if the specified method is called or not.
Also, Mocks can throw an exception if an unexpected function is called.

Using own test class

class MockItemRepository(
    private val itemList: List<Item> = emptyList(),
) : ItemRepository {
    var isGetItemListCalled: Boolean = false

    override fun getItemList(): Result<List<Item>> {
        isGetItemListCalled = true
        return Result.success(itemList)
    }

    override fun getItem(itemId: String): Result<Item> {
        error("This function is not mocked")
    }
}

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `invoke return itemList`() {
        val itemList = listOf(
            Item("item1"),
            Item("item2")
        )
        val repository = MockItemRepository(itemList)
        useCase = GetItemListUseCase(repository)

        val result = useCase.invoke()

        Truth.assertThat(repository.isGetItemListCalled).isTrue()
    }
}

Using MockK

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `invoke return itemList`() {
        val itemList = listOf(
            Item("item1"),
            Item("item2")
        )
        val repository = mockk<ItemRepository> {
            every { getItemList() } returns Result.success(itemList)
        }
        useCase = GetItemListUseCase(repository)

        val result = useCase.invoke()

        verify(exactly = 1) { repository.getItemList() }
        confirmVerified(repository)
    }
}

Relaxed mock of MockK is very useful, but it creates stubs even for parts that are not relevant to the test.
Therefore, it is better to use confirmVerified if you want to test more strictly.

Spy

Spies are very similar to Mocks, but Spies don’t throw an exception if an unexpected function is called.
Therefore, Spies are often used as wrappers for real objects.

Using own test class

class SpyItemRepository(
    service: ItemService,
    private val itemList: List<Item> = emptyList(),
) : ItemRepositoryImpl(service) {
    var isGetItemListCalled: Boolean = false

    override fun getItemList(): Result<List<Item>> {
        isGetItemListCalled = true
        // You can return super.getItemList() as it is if you don’t need the Stub feature.
        return Result.success(itemList)
    }
}

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `invoke return itemList`() {
        val itemList = listOf(
            Item("item1"),
            Item("item2")
        )
        val repository = SpyItemRepository(DummyItemService(), itemList)
        useCase = GetItemListUseCase(repository)

        val result = useCase.invoke()

        Truth.assertThat(repository.isGetItemListCalled).isTrue()
    }
}

Using MockK

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `invoke return itemList`() {
        val itemList = listOf(
            Item("item1"),
            Item("item2")
        )
        val repository = spyk(ItemRepositoryImpl(DummyItemService())) {
            every { getItemList() } returns Result.success(itemList)
        }
        useCase = GetItemListUseCase(repository)

        val result = useCase.invoke()

        verify(exactly = 1) { repository.getItemList() }
        confirmVerified(repository)
    }
}

Fake

Fakes are similar to Stubs, but Fakes have working implementations that are the same as real objects.

In this case, a Fake implementation for getItemList() is the same as the Stub.
But getItem(itemId: String) of the real object has a validation for itemId, so the Fake should have that as well.
Also, getItem(itemId: String) should return a specified item by itemId, so you need to implement that as well, maybe the server side has a similar logic.

class FakeItemRepository : ItemRepository {
    private val item1 = Item("item1")
    private val item2 = Item("item2")
    private val itemList: List<Item> = listOf(
        item1,
        item2
    )

    override fun getItemList(): Result<List<Item>> {
        return Result.success(itemList)
    }

    override fun getItem(itemId: String): Result<Item> {
        if (itemId.isBlank()) {
            return Result.failure(IllegalArgumentException())
        }

        // service or server side has a similar logic
        val item = itemList.firstOrNull { it.id == itemId }
        return if (item == null) {
            Result.failure(IllegalArgumentException())
        } else {
            Result.success(item)
        }
    }
}

But this might not be perfect because ItemService might have some logic, or the server side might have more implementations such as throwing a network error.

Of course, you can omit several implementations as necessary, but if you omit them more and more, it becomes just Stub.
On the other hand, if Fakes have a lot of working implementations, it might be necessary to test them as well.
As you can see, it is very difficult to make Fakes and maintain them, especially when they involve API requests.

For these reasons, mock frameworks should not be used as Fakes because it will make Fake creations more difficult.
As a result, Fakes are sometimes preferred as “they are more lightweight than mock frameworks”, but I think it is missing the point as a reason.

Dummy

Dummies are used just to avoid compile errors. This is not important.

Using own test class

class DummyItemRepository : ItemRepository {
    override fun getItemList(): Result<List<Item>> {
        TODO("Not yet implemented")
    }

    override fun getItem(itemId: String): Result<Item> {
        TODO("Not yet implemented")
    }
}

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `test something`() {
        val repository = DummyRepository()
        useCase = GetItemListUseCase(repository)
        ...
    }
}

Using MockK

class GetItemListUseCaseTest {
    lateinit var useCase: GetItemListUseCase

    @Test
    fun `test something`() {
        val repository = mockk<ItemRepository>()
        useCase = GetItemListUseCase(repository)
        ...
    }
}

Which is the best Test Doubles to use

In my opinion, it depends because each Test Double has pros and cons.

Stubs are useful for all tests, but sometimes they are not sufficient for testing.
Mocks and Spies are especially useful for Unit Tests, because we often want to verify the method calling.
Fakes are useful for Integration Tests and UI Tests using Robolectric or Espresso, because we just want to check the behavior of the application.

Therefore, I don’t think you need to limit yourself to using only one type.
You might need some rules or policies, but Test Doubles are just a tool, and not the purpose.