Building a E2E Encrypted Message Exchange on Android

Recently, I did a deep dive into data encryption on Android platform, including both symmetric encryption for data at rest and asymmetric encryption for data exchange between two devices or with a server. One of the thing I noticed regarding this topic is that the information on the web can be very limited and official cryptography documentation on Android Developer provide little to no help. So, hopefully in this post, I can provide an enough understanding of the cryptography features on Android underneath the hood.

So why End to End encryption?

For some background, we were dealing with a very sensitive private information and since the app is mainly going to be used in a place where people are most likely to be looking for a public WiFi, there is a risk of people connecting to fake hotspots and subsequently opening them up to man-in-the-middle attacks. We specifically didn't want to use certificate pinning because they can lock the user out the app if they are not setup and maintained carefully. Instead, we wanted to look into possibilities of encrypting the data exchange between our app and the backend server so that even if they are intercepted, the malicious actor would not be able to do anything with the data alone. The gist of our initial thought was that

• Both the device and the sever will generate their own sets of private and public keys.
• The backend's public key is bundled into the app as opposed to asking the server's public key on app launch. The disadvantage is that rotating the key out is more difficult. On the plus side, the attacker cannot intercept key exchange and replace the response with their own public key.
• The app will then use the server's public key to encrypt the request body that contains the user's sensitive information and then send this encrypted cipher to the server.
• The server will then use its private key to decrypt the request and process it. For sensitive response, it will then uses the user's public key to encrypt the response before sending them back,
• Similarly to above, the app will then uses its private key to decrypt the response.

Do note that even though our use case only involve between the user's device and the server , the same approach can be applied to exchanging data between peers as well.

What library to use?

The easiest way to implement end-to-end encrypted data exchange is to use a popular library like libsodium but we didn't want to be tied to a third party library. So we started looking for a native first-party implementation within the Android OS. The good news is Android OS is built upon Java, so that means it supports Java Security API protocols.

How Java Security package work is that we have to register a Java Security Provider (JSP) which implements key generations and key algorithms according to the Java Cryptography Specification. One of the most popular JSP library is BouncyCastle which Android bundled into the OS until Android 9. From then on, BouncyCastle was replaced with Conscrypt (also known as "AndroidOpenSSL" ), a Google implementation of Java Security Provider. The main difference between Conscrypt and BouncyCastle is that

• Conscrypt's algorithm implementation are better optimized for Android OS so its performance is better compared to BouncyCastle.
• However, Conscrypt has a smaller set of algorithms that are supported compared to BouncyCastle's wide range of supported implementations.

The idea is that if you want more performance and if your use case is not niche enough to require many capabilities, you can just stick with Conscrypt, otherwise, you may want to use BouncyCastle.

Registering a Java Security Provider

If your minimal supported Android version is Android 9, you can skip this step as Conscrypt should already be part of the OS. However, I do recommend adding a specific version of Conscrypt as dependencies and register an instance of the provider as Android doesn't guarantee a particular provider when you are not using Android Keystore provider (which I will get into later when we get to key storage). To register a provider, all you have to do is to add this line in your Application's onCreate function.

Security.addProvider(Conscrypt.newProvider())

You can also register multiple providers in which case, the API will first check whether the first provider support a certain cryptography algorithm that you've asked for, and either uses that provider if it does or will keep on checking the entire list of providers that you've registered until it can find a suitable one.

// Conscrypt is first priority with BC as second
Security.addProvider(Conscrypt.newProvider())
Security.addProvider(BouncyCastleProvider())

// You can also provide priority
Security.insertProviderAt(BouncyCastleProvider(), 1)

// You can also remove existing one
Security.removeProvider("BC")

Generating Key Pairs

Because we can register multiple security providers, when we create an instance of Key Generator, it's best to also provide a specific provider name to make sure we get the exact provider we want. The following code snippet generates a private-public key pairs using Conscrypt.

// Use "BC" for provider name if you want to use BouncyCastle JSP
val keyPairGenerator = KeyPairGenerator.getInstance(
    KeyProperties.KEY_ALGORITHM_RSA,
    "AndroidOpenSSL"
)
// Creating 2048-bit RSA keys
keyPairGenerator.initialize(2048)

val publicKey = keyPair.public
val privateKey = keyPair.private

println("publicKey : ${Base64.encode(publicKey.encoded)}")
println("privateKey : ${Base64.encode(privateKey.encoded)}")

Encrypting using a public key from server

To send an encrypted request body to the server, we need to use the server's public key instead of the one we just generated in the app. Only then, the server can decrypt the body using its private key. For demonstration purpose, the server's public key will be sent to the device in a base64 encoded format. To make it compatible with Java protocols, we would have to convert this base64 string into an instance of PublicKey.

fun encrypt(base64EncodedPublicKeyString: String, requestBodyJson: String) {
    val base64DecodedKeyBytes = Base64.decode(base64EncodedPublicKeyString)

    val keyFactory = KeyFactory.getInstance("RSA")
    // Java support X-509 key format which also includes PKCS
    val keySpec = X509EncodedKeySpec(base64DecodedKeyBytes)
    // Generate a public key from key specification and key data
    val severPubKey = keyFactory.generatePublic(keySpec)

    // You can check key alogrithim names at https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html
    val cipher: Cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding", "AndroidOpenSSL")
    cipher.init(Cipher.ENCRYPT_MODE, severPubKey)
    val encryptedBytes = cipher.doFinal(requestBodyJson.toByteArray())

    val Base64encodedData = Base64.encodeToString(encryptedBytes, Base64.DEFAULT)
    makeApiCall(body = Base64encodedData)
}

Decrypting using private key

Decrypting is similar to encryption, however, this time we don't need to decode the base64 string and can just use the private key we created before.

fun decrypt(encryptedMessage: String) {
    val base64DecodedMessageBytes = kotlin.io.encoding.Base64.decode(encryptedMessage)

    val cipher: Cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding", "AndroidOpenSSL")
    cipher.init(Cipher.DECRYPT_MODE, keyPair.private)
    val decodedString = String(cipher.doFinal(base64DecodedMessageBytes))

    println(decodedString)
}

With this three pieces of snippets, you should be able to do an end-to-end encrypted exchange between the devices and the server!

S0 that's what we thought. It doesn't take long for us to find out that there's a huge problem with this approach. Asymmetric encryption like RSA cannot encrypt data more than their block size! What it means is that if you're encrypting a large file, you first have to split the file into smaller chunks, and encrypt these chunks individually. The same goes for decryption, you have to decrypt them individually and combine the chunks back into the payload. This can be a massive performance overhead depending on the size of your payload!

Dealing with Larger payload

A solution to this problem however, is to combine both symmetric encryption like AES with asymmetric encryption like RSA. This is called hybrid encryption. In hybrid encryption, we first encrypt our data using a one-time use symmetric key. Since symmetric encryption has no limitation on data size, there is no performance overheard even with larger payloads. We then encrypt the symmetric key with the public key from the server. The encrypted data is then sent together with the encrypted symmetric key. The server will first decrypt the symmetric key using its private key, and then decrypt the data with the symmetric key afterwards. Another good thing about this mode of encryption is that the frontend app don't need to generate its own sets of keys. Since the one-time symmetric key is unique to each request and subsequently a response associated with that request, the server can send a response that is encrypted with this symmetric key. The device already hold this key in memory until a response is received and decrypted. After which, we can clear the key out from the device since it's no longer needed. This process also eliminated the needs for the device to generate its own set of public/private key pair.

The code for this looks something like this (shortened for readability purpose)

fun generateSymmetricKey() : SecretKey {
    val keyGenerator = KeyGenerator.getInstance("AES", "AndroidOpenSSL")
    keyGenerator.init(256)
    return keyGenerator.generateKey()
}

fun encryptSymmetricKey(publicKey: PublicKey, symmetricKey : SecretKey) : ByteArray {
    val cipher: Cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding", "AndroidOpenSSL")
    cipher.init(Cipher.ENCRYPT_MODE, publicKey)
    val encryptedKey = cipher.doFinal(symmetricKey.encoded)

    return encryptedKey
}

fun sendEncryptedPayload() {
    val symmetricKey = generateSymmetricKey()
    val encryptedData = encryptData(data, symmetricKey)
    val encryptedKey = encryptSymmetricKey(publicKey, symmetricKey)

    // You also need to send the metadata of symmetric key such as key algorithm used
    sendRequest(encryptedData, encryptedKey)
}

Do note that in addition to encrypted key, you will have to send metadata of the symmetric key such as the key length, key algorithm that was used and so on. If we add all of that, what was a few line of code has quickly turned into a big chunk of work! The good news is that there is already a RFC standard called JSON Web Encryption which is similar to what I've just described. What this also means is there are already numerous libraries out there which adhere to this standard so you don't really need to implement all of this yourself. But if you ever want a custom implementation, you now know how to code one 😉

This last part about key storage only matters if you are storing your own set of public/private key within Android device. If you're using hybrid encryption, you don't need to worry about this as we don't need to generate a key pair on Android.

Key Storage

When it comes to storing keys, Java already provided KeyStore API. However, on Android, the only KeyStore provider that the OS support is its own Android KeyStore Provider. Meaning, even though BouncyCastle's KeyStore may work in other JVM environments, they will not work on Android. With AndroidKeyStore, your keys will be stored in a secured container and the OS makes it very difficult for an attacker to extract them. Our key generation code looks a little bit different if we factor in KeyStore into it.

fun generateKey(keyAlias: String): KeyPair? {
    val keyStore: KeyStore = KeyStore.getInstance(ANDROID_KEYSTORE_PROVIDER).apply {
        load(null)
    }

    // Check if the key is already stored with given alias
    val doesKeyAliasExists = keyStore.aliases().toList().find { it == keyAlias } != null

    if (!doesKeyAliasExists) {
        // Specifying key parameter
        val keyPairGenerator: KeyPairGenerator = KeyPairGenerator.getInstance(
            KeyProperties.KEY_ALGORITHM_RSA,
            ANDROID_KEYSTORE_PROVIDER
        )
        val parameterSpec: KeyGenParameterSpec = KeyGenParameterSpec.Builder(
            keyAlias, // Telling the OS to store our key with this alias
            KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT
        )
            .setBlockModes(KeyProperties.BLOCK_MODE_ECB)
            .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_PKCS1)
            .build()

        keyPairGenerator.initialize(parameterSpec)
        return keyPairGenerator.genKeyPair()
    } else {
        // Key already exists, retrieve from keychain
        val entry = keyStore.getEntry(keyAlias, null) as? KeyStore.PrivateKeyEntry
        return KeyPair(entry?.certificate?.publicKey, entry?.privateKey)
    }

This provides maximum security for your keys, but the catch is that due to Android fragmentation, the KeyStore API can be unstable and in some OEM devices, it will have fatal crashes. The keys can get corrupted, and sometimes they key generation takes too long that it causes your app to just crash from ANR. So much so that Tink (the cryptography library that is used underneath the hood byEncryptedSharedPreferences & EncryptedFile from the javax.crypto package) encourages people not to use the Android KeyStore and instead to store in plaintext within the app directory.

Warning: because Android Keystore is unreliable, we strongly recommend disabling it by not setting any master key URI.

If we're following Tink's key storage strategy, we can modify our key generation code to st0re the key pair in a shared preference file instead.

private val sharedPreferences = context.getSharedPreferences("key_storage", Context.MODE_PRIVATE)
private val KEY_PUBLIC = "public_key"
private val KEY_PRIVATE = "private_key"

fun getKey(): KeyPair {
    val savedPublicKey = sharedPreferences.getString(KEY_PUBLIC, null)
    val savedPrivateKey = sharedPreferences.getString(KEY_PRIVATE, null)

    // Key has been genereated already, read from shared preference and decode Base64
    if (savedPublicKey != null && savedPrivateKey != null) {
        val keyFactory = KeyFactory.getInstance(KeyProperties.KEY_ALGORITHM_RSA, CONSCRYPT_PROVIDER)
        val pubKey = keyFactory.generatePublic(X509EncodedKeySpec(Base64.decode(savedPublicKey, Base64.DEFAULT)))
        val privateKey = keyFactory.generatePrivate(PKCS8EncodedKeySpec(Base64.decode(savedPrivateKey, Base64.DEFAULT)))

        return KeyPair(pubKey, privateKey)
    } else {
        // Key doesn't exist yet, generate new key pair, and then encode and save to shared pereference
        val keyPairGenerator = KeyPairGenerator.getInstance(
            KeyProperties.KEY_ALGORITHM_RSA,
            "AndroidOpenSSL"
        )
        keyPairGenerator.initialize(2048)
        val keyPair = keyPairGenerator.genKeyPair()

        sharedPreferences.edit {
            kotlin.io.encoding.Base64.encode(keyPair.public.encoded)
            putString(KEY_PUBLIC, Base64.encodeToString(keyPair.public.encoded, Base64.DEFAULT))
            putString(KEY_PRIVATE, Base64.encodeToString(keyPair.private.encoded, Base64.DEFAULT))
        }
        return keyPair
    }
}

Now, you might be wondering if the key stored in plaintext would actually be secured and the answer is "Yes and No". It is "Yes" due to the fact that Android enforce File-Based Encryption since Android 10, which provides a very strong hardware-backed encryption on all your app's internal storage using a hash generated from user's lock pattern or bio-metrics. So as long as your minimum SDK version is not less than Android 10, pretty much everything stored in your internal storage is secured enough that even attackers with physical access cannot retrieve these easily. In fact, this is also the reason Tink gave in their documentation on why you should not use Android KeyStore.

When Android Keystore is disabled or otherwise unavailable, keysets will be stored in cleartext. This is not as bad as it sounds because keysets remain inaccessible to any other apps running on the same device. Moreover, as of July 2020, most active Android devices support either full-disk encryption or file-based encryption, which provide strong security protection against key theft even from attackers with physical access to the device

Side note: Isn't it quite funny that Android which is developed by Google, said to use Android KeyStore but then Tink, which is also developed by Google said not to use Android KeyStore? 🤣

In addition, the only way an attacker could retrieves these data inside your internal app directory is if they rely on a method that require rooting the device. But then if an attacker have God mode enabled on a device, can Android Keystore really prevent it, some doesn't think so. It is also part of the reason why Google deprecated java crypto library because people were getting too comfortable storing sensitive data on the device when in reality, no frontend is secure! So instead of putting our key in AndroidKeyStore and petting ourselves on the back, we should be focusing our effort into data expiration strategies, cache pruning, key rotations, remote log out (which include data wipe), or in the best scenario, a system where we don't need to store sensitive data on the frontend at all. The key takeaway (Pun intended) is that it doesn't matter where you store your keys, the more important thing for your end to end encryption system is to have a Key Rotation Strategy, especially if you are planning to store keys for a long time instead of a one-off exchange. Cheers!