Creating Multiplatform Mobile App with GraalVM Native Image. Philip K. by Han | October, 2022

An in-depth, step-by-step guide to help you build in more places

Unsplash. Photo by David Vihovec on

In this follow-up article, we’ll explore demo apps in iOS and Android with shared cores written in Kotlin/Java. The introductory article is here.

This is a simple voice dictation app. The main point in this demo is to incorporate platform services with the domain model. We’ll first get the speech recognition text from iOS and query a natural language processing (NLP) server to determine whether the captured sentence was a query. And the resulting domain model data will be distributed to view model for further transformation.

ios sample app

The iOS app will demonstrate how to interact with the core component’s C interface. This involves consuming the flow/channel published by the main component and exporting the platform service via C function pointers.

Creating a C interface is mainly done in Java with the GraalVM Native Image C API and some C headers for declaring functions and structs.

This demo app was built with GraalVM Community Edition (CE). GraalVM Enterprise Edition (EE) is available at a cost, but it is unnecessary. If you elect to go with EE instead of CE, you will get access to a richer feature set, especially better GC algorithms.

Please install GraalVM Tools for Java – Visual Studio Marketplace first. Once the extension is installed you will need the following modules. You can also download and install GraalVM using this extension.

This was tested on GraalVM version 22.1.0 or lower. Higher version is not working at this time due to module not being imported correctly. I’ll update you as soon as it’s working.

The architecture here is MVVM with slight adjustments. Of particular note is that the viewmodel is split between the host platform and the platform agnostic viewmodel. It can be shared to allow non-platform-specific viewmodel logic.

The domain model handles all business logic and is the source of truth. It handles most of the concurrency to keep the data consistent with the observations. It is designed in the Actor paradigm, so its updates are published in a systematic manner.

All business logic is represented as pure functions, so it is easy to test.

Let’s look at an example of such a change. here we have StateFlow To listen to the status of the speech recognizer.

val speechRecognizerState: StateFlow<Boolean> =
model.speechRecognitionState

Then we convert this position to Android viewmodel,

val speechRecognizerState = speechCore.speechRecognizerState.map(viewModelScope) { listening ->
if (listening) Pair(R.string.listening, Color.Red)
else Pair(R.string.sleeping, Color.Black)
}

and on the corresponding ViewModel on Swift.

func convertRecognizerState(listening: Bool) -> (String, Color) {
listening ? ("LISTENING", Color.red) : ("Sleeping", Color.black)
}

Well, you must be wondering how a Kotlin StateFlow Can be viewed and passed to Mapper function in Swift. To be sure, there’s a lot of work to do in between, so let’s take a deeper look.

Detach a mini container of the Java Runtime. It maintains the execution stack and heap of your Java/Kotlin code compiled on the target platform.

A detailed article about isolate is here: Isolates and Compressed Reference: More Flexible and Efficient Memory Management via GraalVM | by Christian Wimmer | GrailVM | medium

An isolate must be created first and then used in all native method invocations. Isolate is bound to a specified thread upon creation, and the original invocation must include the thread context. If the original invocation needs to run on a different thread, the detach should be detached and then reattached to the target thread prior to invocation.

The isolate must be destroyed manually and will not be automatically retrieved by the host OS

Different C API:

  • graal_create_isolate
  • graal_attach_thread
  • graal_detach_thread
  • graal_tear_down_isolate
  • graal_detach_all_threads_and_tear_down_isolate

The Isolate C API can be used directly in Swift.

In the demo app, we’ll create a dedicated Swift thread and attach an isolate for the duration of the screen instead of creating and destroying each function invocation.

You must first create a C header file for the declarations. Then import the header file into Java.

All methods we want to expose as C functions must be static and annotated @CEntryPoint in class annotated by @CContext, @CContext is similar to import <header> instructions in C

In the end, you have to accept IsolateThread as an argument in @CEntryPoint Celebration.

More complex data types such as structs are passed and returned as pointers. PointerBase Equivalent to a pointer in C.

@CStruct("CError")
public interface CError extends PointerBase {
@CField("code") void setCode(int code);
@CField("code") int getCode();
@CField("message") void setMessage(CCharPointer message);
@CField("message") CCharPointer getMessage();
}

The above structure is first defined in the C header, then imported into Java. This is the declaration in the C header file as follows:

typedef struct CError {
int code;
char* message;
} CError;

We can only work with types defined in the header file other than primitives. boolean requires a standard header file stdbool.h, String is not a standard type in C. it is a pointer char and terminated by null, It doesn’t mean generic at all.

Note that in C, evaluation is top to bottom. Therefore, you cannot refer to things backwards. Remember that the order of declaration matters in C.

We can also import/export C function pointers. For a function pointer declaration in a C header:

typedef void (*ErrorDispatcher)(void*, char*);

In Java, imported as shown above:

public interface ErrorDispatcher extends CFunctionPointer {@InvokeCFunctionPointer
void invoke(PointerBase ctx, CCharPointer errorMessage);
}

UnmanagedMemory The package provided C types of heap allocation and deallocation.

The allocated memory is released with freeLike in C.

import org.graalvm.nativeimage.UnmanagedMemory;CSpeechBusinessData data = allocData();
UnmanagedMemory.free(data);

NB Native Image C API is most likely to be deprecated and replaced by Project Panama Foreign Function and Memory API. Project Panama is in Java 19 as a third preview.

Swift C Interop is similar to GraalVM C API. They just have different names. PointerBase Swift is equivalent to UnsafeMutableRawPointer And CCharPointer Is equal to UnsafeMutablePointer<CChar>. they are void* And char* in C, respectively.

Also, Swift can represent closures as C using functions @convention(c) Annotation.

let httpPostFunc: @convention(c) (
_ uri: UnsafeMutablePointer<CChar>,
_ speech: UnsafeMutablePointer<CChar>
) -> UnsafeMutablePointer<Either>?

The above declaration can be initialized as:

httpPostFunc = { uri, speech in 
// return http post operation in swift
}

Alternatively, we can assign a function with a matching signature:

httpPostFunc = httpPostRealfunc httpPostReal(
uri: UnsafeMutablePointer<CChar>,
speech: UnsafeMutablePointer<CChar>
) -> UnsafeMutablePointer<Either>? {
// implementation
}

We can pass these function pointer instances to the GraalVM C interface, e.g. CError Example in the previous section.

First, we have to convert collect For the Observer pattern in Kotlin/Java.

Then, in the Java interop layer, we wire in the C callback object.

// recognizerStateDispatcher is C callback provided by the host.
// passed as speechRecognizerStateCallback above.
(Boolean state) -> {
if (recognizerStateDispatcher.isNonNull())
recognizerStateDispatcher.invoke(ctx, state);
}

Here’s how the Swift C interop callback is defined:

In this demo app, we have three projects and five modules.

The Kotlin core module is a pure Kotlin/Java module because Java C Interop references it.

The Java C Interop Module exposes Kotlin core functionalities and maps data structures from Kotlin/Java to C.

The iOS app module then consumes C functions and data structures, as well as provides infrastructure services to C interfaces. In this demo app iOS speech recognition is exported via function pointers.

The Java Agent App module is optional. It is responsible for executing parts of your code with reflection and generating the reflection configuration to be used during AOT compilation. In this case, Moshi. it is needed for KotlinJsonAdapterFactory For reflective Json deserialization. You can also write this configuration manually or use the CodeGen version to eliminate reflections entirely.

The domain model serves as a source of truth for the app. This includes both data and behavior. DTOs are used to build the data model part of the domain model. Behaviors are pure business functions that are very easy to test.

This layer is completely platform and client agnostic. It can be reused to serve any client whether it is mobile or desktop.

It handles concurrency keeping its state constant at all times. It follows the Actor model with the structured concurrency of Kotlin coroutines.

The model layer organizes domain models and services, then exports observable state (platform agnostic view models) to clients.

Exception propagation in C interface is done with Either Monad in C. This is my attempt at implementation:

In addition, we can enable -H:+ReportExceptionStackTraces And -H:+GenerateDebugInfo=1 In build.gradle For a more detailed crash output.

And in case of crash with Xcode you are left in breakpoints in assembly, it is most likely a problem with pointers or memory allocation in your Java C interface code. In these cases, there’s not much we can do but use intuition, then prove/disprove cycles in my experience.

Which one?

Works with AOT with the Reflection configuration file provided in the Resources directory. It has a large runtime memory footprint, so in this app, we use iOS URLSession rather than.

Kotlin Serialization

Json is used with Ktor for deserialization. For AOT, it must be used with the agent to generate the reflection configuration.

Swift URLSession

URLSession needs an extension function to synchronize this because it already runs in the coroutines IO dispatcher. Chose to use the iOS HTTP client because of a significant reduction in memory footprint compared to Ktor for AOT compilation.

moshi

Moshi Json Mapper was used, and it works well except for requiring a config file for mirrored DTO instantiation.

coin

No problem with the coin.

coroutines/flows/channels

The only caveat here is that there isn’t Dispatchers.main, which is only available in environments with a core event loop, such as Android or Swing. you a. have to use callback Handler via the C interface, which fires off the iOS main dispatcher. And the example is in the “Reactive Streams in C/Swift” section above.

The easiest way to run an agent for reflection configuration generation is to use the GraalVM Native Image plugin for VS Code. This will provide Gradle tasks to attach to the agent. You can use this Gradle task command: ./gradlew app:run -P agent. After that, you need to indicate the directory where the generated files are located, for example, app/build/native/agent-output/run

The Stanford CoreNLP server is required to parse the text provided by the speech recognizer. You can download it here:

Download — CoreNLP (stanfordnlp.github.io)

Just extract and run this command from where it was extracted:

#/bin/shjava -mx3g -cp stanford-corenlp-4.5.1.jar:stanford-corenlp-4.5.1-models.jar:joda-time.jar:jollyday.jar:protobuf-java-3.19.2.jar:xom.jar:ejml-core-0.39.jar:ejml-simple-0.39.jar:ejml-ddense-0.39.jar:jaxb-api-2.4.0-b180830.0359.jar:jaxb-impl-2.4.0-b180830.0438.jar edu.stanford.nlp.pipeline.StanfordCoreNLPServer -preload tokenize,pos,parse,sentiment,ssplit

then create file nlpserver.properties below speechcore/src/main/resources/

nlp.host=<your server ip>
nlp.port=9000

After changing this file needs to be rebuilt.

The source code is available on GitHub:

android app repository
shared library repository
iOS App Repository

Leave a Comment