A Rust cross compilation journey

I recently had to configure cross-compilation for one of my Rust project, and, oh boy... What a journey ⛵

Manual test on target platform

I wanted to be able to compile on Linux / MacOS / Windows and _x8664 / aarch64.

Before adding the layer of complexity required by cross-compilation, I wanted to test on "plain" Windows and MacOS systems. It would remove a subsequent layer of complexity: confident that the code itself works on the target platform I would be able to focus on cross-compilation itself - which would bring its own layer of difficulties and bugs. One step at a time!

quickemu allowed me to run a Windows and MacOS machine without too much hassle. Installing Rust and cloning my repository allowed me to solve platform-specific issues before diving into cross compilation. This was especially practical for MacOS which is hard and/or expansive to run otherwise.

Platform specific code

Obviously my code relied on a few platform-specific libraries - such as std::os::linux to get files metadata:

// Linux only
use std::os::linux::fs::MetadataExt;

// ...

let attr = fs::symlink_metadata(lnk)?;
let uid = attr.st_uid(); // st_uid() is linux-specific

I had to use std::os::unix instead (Mac and Linux compatible)

// Any Unix system (Linux, Mac...)
std::os::unix::fs::MetadataExt

// ...

let attr = fs::symlink_metadata(lnk)?;
let uid = attr.uid(); // now code is valid for any Unix system

Another method could have been to rely on conditional compilation, which I will probably have to do in order to support Windows. Something along the line of:

#[cfg(target_os="windows")]
fn check_file_permission(){
    // ...
}

Alternatively, I could have used a Rust crate working transparently for all platforms which would hide way a conditional compilation. It would make the code more generic and improve maintainability.

The right cross compilation tool

Rust cross compilation is officially supported, but documentation is somewhat scarce on the subject. Searching for Rust cross compilation tool quickly yields cross, a “Zero setup” cross compilation tool.

cross builds from containers (Docker or Podman) so it does not require anything much apart a few dependencies and a container engine (Docker or Podman) - except to target macOS which is another hell of a story I'm getting to below.

In short I had to install and configure cross. For configuration, amongst other options, I created a Cross.toml file with specific instructions for each platforms such as

# Cross.toml

#
# Linux
#
# Install specific dependencies required to build application
# Here libssl-dev is required to build openssl crate
#
[target.aarch64-unknown-linux-musl]
pre-build = [
    "dpkg --add-architecture arm64",
    "apt-get update && apt-get install --assume-yes libssl-dev:arm64"
]

[target.x86_64-unknown-linux-musl]
pre-build = [
    "dpkg --add-architecture amd64",
    "apt-get update && apt-get install --assume-yes libssl-dev:amd64"
]

#
# macOS
#
# Use a custom image as Cross does not provide image out-of-the-box
# See https://github.com/cross-rs/cross-toolchains?tab=readme-ov-file#apple-targets
#
[target.x86_64-apple-darwin]
image = "x86_64-apple-darwin-cross:local"

[target.aarch64-apple-darwin]
image = "aarch64-apple-darwin-cross:local"

It worked (almost) magically as advertised on both Linux target...

cross build --target aarch64-unknown-linux-musl
cross build --target x86_64-unknown-linux-musl

... except for a tiny issue with openssl which wouldn't compile with Linux musl unless vendored Cargo feature is used. Updating Cargo.toml solved the issue:

# Cargo.toml

# See https://github.com/sfackler/rust-openssl/issues/1627
# and https://docs.rs/openssl/latest/openssl/#vendored
[target.x86_64-unknown-linux-musl.dependencies]
openssl = { version = "0.10.62", features = ["vendored"] }

[target.aarch64-unknown-linux-musl.dependencies]
openssl = { version = "0.10.62", features = ["vendored"] }

As for *-apple-darwin targets I had to (ominous voice) package the macOS SDK locally 😨

Targetting macOS / Darwin

cross supports quite a few targets but redirects to cross-toolchains for Apple / Darwin. Apple target images are not provided due to apparent licensing issues - instructions are given to package macOS SDK through oscroxx in order to build a custom container image.

Well. I knew macOS cross-compilation was difficult, but what the actual **** ! It was unbelievably hard to achieve. Apple seems committed to make it impossible to do anything Mac-related without natively using their platform. Well done, Apple, well done, now I'm committed to just do it for the sake of fighting this stance.

Anyway, here's the short how-to version:

Create a developer account on developer.apple.com to download Xcode
- Of course for some reason account creation didn't work from my Linux machine. I had to go through the macOS box I created earlier with Quickemu for registration process to work.
- xcodereleases.com provides a handy list of Xcode release (though you'll still need an account)
Package the SDK following oscrossx instructions

Build *-apple-darwin-cross container images locally following cross-toolchains instructions. Something along the line of:

# Clone repo and submodules
git clone https://github.com/cross-rs/cross
cd cross
git submodule update --init --remote

# Copy SDK to have it available in build context
cd docker
mkdir ./macos-sdk
cp path/to/sdk/MacOSX13.0.sdk.tar.xz ./macos-sdk/MacOSX13.0.sdk.tar.xz

# Build images
docker build -f ./cross-toolchains/docker/Dockerfile.x86_64-apple-darwin-cross \
  --build-arg MACOS_SDK_DIR=./macos-sdk \
  --build-arg MACOS_SDK_FILE="MacOSX13.0.sdk.tar.xz" \
  -t x86_64-apple-darwin-cross:local .

docker build -f ./cross-toolchains/docker/Dockerfile.aarch64-apple-darwin-cross \
  --build-arg MACOS_SDK_DIR=./macos-sdk \
  --build-arg MACOS_SDK_FILE="MacOSX13.0.sdk.tar.xz" \
  -t aarch64-apple-darwin-cross:local 
  .

I had in fact to try multiple versions of Xcode since cross-toolchains uses a specific commit of oscrossx which does not (yet) support Xcode > 13.0

Run
```
cross build --target x86_64-apple-darwin
```
which will leverage locally available *-apple-darwin-cross:local images for cross-compilation

Et voiilà ! A working macOS binary built with Rust from Linux 🥳

$ file novops
novops: Mach-O 64-bit x86_64 executable, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|PIE|HAS_TLV_DESCRIPTORS>

Here's the PR I finally crafted for novops cross-compilation.

Wrapping-up

So cross-compiling Rust is no easy feat, especially targetting macOS / Darwin - but the journey alone with all its lessons and learning is worth it !

Of course I could also simply spin-up GitHub Actions matrix jobs with the OS I want to target, but that wouldn't be fun, would it?

Post Views: 4,921