Add aarch64 support by ludfjig · Pull Request #32 · hyperlight-dev/cargo-hyperlight

ludfjig · 2026-03-10T17:44:20Z

Adds support for compiling aarch64, with some fixes for cross compiling from x86_64 as well

syntactically · 2026-05-26T19:11:22Z

@ludfjig I rebased this PR and added some minor fixes used by the main hyperlight PR.

Looks like CI is passing, so hopefully we can merge that soon.

Copilot

Pull request overview

This PR adds support for building Hyperlight guest binaries targeting aarch64-hyperlight-none, and adjusts argument forwarding / toolchain behavior to improve cross-compilation correctness.

Changes:

Add a new aarch64-hyperlight-none Rust target spec derived from aarch64-unknown-none, and update target-spec generation to pass --target explicitly.
Adjust clang/cflags generation to use an aarch64 --target and make -mno-red-zone x86_64-only.
Improve cargo invocation behavior by avoiding canonicalize() on the cargo path and filtering --target / --target-dir from forwarded CLI args; update docs/CI/justfile for multi-arch support.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`src/toolchain.rs`	Adds aarch64-aware clang target selection and changes libc header preparation logic.
`src/sysroot.rs`	Introduces `aarch64-hyperlight-none` target-spec customization and adjusts how base target specs are queried.
`src/command.rs`	Filters `--target` / `--target-dir` from forwarded args to avoid overriding resolved env-based target settings.
`src/cargo_cmd.rs`	Stops canonicalizing the cargo binary path to avoid resolving rustup symlinks.
`README.md`	Documents default target selection on ARM and updates output path description.
`justfile`	Uses `arch()` to make the guest path target-arch independent.
`.github/workflows/ci.yml`	Adds concurrency cancellation and gates KVM / guest execution based on runner architecture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ludfjig

LGTM but I don't have approval rights to my own pr :)

syntactically · 2026-06-25T01:57:54Z

@jprendes I think this should be properly ready to merge now as all the odd only-reproducible-in-CI issues seem to be fixed; would appreciate if you took a quick look at it. It more or less follows the plan we talked about earlier.

jprendes

I think I would have preferred the aarch64 support and the clang wrapper to be different PRs, I think most of the changes in the PR stem from the clang wrapper and associated refactors rather than from adding the aarch64 target. But let's keep it this way now.

jprendes · 2026-06-25T10:04:52Z

+    panic!("Could not base system implementation of {}", tool_name);
+}
+
+fn clang(args: &Args, tool_name: &str, cmd: env::ArgsOs) {


The logic in this function is quite involved, could you add some comments? maybe with example invokations that we are trying to handle in each case?

Does the extra comment in 6bd4e9c look good to you?

jprendes · 2026-06-25T12:24:40Z

    if !args.target.starts_with("aarch64") {
-        // Detect which libc variant is present: picolibc or legacy musl
-        let include_dirs: &[&str] = &[
-            // directories for musl
-            "third_party/printf/",
-            "third_party/musl/include",
-            "third_party/musl/arch/generic",
-            "third_party/musl/arch/x86_64",
-            "third_party/musl/src/internal",
-            // directories for picolibc
-            "third_party/picolibc/libc/include",
-            "third_party/picolibc/libc/stdio",
-            "include",
-        ];
-
-        for dir in include_dirs {
-            let include_src_dir = libc_dir.join(dir);
-            let files = glob::glob(&format!("{}/**/*.h", include_src_dir.display()))
-                .context("Failed to read include source directory")?;
-            for file in files {
-                let src = file.context("Failed to read include source file")?;
-                let dst = src.strip_prefix(&include_src_dir).unwrap();
-                let dst = include_dst_dir.join(dst);
-
-                std::fs::create_dir_all(dst.parent().unwrap())
-                    .context("Failed to create include subdirectory")?;
-                std::fs::copy(&src, &dst).context("Failed to copy include file")?;
-            }
-        }
+        include_dirs.push("third_party/musl/arch/x86_64");


I think musl was never supported with aarch64. We never had a release using musl and with aarch64 support simultaneously.

Indeed, that is why there is no code path here for musl aarch64 arch headers.

ah, sorry, I should have been clearer. If the path doesn't exist it's silently ignored (that's how the glob crate does it), so you can just leave that one in the main list, and not worry about the including it for one case or another.
Just a nit, this is fine too.

jprendes

Just a small question

jprendes · 2026-06-25T14:17:07Z

+    // - `found_non_flag`: If true, there was at least one non-flag
+    //   argument, so the driver will actually be compiling
+    //   something. This is important to catch someone running
+    //   e.g. `clang -v` and avoid inserting the linker arguments,


Would clang -v trigger the found_non_flag = true? I think it wouldn't?
Also, what's with the special handling of "--" and flags_valid?
Can we stop iterating once flags_valid is set?

Yes, clang -v has found_non_flag = false, i.e. there were 0 non-flag arguments, so we do not include the ldflags.

We keep going after flags_valid is set to false because clang -v -- should have that behaviour, but clang -v -- src.c should not. We could bail once both flags_valid is false and found_non_flag is true, but that seemed like an unnecessary optimisation.

So, as it is clang -v -- adds the ldflags: https://play.rust-lang.org/?version=stable&mode=debug&edition=2024&gist=4c81beb817f8ddd6f64a153b45a30258

I think this means we need tests for this.

Ah, good catch, thanks, fixed :)

I also thought tests for this logic would be nice at the time, but I don't think it makes sense to put them in this file that gets copied all over the place and ideally builds quickly, which is why I didn't do it at first. I added some in 04217f7 (now squashed in).

canonicalize() resolves the cargo -> rustup symlink, causing commands like 'cargo rustc' to be invoked as 'rustup rustc' which doesn't understand cargo-specific arguments. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

User CLI args like --target aarch64-unknown-none were passed through to the final cargo build command, overriding the resolved CARGO_BUILD_TARGET env var set by populate_from_args. Filter these out since the resolved hyperlight target is set via environment variable. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

- Add aarch64-hyperlight-none target spec derived from aarch64-unknown-none with hyperlight customizations - Select correct musl arch headers (aarch64 vs x86_64) - Use appropriate clang --target for aarch64 - Make -mno-red-zone x86_64-only (aarch64 has no red zone) - Fix get_spec to pass --target as CLI arg instead of env var Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

- Add concurrency group to cancel stale CI runs - Gate KVM setup on X64 runners - Gate run-guest on X64 (no host support on ARM yet) - Update README to mention aarch64 target - Use arch() in justfile for target-independent paths Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

`Command::exec()` is never used anymore, since its use precludes running on Windows. `CargoCmd::resolve_env()` is not used, because the essentially-identical `Command::resolve_env()` is used instead. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

This started with the observation that we didn't actually build libc headers properly on aarch64, leading to the realisation that we weren't using cargo-hyperlight to build C guests (or anything that depended on most of libc), so all that code was essentially untested. cargo-hyperlight now supports building the guest CAPI library, and can directly output the appropriate cflags/ldflags/libs needed to build and link a C guest. On top of that, it can now build a `hyperlight-config` executable into its produced sysroots which provides the same information, as well as a clang wrapper that uses the same logic for ease of use. This is done in order to allow downstream consumers to build C guests using the same logic as cargo-hyperlight without having to build/install the tool themselves, removing opportunities for e.g. target features to get out of sync. Finally, `cargo-hyperlight build-c-sysroot --sysroot-dir </path/to/sysroot>` will now copy just the final build artifacts (header files, libraries, and the aforementioned hyperlight-config and clang wrapper executables) into a new directory tree, which makes it easy to build a redistributable artifact that bundles everything downstreams need to build C guests (including both build logic and libraries/interfaces themselves). Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Previously, the CI checks that cargo-hyperlight can actually build a guest ran only on the GH hosted runners, which are poorly-specified and not particularly reflective of the environments where cargo-hyperlight is actually called upon. This changes the workflow to also run on the same runners that we use for Hyperlight CI. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

…des) Previously, the sysroot build copied include files from several different directories into the final sysroot include dir, but did not do anything to reset the state of the include directory. If the same sysroot dir was used to build sysroots from multiple different versions of hyperlight-libc, since the headers provided are different, downstream code would get a confusing (and likely to fail to compile) mix of the include files from the two libcs. This changes the way that the include directory (and other copied sysroot directories) are constructed, to ensure that stale files are removed as well as new files being copied. In a somewhat-related issue, when building a C sysroot, cargo-hyperlight needs to build hyperlight-libc (as part of hyperlight-guest-capi), and it turns out that that build was itself getting the output include directory, which meant that rebuidls could also result in stale include files being added. This commit changes the logic involved in building hyperlight-guest-capi to ensure that the output sysroot include directory (which is not yet populated) isn't on the include path. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

jprendes

LGTM

ludfjig force-pushed the aarch64 branch 3 times, most recently from 04c7534 to ddedf1c Compare March 10, 2026 18:39

syntactically force-pushed the aarch64 branch from ddedf1c to 28ac7b5 Compare May 26, 2026 18:29

ludfjig requested a review from Copilot May 27, 2026 17:41

Copilot started reviewing on behalf of ludfjig May 27, 2026 17:41 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread src/toolchain.rs Outdated

Comment thread src/command.rs

Comment thread src/cargo_cmd.rs

ludfjig marked this pull request as ready for review June 10, 2026 16:16

syntactically force-pushed the aarch64 branch 4 times, most recently from 8506520 to baf9c31 Compare June 23, 2026 05:11

ludfjig commented Jun 23, 2026

View reviewed changes

syntactically force-pushed the aarch64 branch 14 times, most recently from de555d3 to db8c8ad Compare June 25, 2026 01:47

syntactically previously approved these changes Jun 25, 2026

View reviewed changes

jprendes reviewed Jun 25, 2026

View reviewed changes

syntactically dismissed their stale review via 93ddc62 June 25, 2026 14:05

jprendes previously approved these changes Jun 25, 2026

View reviewed changes

syntactically dismissed jprendes’s stale review via 12ced53 June 25, 2026 14:33

syntactically force-pushed the aarch64 branch 5 times, most recently from 530197b to 31eed90 Compare June 25, 2026 15:01

ludfjig and others added 6 commits June 25, 2026 16:30

Update examples to 0.15.0

a4a79a2

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically force-pushed the aarch64 branch 4 times, most recently from 04217f7 to be92bdb Compare June 25, 2026 18:05

syntactically added 4 commits June 25, 2026 19:24

Tweak the search logic for ar for better macOS compatibility

47e7290

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically force-pushed the aarch64 branch from be92bdb to f883481 Compare June 25, 2026 18:24

jprendes approved these changes Jun 25, 2026

View reviewed changes

jprendes merged commit a7ef7fb into hyperlight-dev:main Jun 25, 2026
8 checks passed

Uh oh!

Conversation

ludfjig commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

syntactically commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

syntactically commented Jun 25, 2026

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ludfjig commented Mar 10, 2026 •

edited

Loading