THERE WILL NEVER BE LLM USE ON THIS WEBSITE; THERE WILL NEVER BE LLM USE IN THE CODE I WRITE FOR FREE; BECAUSE I DO NOT HATE MYSELF AND OTHERS;
risky emulation
I wrote a RISC-V (userspace) cpu emulator. It runs 64bit static-pie linux programs on other platforms using a jit compiler (not an interpreter).
September 2026
(this write up is still a work in progress)
Table of Contents
loading elves
I want to be able to run ordinary programs in the Executable and Linkable Format.
The important thing in an ELF file is headers
that describe (what regions of memory are in the program),
(how spaced out they should be in virtual memory) and (what memory protection flags they should have).
Dynamically linked programs have a lot more work for the dynamic loader to do but static ones are fairly simple.
It's a popular enough format that I'm not interested on talking much about the details here.
I already had examples/elf_loader
from a past project so we're off to the races.
compiling traces
The most natural idea would be to translate all the riscv instructions in the binary to native machine code ahead of time and then run the new program.
My (and most?) compiler backend represents programs
as functions (graphs of blocks with one entry point)
containing basic blocks (flat sequences of instructions that end in one jump).
When generating machine code, compilers discard that structure and output
one stream of instructions with arbitrary jumps to offsets in that stream.
Often the jump targets are statically encoded in the instructions but there are also cases that compute them dynamically
(like jump tables (for switch/match) and vtables (for interfaces/traits)).
Nothing in the executable declares where the function and block boundaries will be.
My point is that expecting the shape of program compiled by the emulator to match the original program is a dead end.
could assume something about the structure of the executable
because most programs are compiled mechanically and have boring shapes of control flow.
(like that the elf will have a symbol table and there won't be arbitrary jumps into the middle of functions).
but that feels fragile to me and also won't work later when i want to support running a jit compiler inside the emulator (see later section).
The extreme case of that idea would be to take relocatable object files as input instead of executables.
They preserve much more information about program structure for the linker in the form of relocation tables.
I'm not going to do that because a) I don't want to, b) jit compilers again, and c) I have a healthy contempt for linkers as a concept.
Instead of all that, I'm going to keep the compiler around at runtime and lazily translate instructions as they're needed.
When executing a jump instruction for the first time, decode the flat sequence of instructions until it reaches another jump
and compile that tiny trace as one function that returns the next pc value.
Some top level loop can string them together by calling one and then looking for a cached trace for the next pc or compiling a new one as needed.
Currently I save the registers to memory after executing each trace.
(which is wasteful; better would be to keep them in native registers as much as possible).
I do track which are read/written when compiling the trace so only have to load/store the ones that are actually used.
Returning back to the dispatch loop and doing a hashmap lookup of the new instruction pointer
for every jump instruction the program executes is a significant performance hit.
A better strategy, for constant jumps, is to eagerly start compiling a new trace starting from the destination
and have the old trace directly call the new one instead of looking it up every time.
Doing this turns loops in the guest program into recursion in the emulator which risks running out of call stack space.
Luckily these chained traces are trivially tail recursive,
so I added very limited tail call elimination to my backend
(limitations: only when explicitly enabled, on single block functions, with the same signeture, and no stack arguments)
which is all this emulator needs. Now it only actually returns to the top level for indirect jumps.
Eagerly compiling traces for direct jumps means I'm often compiling code before knowing if it will actually execute.
It's legal to have a conditional jump into garbage memory that just happens to never be taken.
This means I have to add a new way of ending a trace that found an illeigal instruction.
The error needs to be deferred until the trace is actually executed.
For indirect jumps, I keep a couple cache slots on each trace with the previous destination from this trace.
If the instruction pointer value matches, it can skip the lookup and call the trace again.
Real CPUs do much more sophisticated branch prediction but this is a cute start I think.
An extra downside to doing control flow in this convoluted way is that
the native callstack doesn't correlate at all with the emulated program.
So errors can't automatically get nice stack traces through the guest code into the host code.
I've done some compilers for higher level things
(wasm,
lox,
kaleidoscope,
) where one source function maps to one backend function and that presents a much more
understandable view of the universe for external profilers or whatever.
My backend does some
extra work to report jitted code with linux's perf jit interface to make those situations work better.
I'll probably try to extend that to work for this emulator as well eventually.
other tracing compilers
there are also higher level language implementations that work this way.
i think it was more of a thing in the past but there are still some examples:
luajit, pypy, ruby's yjit.
"basic block versioning" is another magic phrase to google if you're interested in this.
it's also like direct threaded code that's popular for forth.
basic instructions
An important disclaimer for this section is I'm mostly only implementing
whatever instructions my own compiler happens to output.
Since I've already written a riscv backend for my compiler and I'm using the same ir for this emulator,
implementing the instructions for the body of a trace that just do computation is just an exercise in looking at
what my other compiler does to output riscv and doing the reverse thing to map them in the other direction.
It's error prone because it takes lots of boring bit fiddling but not many surprises lurk here.
As someone interacting with riscv from software, I'm legally obligated
to complain about how the immediate encodings have the bits mixed around.
I'm told the hardware people like it that way
because it helps you tangle your wires or something
but that's not my department (yet...!).
One really nice thing about the riscv spec is that they have little blurbs about the rational behind some of the choices they make.
Another quirk to be aware of is that since OP_IMM have 12 bit immediates but shifts only need 6 bits,
the encodings that would be (shift/rotate) (left/right) by (some number too big for the register)
are used for other unary operations (like clz,ctz,popcnt,etc.).
here's the riscv isa spec 696 page pdf
The instructions I need are: (notably no compressed ones)
integer: lui,auipc,add,or,and,xor,sltu,sltu,shift/rotate left/right,sub,orn,mul,s/u div,mod,czero,ctz,clz,popcnt,sign extend(8/16),zero extend(16/32)
jumps: jalr,jal,bcmp(eq,ne)
float(32/64): add,sub,mul,div,sqrt,min,max,cmp,load,store
- sgnj cases that mean copy and negate
- many of the fvnvt varients but not all the rounding modes
ecall,ebreak
amo.cas,fence
This took a lot of (run a program, assert on a new instruction, find it in a disassembler, add it to the massive switch statement, repeat) iterations.
At first i was being stubbern and trying to do it without a disassembler and find it in my own compiler's emit code instead but that got old real fast.
Some important tools for the tool box: nix-shell -p pkgsCross.riscv64.buildPackages.binutils; riscv64-unknown-linux-gnu-objdump,
I also like aboutrv.com/en/tools/disassembler because you can type in one hex instruction at a time.
In hindsight maybe I should have just pasted my riscv compiler backend and inverted every switch statement all at once...
But it's easier to catch bugs when doing it incrementally
and it's much better for morale to always have a program that compiles and runs even if it's a tiny boring one.
where the io go
This is not a full system emulator, it's more like qemu-user.
I don't implement any of the privileged instruction set or address translation or anything about emulating memory mapped io that you'd need
to run a real operating system kernel in the emulator.
Instead the emulator stops at the level of linux syscalls and passes them off to the host system.
So the guest program interacts with the real file system, etc.
The easiest host platform is aarch64 because the syscall numbers are the same as on riscv64
(with a special case for openat flags that are different... which suggests there might be others I don't know about yet).
It's not rocket surgery to remap them to let it run on x86_64 too.
Most programs don't use any interesting linux specific functionality
so it's also feasible to translate the syscalls to libc calls to run on macos
(and other operating systems if my language supported them).
I could also have other programs that embed the emulator as a library and define their own syscalls instead.
I haven't done a comprehensive job of any of this yet, just enough to play with a few of my old programs on various platforms.
Some syscalls need special treatment. For example, uname has a field that reports the architecture so i carefully lie about that.
More interesting ones (clone, execve, mprotect, munmap, riscv_flush_icache) are discussed in later sections.
Signals will be much more involved because mcontext is arch specific and if you fault in the middle
of the jitted code have to reconstruct the emulated registers somehow. For now, rt_sigaction is a nop.
Guests should simply not make mistakes :)
guest jit compilers
The most fun part of this project is being able to run my own programs that I wrote before,
many of which are Just-In-Time compilers. Which means I need to support
guest programs that write new machine code into memory and then jump to it.
sadly this means i need a system for invalidating compiled traces.
luckily this is a problem that affects real cpus as well.
so modern ISAs (like riscv) often go out of their way to allow incoherent data and instruction caches.
orange you glad im not doing this for x86
on an isa that has more strict coherency rules, like x86, you'd need to do something
like tracking which store instructions happen to hit code you've already compiled
or checking that the contents of memory hasn't changed before executing it.
you can be a bit more efficient by making sure that memory is mapped as not writable
and catch the fault signal when trying to access it and then do something hand wave-y.
but that's still going to be slow if it happens a lot.
see some notes on trying to get my program to run well in a system that works that way.
The primitive the isa gives you for managing these caches is the FENCE.I instruction.
It isn't part of the linux user space ABI because it would be impossible to use correctly.
It only affects one cpu core (not one thread, and the kernel is allowed to transparently move your thread between cores).
Which is actually great for me because it would also be hard for me to implement efficiently
because it doesn't take a memory range to act on (unlike the arm equivalent instruction: dc cvau / ic ivau).
what linux userspace programs do instead is make the riscv_flush_icache syscall (number 259).
it's parameters are a flag to choose if it should affect the whole system or just one thread (not core) and
the range of memory that must be flushed.
i can implement that syscall by discarding any compiled traces.
i just need to keep track of the map of ranges of memory to the trace
(not just starting instruction to trace).
notably the ranges can overlap because of the fallthrough thing (same reason its hard to know where the trace boundaries will be ahead of time)
also important that munmap acts like a cache flush because its scary/unnatural/disrespectful/etc
for a program to be able to allocate new memory and jump to it and
get whatever random junk was at that address before it was reused.
this makes the direct jump tail optimisation from before more complicated.
need to keep track of incoming traces so can flush them too since they'll be
directly jumping to the invalid old trace instead of looking up the new one.
right now i do this blatently wrong and rely on none of my example programs doing the shape of flushes that would break.
so more work for another day.
clone
currently i only support the combinations of flags that correspond to spawning a thread or forking the process.
i also don't do anything for the tls parameter because my language does thread locals differently
(but i think its just putting that value in register number 4, i haven't gotten around to finding someone else's program to test it yet).
clone has the exciting property that it returns twice (once to the caller and once in a new thread, with different values).
since the jitted code is split into tiny traces that yield back to the emulator often, i could implement threads
by cycling between them running a few traces at a time like how a real operating system preempts on a real cpu.
however, i do spawn a real thread tho which means the tid returned is a real one and any syscalls (or commands on the host system)
that inspect running tasks will work for the emulated ones automatically.
fork is the same except that the new thread has its own copy of the address space
which i couldn't easily fake but just passing it off to the real one works fine.
allowing normal fork needs to be careful that the emulator's compiler doesn't
rely on a separate thread to run the jit because fork doesn't take them with you.
exec
looking at the file to see if it's an elf and then execing the emulator instead
passing along emulator configuration.
this leads to readlinkat(/proc/self/exe) hell.
i special cased that one situation because my programs use it but its a slippery slope.
im sure there's much more that /proc/ can observe that will be inaccurate because they
report the real system not the emulated one.
(not) leaking traces
tracking regions of memory that are mprotected as exec and then dropping the compiled code when those are unmapped.
can we just not bother
could get away with not doing this except that examples/os/host/user.fr wants to implement processes as threads (see the libc section below for a bit more info).
do you are have sandbox
i dont. this is why the title of this article is such a good pun.
since im not doing bounds checks on memory accesses a malicious guest program can reach out
and modify the emulator's internal memory that wouldn't exist in the emulated universe.
im sure there's someway to get arbitrary code execution that way because my backend
tends to be haphazard about leaving W+X memory lying around.
but since the way i implement io is just by forwarding syscalls to their native equivalents,
you can already do anything you want so like whatever, im not in the business of being a sandbox at this time.
if you want that you can run the whole thing in bubblewrap.
(i even have an example that compiles bubblewrap with my c compiler which is a fun party trick
since my language's compile time execution is already too trusting for compiling other people's unknown programs:
tests/external/bubblewrap.fr).
im really learning to embrace the unix philosophy: a program should do half of a thing and do it poorly.
do you are have libc
i dont.
Franca
programs can compile with -syscalls.
Other toolchains can statically link with a libc (often musl),
or use a language that just does the task without cosplaying as c (zig, go, etc., etc.).
This is all hypothetical because other people's compilers
will probably use instructions i haven't implemented yet.
At some point I'll be consistant about supporting whole extensions
so others can be configured to abide by my arbitrary limitations.
i can also run the userspace of my examples/os (which does have parts of a libc).
os/host/user.fr uses the native mmap/unmap/mprotect/futex/clone syscalls and fakes the rest in userspace with an memory file system.
very WIP writeup about that os: [...]
an interesting experiment would be to let the emulator load a dynamic elf and
have the imports resolve to something that remaps the registers and calls the native ones.
so important things to be fast like memcpy or whatever could be native code not going through the emulator.
but i haven't done that yet.
but can it emulate itself
yes... if you're very very patient...
000650ms: FRANCA_NO_CACHE=1 ./host.out examples/kaleidoscope.fr
005800ms: FRANCA_NO_CACHE=1 ./host.out examples/emu/run.fr ./q.out -env_count 1 -- FRANCA_NO_CACHE=1 examples/kaleidoscope.fr
040380ms: FRANCA_NO_CACHE=1 ./host.out examples/emu/run.fr ./q.out -env_count 1 -- FRANCA_NO_CACHE=1 examples/emu/run.fr ./q.out -env_count 1 -- FRANCA_NO_CACHE=1 examples/kaleidoscope.fr
282800ms: FRANCA_NO_CACHE=1 ./host.out examples/emu/run.fr ./q.out -env_count 1 -- FRANCA_NO_CACHE=1 examples/emu/run.fr ./q.out -env_count 1 -- FRANCA_NO_CACHE=1 examples/emu/run.fr ./q.out -env_count 1 -- FRANCA_NO_CACHE=1 examples/kaleidoscope.fr
---
b18a0d091bbc14045543c7794c40f28e949295ed
./target/franca.out examples/default_driver.fr build compiler/main.fr -o host.out -os linux -arch aarch64 -unsafe
./target/franca.out examples/default_driver.fr build compiler/main.fr -o q.out -os linux -arch rv64 -unsafe -syscalls
that's with it re-jitting the emulator at each level and debug assertions enabled in the emulator.
interestingly not the same slow down factor at each level (9x, 7x, 7x).
sane alternatives
qemu or libriscv are probably better than my emulator for every serious usecase
but i don't learn anything by just running git clone so here we are.
i hope you enjoyed your stay.
if you notice a mistake here please tell me about it!
i want to learn information!
you can email anything@lukegrahamlandry.ca
prev: Shapes and Colours Considered Insecure
all