A CPU in SystemC, without RTL

How far can a model without RTL be trusted about time, and what does the precision cost?

I wanted to answer that with a machine small enough to finish: a five-stage RISC-V core written in SystemC with TLM-2.0. No hardware description language, no waveforms, no synthesis. I checked its instruction set against an independent simulator, then timed the same pipeline twice, once with a scoreboard and once with one thread per stage.

Before the details, the caveat that matters most: every cycle count in this post comes from a model I wrote, not from silicon. The model follows its rules exactly, and I can show that. Whether those rules describe a real core is a different question, and I did not answer it.

Every measured number below comes from a result file that a script produced, and the tables are generated from those files. The hand calculation is worked out in full in its own section.

01Setup

The core implements RV32I plus mul. I build it with -march=rv32im and reject the other multiply and divide instructions (mulh, mulhsu, mulhu, div, divu, rem, remu) with an explicit error. There are no CSRs, no interrupts, no MMU, no floating point and no second hart. The pipeline is in order, single issue, with five stages: IF, ID, EX, MEM and WB.

Memory is flat: 1 MiB of RAM at 0x80000000. The MMIO page at 0x10000000 has two registers. A store to 0x10000000 prints its low byte. A store to 0x10000004 ends the simulation, and the stored value becomes the exit code.

The model is trace-driven. The instruction-set simulator (ISS) executes the program, and the timing models consume the retired instructions. Wrong-path fetches are not simulated.

There are three hardware variants. Every timing parameter lives in config/variants.cfg:

These rules fix the penalties. A taken transfer resolved in EX costs 2 cycles. A taken branch predicted in ID costs 1 cycle. A misprediction costs 2 cycles. An instruction that uses a load's result right after the load waits 1 cycle with forwarding and 2 without it. Without forwarding, an instruction that reads the result of the one before it waits 2 cycles.

A run's cycle count is the cycle in which the last instruction leaves WB, plus one. A program of N instructions with no hazards takes N + 4 cycles. The exit store counts as an instruction.

02What the model does not model

• Wrong-path fetches. A mispredicted branch pays its penalty, but the squashed fetches never reach the cache.
• Structural hazards. Instruction and data accesses have separate ports. There is no bus or write-port contention.
• Memory. RAM takes one cycle in H0 and H1. In H2 a miss costs a fixed 10 cycles. There is no DRAM, no write buffer, no prefetch and no replacement policy beyond direct mapping.
• Arithmetic. mul takes one EX cycle. There is no divider, because none is implemented.
• Prediction. Only the static rule above. There is no branch target buffer and no return address stack, and jalr always waits for EX.
• Traps and misalignment. No traps. Misaligned loads and stores complete in one step.
• Physical time. There is no clock frequency, power or area. A cycle is a cycle; nothing here knows a nanosecond.

03The ISS

The decoder is one switch on the opcode. The M-extension case shows how the model rejects what it does not implement:

Every memory access goes through a TLM-2.0 b_transport call. The CPU builds one generic payload per access and sends it to the bus, which forwards it to RAM or MMIO according to the address. The bus, the RAM and the MMIO target are three SystemC modules:

View larger ↗

The MMIO target does two things, and it is the only place where the program talks to the host:

The RAM latency is annotated in the transport delay. It is zero in the ISS, so the CPU never waits on it. That is the whole memory model.

Test suite. I took the rv32ui list from the riscv-tests repository at commit 7b68f7b. It has 42 programs. Their reference environment, env/p, depends on CSRs and traps, so I wrote a replacement with the same macros: no CSRs, and a test ends by storing its result to the exit register. The misaligned-access test checks values rather than traps, so it passes with misaligned accesses handled in one step. Result: 42 of 42 pass.

Decoder cross-check. llvm-objdump decodes the same binaries. For 10,317 instruction words in 47 ELF files (the 42 tests and the five kernel programs below), the address, the word and the mnemonic agree. The encodings the model rejects on purpose are reported by llvm-objdump as system or M-extension instructions. Result: 0 mismatches.

04The reference check

The reference is QEMU 8.2.2 in user mode, qemu-riscv32-static, single-stepped with -singlestep. QEMU prints the register file before each instruction, along with the instruction word. The test binaries carry an MMIO shadow section at 0x10000000, which exists so that QEMU maps the putchar and exit registers; the model never loads it into RAM. For every instruction, the trace must agree on:

• the PC and the instruction word;
• the destination register and its value, read from QEMU's registers after the instruction;
• for loads and stores, the address, the size and the value, computed from QEMU's registers before the instruction;
• the end of the run: the instruction after the exit store is unimp in both.

The field extraction is written in Python, so the C++ decoder does not check itself. The programs are the four kernels in a small size, plus the microbenchmark: fib (2,014 instructions), bubble (962), dot (1,044), matmul (2,366) and the microbenchmark (20). Result: 15,430 checks, 0 mismatches.

A comparison that never fails proves nothing, so I tested the checker. I made a copy of the simulator in which add returns a + b + 1, and ran the same comparison on it. The checker flags every program: 365 mismatches in fib, 44 in bubble, 65 in dot, and 5 in the microbenchmark. In matmul the mutant does not reach the exit store at all, because an address goes out of range, and the checker records that as a mismatch.

05The workloads

• fib: naive recursive Fibonacci of 20. Every call is a jal and every return a jalr. 247,578 instructions.
• bubble: bubble sort of 128 values from a linear congruential generator. Its branches depend on the data. 51,599 instructions.
• dot: dot product of two vectors of 4,096 integers. Each iteration loads two values and multiplies them, so the multiply waits for the second load. 65,557 instructions.
• matmul: product of two 8×8 integer matrices, repeated 8 times. 15,447 instructions.
• micro: a 20-instruction loop, worked out by hand below.

The kernels are C compiled with clang -O2. The microbenchmark is assembly. The model has no divide or remainder instruction, so the kernels avoid % and /. Every run's exit value matched a Python computation of the same result.

06The pipeline

There are two timing models, and they share one set of rules. The rules live in one header: where a control transfer redirects fetch, and how the cache behaves.

The scoreboard runs inside the instruction loop. Each retired instruction gets the cycle in which it enters each stage, from a few recurrences. This is the step for ID: the instruction leaves ID only when its operands can be read in time for EX.

The stage-thread model has one SC_THREAD per stage, driven by an sc_clock. Its latches are sc_signals, so each stage reads the state of the previous cycle and writes the next, as hardware does. The same step, in this model:

The two must give the same cycle count everywhere. They do: for each of the 15 kernel and variant combinations, five runs in each model give identical cycle counts. This is an internal cross-check. It shows that the two implementations agree with each other, not that either agrees with hardware.

The hand calculation

The microbenchmark is a four-iteration loop:

It executes 20 instructions. The comments number the source lines, from 0 to 7. The trace numbers the executed instructions from 0 to 19: instructions 0 and 1 come first, then each iteration contributes 2 to 5, 6 to 9, 10 to 13 and 14 to 17, and the last two are 18 and 19. The text and the figures use these trace numbers.

The hazards are these:

• Instruction 3 uses the value loaded by instruction 2, the one before it. This happens in every iteration.
• Instruction 5 tests the value that instruction 4 has just written. Also in every iteration.
• Instruction 2 uses t1, which instruction 1 wrote, in the first iteration only.
• Instruction 19 stores t3 right after lui (18) has written t6.
• Branches: instructions 5, 9 and 13 are taken, backwards. Instruction 17 is not taken. It is backwards, so H1 predicts it taken and pays for the misprediction.
• Caches (H2): the eight static instructions occupy two 16-byte lines, so the first pass causes two instruction misses. The data cell is one line, so the first load misses once. The stores go to the uncached MMIO page.

With no stalls, the trace takes 20 + 4 = 24 cycles. Each variant adds the following:

• H0: every dependency on the instruction just before waits 2 cycles, which is 10 waits (instructions 2 in the first iteration, then 3, 5, 7, 9, 11, 13, 15, 17 and 19). That is 20 cycles. The three taken branches each pay 2 cycles, which is 6. 24 + 20 + 6 = 50 cycles.
• H1: the four load-use pairs (3, 7, 11 and 15) wait 1 cycle each, which is 4. The three correctly predicted taken branches pay 1 cycle each, which is 3. The mispredicted branch (17) pays 2. A back-to-back addition followed by a branch costs nothing with forwarding. 24 + 4 + 3 + 2 = 33 cycles.
• H2: the 33 cycles of H1, plus the first instruction-fetch miss (10 cycles) and the first data miss (10 cycles): 53 cycles. The fetch miss for the second line runs while instruction 3 waits for the first load's data, so it costs nothing.

The model reports 50, 33 and 53 cycles in both implementations. The hand calculation and the model agree in all three variants.

The three figures below show where those cycles go, one per variant. Each one is folded into bands of 18 cycles so that it stays readable on a phone. Each cell holds the trace number of an instruction in a stage. A dark cell is the arrival of that instruction, a grey cell is a wait for an operand or a blocked stage, and an orange cell is a wait for a cache miss.

View larger ↗

View larger ↗

View larger ↗

The kernels

The first table gives cycles and cycles per instruction (CPI) for each kernel and variant. The numbers come from the scoreboard, and the stage-thread model gives the same values.

View larger ↗

Two things stand out. On fib, H1 and H2 are almost the same, because the program's working set fits the cache and the miss penalty rarely applies. On dot, H2 is much worse, at a CPI of 3.69. The two arrays take 32 KiB together, far more than the 1 KiB data cache, and each miss costs 10 cycles.

The miss penalty is my choice, not a measurement, so I varied it:

On fib the CPI hardly moves. On dot it scales with the penalty, from 2.44 cycles per instruction at 5 cycles to 6.19 at 20. The qualitative result holds at every penalty: under H2, dot is memory-bound and fib is not. The absolute number depends on the penalty, and that is the part of the model I could not check.

07The cost of precision

Simulation speed is measured in simulated MIPS: simulated instructions per host second. Each configuration ran five times, pinned to CPU 0. On this machine, an Intel Core Ultra 7 258V with four performance cores, four efficiency cores and no SMT, CPUs 0 to 3 are the performance cores; lscpu reports them at 4.7 to 4.8 GHz, against 3.7 GHz for CPUs 4 to 7. The table gives the median, and in brackets the minimum and maximum over the five runs.

View larger ↗

The ISS alone runs at 18 to 26 MIPS. The scoreboard runs at 16 to 20 MIPS. For the ISS and the scoreboard, the spread between runs is 1 to 3 MIPS, about the size of the gap between them on fib and matmul. So the MIPS table cannot settle the scoreboard's overhead alone. Instruction counts are steadier. Callgrind counts the host instructions each mode executes per simulated instruction, and it needs no timing at all, because hardware counters are not available to this user:

The scoreboard executes about a quarter more host instructions per simulated instruction than the ISS. The stage-thread model executes about twelve times as many. The percentages are a heuristic: functions are sorted into groups by name. About 42% of the stage-thread cost sits in the SystemC kernel, which schedules the threads and updates the signals. Half of it is in libc and libstdc++, whose functions I did not break down further.

The two models pay for different things. The scoreboard pays per instruction. The stage-thread model pays per cycle, so the kernels with a high CPI run slowest in MIPS. It processes 1.2 to 1.7 million simulated cycles per host second, which is 0.5 MIPS for dot under H2 and 1.1 MIPS for matmul.

So the precision costs this much. A scoreboard that reproduces the cycle counts of the stage model costs about a quarter more host instructions than the ISS. The stage model costs about twelve times the ISS. For cycle counts alone, the cheap model is enough. What the stage model adds is the cycle-by-cycle view in the pipeline figure, and that is worth its cost only when someone is looking at the pipeline.

08Limitations

• The timing rules are mine. The two models agree because they implement the same rules, and the microbenchmark agrees with a calculation made with the same rules. Neither says anything about a real core.
• The miss penalty is a parameter I chose, and the memory-bound result scales with it.
• The model is trace-driven and has no wrong-path behaviour, so the cache pollution caused by a misprediction is missing.
• There are four kernels and one microbenchmark, each at one size.
• Host timings come from one machine, five runs per configuration, with the spread shown. The instruction counts from callgrind are the steadier measure of cost.
• The direction is checked in EX. A predicted branch is redirected in ID, but its direction is verified in EX, where it reads its operands like any EX instruction. A design that verified the direction in ID would wait for its operands; the branch article measures that design. Both timing models share this rule, so the cross-check cannot test it.

09How far can this be trusted?

The implementation follows its rules, and three separate checks show it. The ISS agrees with QEMU instruction by instruction. The two timing implementations agree on every cycle count. A hand calculation agrees with the model in all three variants.

That does not say the model is right about silicon. For comparisons inside the model, such as what one more stall costs or what the cache does to a loop, it is useful. For an absolute CPI, it depends on parameters that a measurement would have to fix first.

10Reproduction

Everything runs from the cpu-systemc folder. The first target fetches SystemC 2.3.4, the lld 18 linker, QEMU user mode and riscv-tests into deps/, without root:

Each step writes its results to results/, as JSON or CSV. make figures and make numbers read only those files. The folder's README describes the layout and the options.