Can an LLM Optimize My Code Better Than I Can?

A few weeks ago I wrote a detailed article about optimizing integer exponentiation in C++. I went deep — binary exponentiation, assembly analysis, SIMD batch processing, hand-written cmov loops. The whole thing took me weeks of profiling, reading Intel manuals, and staring at Godbolt output. I got a solid 7x speedup over std::pow and felt good about it.

Then I built AutoPerf, a tool that uses LLMs to optimize C++ kernels automatically. And I thought: what happens if I give it the same problem?

How AutoPerf Works

The idea is simple. You give it a naive kernel. It calls an LLM, gets back optimized code, compiles it, tests it for correctness, benchmarks it. If it is faster, it keeps it and iterates. If not, it tries something else.

The search uses beam search. Here is how it works:

View larger ↗

At each depth level, the tool generates B variants of each candidate, benchmarks them all, and keeps the top-K (the ones with the lowest latency). The pruned candidates are discarded. This repeats for D levels. Not all variants compile or pass tests, so the actual number of successful evaluations is lower than K x B x D. I used Gemini 3 Flash via OpenRouter for all runs — a fast, cheap model that handles C++ well. Each evaluation costs about 6 seconds: LLM call (~3s) + compile (~0.4s) + test (~0.1s) + benchmark (~2s). A full run of ~50 evaluations takes a few minutes and costs under $0.50.

The LLM sees the current code, the current performance in ns/op, the hardware info (CPU model, cache sizes, SIMD support), and the history of previous attempts with their scores. That is it. No assembly, no perf counters, no fancy meta-prompting. Just "here is the code, here is how fast it is, make it faster." Correctness is verified by a test harness that checks known input-output pairs (e.g., 2^10 = 1024, 3^5 = 243, edge cases like base=0 and exp=0).

The Experiment: Integer Exponentiation

I started with the most naive possible implementation:

uint64_t pow_int(uint64_t base, uint64_t exp) {
    uint64_t result = 1;
    for (uint64_t i = 0; i < exp; ++i)
        result *= base;
    return result;
}

O(n) multiplications. Worse than std::pow. This is the starting point I would never write in real life — but it is a fair test. Can the LLM discover binary exponentiation on its own?

Run 1: K=2, B=2, D=2

The very first variant the LLM returned was binary exponentiation. It discovered the O(log n) algorithm without being told. That is expected — this is textbook knowledge and any decent model knows it.

But it also found two tricks I did not have in my hand-optimized powerix library:

1. base == 2 fast path: return (1ULL << exp) — zero multiplications, just a bit shift
2. First-iteration peeling: avoid the initial result = 1; result *= base by setting result = base directly when the first bit is set

Result: 5.0 ns → 2.08 ns (2.4x speedup).

Run 2: K=3, B=3, D=3

With more search budget, the LLM found __builtin_ctzll — count trailing zeros. For an exponent like 20 (binary: 10100), there are two trailing zeros. Instead of looping through them and testing exp & 1 each time (which is always false), it skips them in one shot and does the squarings directly.

Result: 5.0 ns → 1.95 ns (2.7x speedup).

Run 4: K=7, B=7, D=5 (the big one)

I pushed the search budget to 203 evaluations. This time AutoPerf saved every intermediate version, so we can watch the optimization evolve step by step.

View larger ↗

Step 1 — Binary exponentiation (iter #2, 3.94 ns): The LLM immediately discovers O(log n) squaring. Every LLM I tested finds this on the first try — it is textbook knowledge.

// Iter #2: basic binary exponentiation
while (exp > 0) {
    if (exp & 1) result *= base;
    base *= base;
    exp >>= 1;
}

Step 2 — Edge cases + optimization (iter #3, 1.96 ns): A 50% jump. The LLM adds base == 0 handling and the compiler optimizes the cleaner code path better.

Step 3 — ctzll + branchless mask (iter #21, 1.62 ns): The best result. Uses __builtin_ctzll to skip trailing zeros in the exponent, adds early returns for exp <= 2, and uses a branchless mask trick to avoid the if (exp & 1) branch:

// Iter #21: the winning version
uint64_t pow_int(uint64_t base, uint64_t exp) {
    if (__builtin_expect(exp == 0, 0)) return 1;
    if (exp == 1) return base;
    if (exp == 2) return base * base;

    uint64_t result = 1;
    int trailing_zeros = __builtin_ctzll(exp);
    exp >>= trailing_zeros;
    while (trailing_zeros--) base *= base;

    while (exp > 1) {
        uint64_t mask = -static_cast<int64_t>(exp & 1);
        uint64_t temp = (base & mask) | (1ULL & ~mask);
        result *= temp;
        base *= base;
        exp >>= 1;
    }
    return result * base;
}

The branchless mask is clever: instead of if (exp & 1) result *= base, it computes temp = (exp is odd) ? base : 1 via bit manipulation, then always multiplies. No branch, no mispredict.

After iter #21, nothing improved. 180 more evaluations across depths 3, 4, and 5 — all regressions. The LLM hit a plateau it could not escape.

The Search Landscape

View larger ↗

203 evaluations, 4 improvements (green stars), 197 regressions (red dots). The best was found at depth 2 after just 2 minutes. The remaining 48 minutes of compute produced nothing better. This is a fundamental pattern: LLM-driven optimization has diminishing returns. The low-hanging fruit is picked in the first few evaluations.

I also ran AutoPerf on a naive matrix multiplication kernel (i,j,k triple loop): 4.1x speedup (126K → 30.7K ns). The LLM wrote a tiled 6x16 GEMM micro-kernel with 12 FMA accumulators.

How Does It Compare to My Hand-Optimized Code?

I benchmarked all three approaches with base=3 across 12 exponents (0, 1, 2, 3, 5, 8, 10, 13, 15, 20, 31, 63), compiled with g++ -O3 -march=native, pinned to core 0. The AutoPerf benchmark harness uses a different dataset: 7 bases (2, 3, 4, 5, 7, 11, 13) x 10 exponents (0, 1, 2, 3, 5, 8, 10, 13, 15, 20) — so the aggregate ns/op that the LLM optimizes for is weighted toward small exponents. Here is the per-exponent breakdown — this is where it gets interesting.

View larger ↗

exp	naive	human (hierarchical)	LLM (best)
0	0.59	0.41	0.44
1	0.57	0.31	0.47
2	0.75	1.17	0.42
3	0.63	1.56	1.04
5	1.89	1.52	1.44
8	1.23	0.52	1.14
10	3.35	0.51	1.61
13	2.11	0.62	1.89
15	3.00	0.61	1.91
20	4.78	0.79	1.67
31	7.21	0.86	2.32
63	16.65	1.16	3.16

The crossover happens around exp 3-5. Below that, the LLM wins with early returns and ctzll. Above that, my hierarchical version wins — GCC unrolls the recursion into straight-line code that keeps the branch predictor happy, and the cost stays nearly flat regardless of exponent size.

The Insight: You Optimize What You Measure

This is the most important takeaway. The LLM optimized for the benchmark dataset — a mix of 7 bases and 12 exponents, where small exponents (0, 1, 2, 3, 5) make up almost half the evaluations. With that distribution, early returns and special cases pay off. The LLM correctly identified this and spent its optimization budget there.

If I had weighted the benchmark toward large exponents (say, only exp >= 10), the LLM would have seen different performance feedback and likely converged on a different algorithm — possibly something closer to my hierarchical approach, or perhaps the cmov-based iterative version from the powerix article.

The optimizer does not find the "best" algorithm. It finds the best algorithm for your benchmark. This is true of LLM-driven optimization, traditional auto-tuning, and even hand optimization. The difference is that a human can reason about "what if the workload changes?" A benchmark-driven optimizer cannot.

What the LLM Never Found

Across all three runs (147 evaluations total), the LLM never discovered:

• Recursive hierarchical exponentiation — the divide-and-conquer approach that GCC unrolls into near-optimal code
• cmov (conditional move) — the branchless technique I analyzed in assembly. The LLM does not see assembly.
• SIMD batch processing — processing 4/8 bases at once with AVX2. The kernel signature takes a single base, so the LLM had no reason to think about batching.

These are all things that require either understanding what the compiler does with your code (assembly analysis), or rethinking the problem at a higher level (batching). The LLM works within the function signature it is given.

Bonus: Prefix Sum — Can the LLM Find a Parallel Scan?

Prefix sum is a harder problem than pow or reduce. The naive version has a strict data dependency: each output depends on the previous one. You cannot just SIMD it — you need a parallel scan algorithm.

// Naive: strict sequential dependency
void prefix_sum(const float* input, float* output, size_t n) {
    output[0] = input[0];
    for (size_t i = 1; i < n; ++i)
        output[i] = output[i - 1] + input[i];
}

I gave this to Gemini Flash with K=5, B=5, D=5 and --show-dataset. Result: 6.9 us → 3.2 us (2.14x speedup) on 16K elements, with 6 improvements across all 5 depth levels.

The LLM found the SIMD parallel prefix sum pattern: Hillis-Steele scan within each 8-float AVX2 register (shift-and-add by 1, 2, 4 positions), then cross-lane carry propagation via vpermute2f128 + vpermute_ps. This is a non-trivial algorithm — the cross-lane carry between the two 128-bit halves of an AVX2 register is the hard part, and the LLM got it right.

What makes prefix_sum different from pow_int: the LLM kept improving at every depth level (6855 → 5411 → 4505 → 3506 → 3254 → 3207 ns). With pow_int, it plateaued at depth 2. The reason: prefix_sum has more optimization surface — the LLM progressively unrolled from 16-element to 32-element blocks, refined the carry propagation, and extracted the intra-register scan into a lambda for better inlining. Each step was a small refinement, not an algorithmic breakthrough.

What I Actually Learned

1. LLMs are good at applying known optimizations. Binary exponentiation, SIMD reduction, cache tiling, FMA — these are all well-documented techniques. The LLM applies them correctly and often finds combinations I did not think of (the base==2 bit shift, the ctzll trick).

2. LLMs do not reason about what happens after compilation. They cannot see the assembly, they do not know what the branch predictor is doing, they do not know which instructions the CPU can overlap. My hierarchical version wins at large exponents because GCC's unrolling creates code the CPU loves — but the LLM has no way to know that.

3. Search budget has diminishing returns. Going from K=2 to K=7 helped, but the best result was always found in the first 20-30 evaluations. The remaining 170+ evaluations in the big run produced nothing. More compute is not a substitute for better information (assembly, perf counters).

4. You optimize what you measure. The LLM found the best algorithm for this benchmark, not the best algorithm in general. Change the exponent distribution and you change the winner. This is true of all optimization — automated or manual — but it is easy to forget.

5. Humans and LLMs optimize differently. I spent weeks understanding the CPU pipeline. The LLM spent minutes trying things. I found the asymptotically optimal approach. The LLM found the practically fastest one for the common case. The ideal would be to combine both: use the LLM for rapid exploration, then apply hardware-level understanding to refine the winner.

AutoPerf is on GitHub: AutoPerf. The powerix library with the hand-optimized kernels: powerix.