Here’s the problem. Take a neural network--32 layers, 256 neurons wide, a Multi-Layer Perceptron (MLP, the architecture LLMs grew out of) with its weights set randomly at a very specific setting called “the edge of chaos.” Feed it random noise. Don’t run it. Just look at the weights and guess, for each of the 256 output neurons, what the average output would be if you did run it a billion times.
You’re also given a strict FLOP (compute) budget. Brute-force Monte Carlo (just sample tons of times) would eventually solve the problem, but you don’t have anywhere near enough computation, so the challenge becomes: where can mathematical insight replace raw compute?
That’s the ARC White-Box Estimation Challenge, run with $150,000+ across two phases--Phase 1’s $50,000 (split across score, algorithmic contribution, and discretionary community-contribution money) is exactly half of Phase 2’s $100,000. ARC is the Alignment Research Center, a nonprofit that thinks about how to check whether a powerful AI system is safe without just running it a huge number of times and hoping it doesn’t slip up on the cases you didn’t think to try. Their own framing is blunt: “a sufficiently capable AI system is unlikely to fall for ‘honey-pots’,” so whether it would undermine human control in unusual situations isn’t reliably answered by running it on a pile of inputs. Their bet: “a neural network isn’t a black box, we have its weights, and an algorithm that exploits them should do better than one that only observes outputs.” Read a model’s internals well enough to predict what it will do without running it, and you get a check that a clever, deceptive model can’t just wait out.
Random MLPs are the training wheels: no deception to worry about yet, just a hard, checkable math problem, in the same shape as the real one, that lets people build the algorithmic toolkit before pointing it at anything that might be trying to fool you.
If you’ve been reading this blog you’ll know this stuff is right up my alley.
Two caveats worth being upfront about. First, I entered Phase 1 myself--my focus was on understanding the concepts and the math, and my own tuned entry landed solidly mid-pack (133 of 311 entrants, although I briefly basked in the #1 spot at the start of Phase 1 by sheer happenstance of still being awake). Second, this is still a self-selected slice: one write-up reports scanning 1,405 graded entries and finding 864 that scored better than its own, almost all undocumented. Score prizes reward keeping your method quiet; the contribution prize rewards explaining it, and the field split accordingly. What follows is what people have publicly revealed, not the full extent known in Phase 1.
Method 1: math alone doesn’t get you there
The obvious move is to skip sampling entirely and use the analytic approach. Push the shape of the distribution through the network layer by layer, using formulas, and read off the mean at the end. Fast. Elegant. No randomness needed. This is what I did.
The final week’s write-ups measured precisely how much a pure analytic approach doesn’t compete in Phase 1. One team built a protocol that’s worth stealing for any field: take your estimator, swap one internal quantity for its true value (which you know, because you built the test), and measure how much better you get. That number is a ceiling on every idea--including ideas nobody has had yet--whose benefit flows through that quantity. Their result: a perfect version of the standard formula method, with every approximation replaced by the exact truth, would have placed about fifth. A god-mode version tracking every statistical property they could name barely ties second. The formulas aren’t losing because people implemented them badly. The class loses Phase 1.
A second team measured the same ceiling from the other side, by training a neural network to be the formula--half a million random networks as training data, backpropagation through the whole 32-layer recursion. Their honest headline: the entire remaining research program of making formulas better is worth about a factor of two on the score. Their other headline is the best cautionary tale of the competition: the sophisticated learned mechanism their architecture was designed around turned out, on post-hoc inspection, to have trained itself to zero--it consumed 61% of their compute bill and contributed nothing. They found out by deleting it and watching the fifth significant figure move. Ablate the thing you shipped, not the thing you designed.
And there’s one clean win in this family that I can’t explain: a single mysterious constant, roughly 0.992, that you multiply the formula’s answer by for a free 3x improvement. Two teams have now independently converged on it by trial and cross-validation--0.9921, then 0.99278--and nobody can derive it. It’s still open. Let me know if you know why.
Method 2: just sample, but sample smart
If formulas alone don’t work, the fallback is: run the network on random inputs and average. This is the safe, boring, correct answer, and it remains the actual frontier for Phase 1. The final week added calibration: the pure-sampling plateau was predicted at 6.24e-7 by a back-of-envelope identity and observed at 6.47e-7--and then someone noticed the grader itself runs its own reference sampler on every submission, and its number agrees. When your napkin math, your measurement, and the referee’s own instrument all land within 4%, that part of the problem is over.
Once sampling noise dominates everything else, the familiar 1/√N law takes over: every extra digit of accuracy costs roughly a hundred times as many samples.
Method 3: geometry, now solved with theorems
Because of a quirk in how this particular network is built (it only cares about direction, not size, at the first step), the problem reduces to averaging over directions on a sphere. Early in the write-up wave, two teams independently built the same magic set of 66,048 directions--a structure called a Kerdock design: a carefully arranged collection of directions that spreads samples over the sphere as evenly as mathematics allows. Using Kerdock, the different teams got the same answer. When two people arrive at the identical structure without talking to each other, it’s usually because the structure was there to be found, not invented.
The final week of write-ups turned that from folklore into mathematics, twice. One participant proved the construction is within 0.023% of the best possible rule in its entire class--certified with computer-verified interval arithmetic, hashes published, the works. Another found that the standard construction everyone was using was actually missing a basis--it wasn’t quite the perfect object everyone assumed--fixed it, measured the fix, and found it worth less than 1%. Which forced the right explanation: the construction never worked because of the property everyone cited. It works because every pair of its coordinate frames is maximally spread out from every other--and that property, they proved, is exactly optimal at every depth. The missing basis was not the mechanism. As titles go, that one’s hard to beat.
The upshot: this part of the problem is closed. Anyone can copy the construction; four teams shipped it (18145, 18149, 18171, and Team Puffi’s 18175); the certificates say there’s nothing left to squeeze. Which makes what’s at the top of the leaderboard interesting--hold that thought.
Method 4: not all neurons are equal
Some neurons are always going to fire, some are always zero, and a handful sit at the edge. Figure out which is which cheaply, spend your budget on the uncertain ones. This produced the best raw accuracy for most of the competition (the original chassis), and in the final week it stopped being one team’s technique and became the field’s shared infrastructure--two separate teams explicitly built on it, one of them combining it with the sphere geometry to set a new best published raw accuracy.
The new measured warning: prune gently. One team swept the threshold for dropping rarely-firing neurons and found a cliff--units that fire once in a thousand times carry disproportionate weight downstream, and cutting slightly too deep costs double the error for a sliver of saved compute.
Method 5: Sideways Spheres
This is the finding of the batch, and it emerged independently in two write-ups (18183, 18175). Once you have the 66,048-direction machine, each of its 129 coordinate frames produces its own estimate, and the shipped method just averages them equally. The standard method assumes every coordinate frame is equally informative, so it simply averages them. But there’s no theorem saying equal weights are optimal.
So ask: what if you weighted them cleverly, per network? If you let an oracle peek at the true answer while choosing the weights, the error drops by a factor of five--and that’s not overfitting; it holds up under careful cross-validation. The capacity is real, sitting right there in numbers you’ve already computed.
Then try to find those weights using anything you’re actually allowed to know--the weights of the network, the formulas, the statistics of your own trajectories. One team measured the angle between the direction the lawful information points and the direction the correction needs to go: 89 degrees. Almost exactly sideways. The other write-up’s version of the same conclusion: every observable they tried captured about 1% of a gain they proved was 96% available.
I find this genuinely spooky.
The measurement I couldn’t resist
One write-up ended with a question: analytic formulas beat sampling on wide, shallow networks (a known result from ARC’s own research), and lose at this competition’s dimensions--so where’s the crossover, and is it about width or depth? I had the machinery lying around, so I ran it: a grid over both, posted as a reply in that same thread. Answer: both matter, width dominates, and the crossover depth climbs from about 24 layers at width 64 to beyond 48 layers at width 1024. I noticed that Phase 1’s chosen shape--256 wide, 32 deep--lands at almost exact parity between the two approaches (sampling vs analytic), a ratio of 1.04. Either the organizers placed their problem precisely on the fence between the two philosophies, or the fence found them.
Players gaming the system
A good chunk of the real competition still wasn’t about the network at all. It was about the meter. The final week’s additions: a participant found a genuine billing bug--one operation shape gets its additions for free, roughly doubling your effective budget--used it, then publicly disclosed it, flagged their own top-ten score as invalid, and reported it to the organizers before anyone else noticed.
The same week, another team’s audit of the standard operations found no arbitrage at all. Both are true: the standard routes are clean, and the one real exploit was past the edge of where the audit looked. There’s something quietly reassuring about a competition where the person who finds the money on the ground is the one who turns it in.
Whatever the public result on a submission says, every submission is actually graded on 100 MLPs, and 50 of them stay hidden from entrants. The real result is already sitting in a database nobody outside ARC has seen.
A fair number of entrants worked only the open 50, some clearly spending hundreds of submissions to “mine” the shape of those fifty known MLPs. An interesting optimization exercise. Not the problem ARC is looking for.
The system gaming the players
And a wrinkle was then discovered by an observant participant--that the fifty public test networks were statistically an unusually “easy” draw, a one-in-five-hundred coincidence. Methods tuned to those fifty networks can overfit their peculiarities without improving at the underlying estimation problem.
Will the hidden 50 MLPs be easy like the public ones are, or are they harder nets? If they are the harder nets that will that change the final scoring of Phase 1? Methods that are tuned to the ‘easy’ nets will give way to methods that score all nets better. Entrants won’t know until the results are posted, and I, for one, am looking forward to finding out.
I suspect based on my investigation above that ARC is much more interested in the algorithmic contributions for Phase 2 than in Phase 1’s raw leaderboard. They wouldn’t pick a Phase 1 size right at parity for the two methods if they weren’t going to reward both. My guess is that they will reward good, efficient, machinery in Phase 1 and reward the actual algorithms in Phase 2.
The ending changed
For most of the competition, the top of the public leaderboard was a mystery: scores a thousand times better than anything any published technique could explain, with the best forensic work ruling out essentially every legitimate explanation. The consensus among the write-ups was that whatever was winning was either undocumented genius or wasn’t going to survive contact with the private re-evaluation.
It didn’t survive contact with the rescore. In the final recompute of the public board, the two anomalous entries at the extreme front were removed. Not re-scored--removed. One of the authors reappears mid-board with an ordinary score. Every other entry on the board is essentially unchanged. The forensics were right: those scores weren’t a missing technique.
The community had quietly built something like an immune system. One team measured the benchmark’s noise floor, another audited the compute accounting, a third proved what accuracy was mathematically possible. Together they had already concluded those leaderboard entries couldn’t be genuine. The organizers eventually agreed.
Which leaves the new number one, a score that survived the cleanup, sitting about five times better than the certified-optimal geometry should allow for its cost--which is, almost exactly, the size of that oracle-gap wall from Method 5.
LLMs and this challenge
Nearly every write-up in the final wave carries some version of the same disclosure (LLM use was permitted, but disclosure was required by the competition): the experiments were largely run by LLM agents, directed and audited by a human. One author--of the 82-page report, with its custom fast-multiplication circuits and its careful ablations--mentions in closing that they never opened a code editor during the entire competition, and wonders aloud whether their role was “orchestrator” or “curious sidekick,” and guesses the answer is drifting toward the latter, and proposes the whole challenge would make a good standing benchmark for the next generation of models.
Phase 1 of this competition was, on its face, about predicting a frozen network without running it. Underneath, it quietly became a natural experiment run on what human-plus-LLM research teams can produce: a falsification-disciplined literature, with measured negative results, inline credit, adversarial audits, theorems with proof certificates--and a leaderboard whose top, for a while, was something none of that could explain, until it was.
Where do the humans fit in, then?
I’ve also been impressed with community involvement on the forum thread for the challenge. It was relatively quiet during the pre-phase, but there’s been a flurry of activity and publishing lately. Granted, some of this is no doubt incentivized by the $10,000 Phase 1 algorithmic-contribution prize (plus whatever discretionary community-contribution money is on the table), but people have also just been sharing findings and helping us--and AIcrowd, who runs the competition--debug the system and the scoring itself.
Early in the competition I was struggling, and keenanpepper spent real GPU time generating and publishing two datasets on Hugging Face--500,000 regenerable MLPs with per-layer mean targets, and 10,000 more with full moments to fourth order--that ended up powering my own work, and at least a couple of other entrants’ too. Twenty separate participants published their findings, including quite a number of the top-ranked entrants.
Phase 2 and beyond
I don’t know what Phase 2 will look like beyond the guess I make above. My hope is that the collaboration will continue. I am genuinely curious to see where this challenge takes the ARC research program and whether or not it produces something genuinely novel to the field.
This has been fun, and I’ve spent about as much time reading other papers and other entrants’ work as I’ve spent on my own submissions. Without the aid of LLMs helping me and virtually every other participant I doubt this challenge would be so vibrant. If this isn’t anthropotechnic mutualism, I don’t know what is.
James is a security engineer who entered a math competition to predict what a neural network would do without running it, and came in 133rd of 311--though he briefly held #1, on account of being the only one still awake. He publishes at waypoint.henrynet.ca.
Originally published at https://waypoint.henrynet.ca/2026-08-18-predicting-a-network-without-running-it/ · Signed by did:web:james.henrynet.ca · sha256:2e424439