I write a lot of agent skills for Claude Code. Every one of them looked good to me, because I knew what it was for. That is not evidence. So I built Skill Arena. It runs the same prompt twice on the same model, once with the skill and once without, and I have to vote before it tells me which was which. LMArena, for one variable.
The second reason was teaching. At Velais we run a four-day corporate AI training, and skills are the hardest thing in it to show. To someone who has never watched a model answer the same prompt twice, a skill is a text file that changes nothing they can see. Two answers side by side, one with and one without, is the demonstration.
How a run works
Both sides get the same model, always. Two models would test the model, not the skill. Skills come straight from ~/.claude/skills/, or you paste a SKILL.md.
The runs go through the Claude Agent SDK, pointed at your own Claude Code install, so they bill the login you already have. Each side gets its own folder and only its own settings, and the skill is copied into one of them. The two sides are then shuffled into A and B, with random folder names, so nothing about a path can tell you which one had the skill. Everything both sessions stream is written to a log, and every number the app shows later is recomputed from that log.
The first version got this wrong. The file that said which side was which lived inside the working folders, and an agent with Bash found it and opened its answer with "I'm the left arm, with the skill loaded". That file now lives where the sessions cannot reach it, and the app refuses to show it until you have voted. This is obscurity, not a sandbox. An agent with Bash can still wander the disk. There is a guard on the tools that say where they are writing, and a text check on Bash commands, and the code says in a comment that this is all it is.
The comparison that measured nothing
The first real result was fake. I ran a prompt through unslop, poteto's skill for cutting the AI tells out of writing, which I use on everything and am grateful for, and the side that had it never invoked it. Two identical answers. The skill was loaded. The model decided it did not need it.
That split the problem in two. A skill has a description, which decides whether it fires, and a body, which decides what it does. Those are different failures and need different tests. Since then, a run where the skill never fires is stopped and marked as such, and the app offers to test the description on its own instead, five short runs that report how often the skill fired. One detail for anyone doing this with the SDK: disallowedTools removes tools, allowedTools only pre-approves them. The probe needs the first.
One gap stays open. Even when the skill never fires, the two sides are not equal. The SDK puts every enabled skill's frontmatter into the context from turn one, and the file sits on disk where Bash can read it. A run where the skill did not fire tells you about the pull of its description, not its body. The reveal says so.
What it shows
After the vote: wall time, turns, reply length, output tokens, thinking tokens, cost and tool calls, for both sides, with the percentage change. An eye icon lets you peek before voting if you want to spoil yourself. History keeps a record per skill, wins, losses and ties, a tie counting as half a win, and the average cost difference.
| without | with | |
|---|---|---|
| wall time | 14.0 s | 11.7 s |
| turns | 1 | 3 |
| reply | 3,573 chars | 2,078 chars |
| output tokens | 1,622 | 941 |
| cost | $0.026 | $0.116 |
| tools | none | Skill ×1 |
There is no judge model. I thought about it and decided the point was to make me look. Fifteen votes in, the skills are 13 to 2. The two losses were progressive-complexity, which runs on every conversation I have, and app-mascot. I did not have those two numbers before.
The cat
Most developer tools look like the same grey rectangle, and I did not want to make another one. So the app has a mascot, a Scottish fold with a bit of Baymax in it, drawn from a photo I gave the model. It walks along the top rail of the prompt box, sits on the edge with its legs over the input, and while the two sessions are thinking it plays with a rock and a stick, or attempts a pirouette. It falls off the pirouette most times and lands it occasionally, which is the difference between a gag and a character.
The first version was drawn as poses. It lacked detail and could not turn. So I threw the movements away and asked for what an animator would make first: a turnaround sheet, the character from eight angles, and then the extreme pose of every move. That changed the architecture. The cat is now a rig. Every part is a point in its own 3D space, yaw is a number, and one projection, x·cos(yaw) + z·sin(yaw), produces the whole turnaround. Eyes converge, the far one slides off at profile, limbs swap draw order. It is plain SVG, one animation loop, no React renders per frame. Roughly a quarter of the app's lines are the cat. The model sheet is still in there, behind ⌘M.
The process was the useful part, so I had it written up as a skill, app-mascot, with its own list of failures from that day: legs visible from the side, an inverted Y axis on cursor tracking, a smile too curved, a drop shadow that looks pasted on. Seven minutes after that skill existed it was in Skill Arena, tested against the same prompt without it: "create a chick as an app mascot, keep it simple". Both outputs, as they came back:
Try it
It is on Glaze as Skill Arena. It runs on your own Claude Code login and writes nothing outside its own folder. Try it and let me know what you think.