Measure what you can. Judge what you must.

The first version of my design eval tool was too eager. I pointed it at a prototype, it came back with forty findings, and within a minute I'd stopped reading. Some were real. Some were the model inventing standards I hadn't asked for. Some were confident-sounding guesses about contrast ratios it couldn't actually see.

The feedback I got, and gave myself, was simple: it's too sensitive. It should check the rules I switched on, and nothing else.

Fixing that taught me the most useful idea in this whole project. An eval for design isn't one thing. It's three different jobs, and each needs a different tool.

Job one: measurement

A lot of design standards are numbers. Text contrast of at least 4.5 to 1. Tap targets at least 24 by 24 pixels. Body text at least 16 pixels. No more than two typefaces. These are facts about the page, and the browser already knows all of them.

Asking a language model to judge these is a category error. It's slow, it costs money, and it's wrong some of the time, because the model is estimating from a description or a screenshot what the browser can compute exactly.

So in Glitterball, any rule with a number in it is measured in code. Glitterball walks every text element and every interactive target on the page, computes the real values, and decides pass or fail with no model involved. A measured failure always has a confidence of 1.00, and it comes with the exact elements that failed and by how much.

The same goes for patterns. Gradient-filled text, a backdrop blur, a link that goes to #, a banned word in the copy: all of these can be detected directly. When I wrote the anti-slop packs, 23 of 28 rules turned out to be measurable. A whole page is checked in about a tenth of a second.

The lesson: if a rule can be measured, measure it. Don't ask anyone's opinion, including a model's.

Job two: judgement

Some rules can't be measured. The headline says what the product does and for whom. There's one clear primary action. The copy reads as one human voice. These need judgement.

For these I use Jev, a model on OpenRouter that's built for decisions rather than conversation. You give it a specific question and a set of possible answers, and it returns its choice along with how likely each answer is. It doesn't write paragraphs. It doesn't wander off topic. It answers the question it was asked.

That shape matters for an eval. Each judged rule becomes one question: does this page pass, fail, or is the rule not applicable here? The answer comes back with a probability attached, which gives me the thing my first version was missing.

The judge is allowed to be unsure

In the current version, a judged rule only counts as a failure when the judge puts the probability of failure at 60% or higher. Below that, the rule is marked uncertain. It shows up in the results so you can look at it, but it doesn't go into the list of fixes sent to your coding agent.

This one threshold changed the tool more than anything else. Before, every hunch became a finding, and the findings that mattered were buried. After, the list got shorter and every item on it was worth doing.

I think there's a general point here about AI in design tools. Models are very good at producing confident-sounding output. A tool that passes all of it through to the user is outsourcing the hardest part, deciding what's worth attention, back to the person it was meant to help. Calibration, and a willingness to show not sure, is what makes a judge trustworthy.

Job three: writing the fix

The last job is turning a failure into something a builder can act on. Fails WCAG 1.4.3 is a verdict. Change the colour of .hero p from #9a9a9a to #6b6b6b to reach 4.6 to 1 is a fix.

Jev doesn't write prose, so a second model does this part. It only sees the rules that actually failed, and its only job is to describe the fix: which element, what to change, and what to leave alone. It isn't allowed to add new findings. That constraint is deliberate. The writer is there to be useful, not to have opinions.

The fixes go to your coding agent over MCP, or you copy a prompt for the elements you picked. Either way, the instruction names each element by selector and text, so the agent can find it without guessing.

Only the rules you switched on

The other change from my first version is less technical and more important. The tool now evaluates only the rules that are active. It doesn't infer extra standards. It doesn't add ideas unless you ask for them.

That sounds obvious, but it runs against how most AI features are built. The instinct is to be helpful, which in practice means saying more. In an eval, saying more is the opposite of helpful. The value of an eval is that it's a contract: these are the rules, this is how the page did against them. Anything outside the contract is noise, however well-meaning.

The shape I'd recommend

If you're building anything that evaluates design with AI, internal or commercial, this is the structure I'd start from:

1. Measure in code every rule that has a number or a detectable pattern. Report it as a fact.
2. Judge with a decision model only the rules that need taste. Ask one specific question per rule and keep the probability.
3. Threshold the judgement. Mark low-confidence verdicts as uncertain instead of failing them.
4. Write fixes separately, only for real failures, with no licence to add findings.
5. Evaluate only what's switched on. Let people add rules, but never invent them on their behalf.

Each layer does what it's best at. The result is a tool that flags less and gets taken seriously, which, for anything that wants to change how a team works, is the only result that matters.

Next, I want to look at the other side of the loop: what happens when the agent reads the rules before it builds.