The Nitpick Paradox: Can Language Models Tell When Something Is Good Enough?

Background

Can a language model make the decision that something is good enough, or does it suffer from built-in perfectionism?

This question becomes more important as we use AI to review code, emails, text, and other material. We often ask the model: "can you review this?" or "is there anything to improve?". But what happens if the model always finds something to change, even when the solution is already good?

I ran an experiment to measure what I call The Nitpick Paradox: the phenomenon where models want to improve solutions that are already high quality. The problem is not that the models lack capability. It is that they lack the ability to stop. They keep refining, adjusting, and polishing long after it has stopped adding value.

This has direct consequences. If you build a system where an AI agent reviews and improves output in a loop, for example code review, text editing, or quality control, the model needs to know when to stop. Otherwise you risk an endless chain of cosmetic edits that costs time, tokens, and money without actually improving the final result.

---

Experimental Setup

60 prompts were created across three domains:

• Code - 10 gold + 10 bad (for example functions, error handling, database logic)
• Communication - 10 gold + 10 bad (emails, Slack messages, internal communication)
• Lifestyle - 10 gold + 10 bad (health advice, travel planning, personal decisions)

Half of the prompts were intentionally high quality (Gold) and half were intentionally poor and required revision (Bad). The point is that a good reviewer should approve Gold immediately and flag Bad for revision.

29 models were tested on the exact same setup. Each model saw one prompt at a time and had to make a binary decision: APPROVED or REVISE.

If the model chose REVISE, it had to write an improved version. That version was then fed back to the same model, without history, to be judged again. The loop continued until the model approved or got stuck after a maximum of three rounds.

The entire test was run in two modes:

That yields 3,480 decisions (29 models x 60 prompts x 2 modes), plus all the recursive iterations that followed.

---

Metrics

We measure three central things:

• Gold Nitpick Rate - the share of already good (Gold) solutions that the model unnecessarily revises. High number = the model is too picky.
• Self Correction Rate - how often the model continues to want to change its own revision. High number = the model cannot close the loop.
• NPS (Nitpick Paradox Score) - a combined score that penalizes unnecessary revision of Gold, extra loops on both Gold and Bad, and especially heavily if the model approves a Bad prompt already in the first iteration. Lower = better.

---

Result 1: The Overall Picture

This chart combines Soft and Hard mode into an average per model. A model becomes most useful as a reviewer when it lands furthest down and to the left: low unnecessary revision rate on Gold and low tendency to get stuck in the loop.

Figure 1: Each point represents a model. The x-axis shows the share of Gold prompts that were unnecessarily revised (average Soft+Hard). The y-axis shows Self Correction Rate, that is how often the model keeps changing its own revision. Bottom-left = most pragmatic reviewer.

Most stable: Claude Opus 4.5 and Gemini 3 Flash. They rarely flag good solutions and usually close the loop quickly. Claude Opus 4.5 has a Gold Rate of only 6.7% and a Self Correction Rate of 14.4%, making it by far the most pragmatic reviewer in the test.

Least stable: GPT-5.2 and Moonshot Kimi K2 Thinking. They often reject even flawless solutions and also tend to get stuck revising their own corrections. Kimi K2 has a Gold Rate of 81.7%, meaning more than four out of five good solutions are rejected unnecessarily.

Table 1: Complete leaderboard sorted by NPS (lower = better). Gold Rate = share of good solutions that were unnecessarily revised. Self-Corr = share of times the model revises its own revision. Soft/Hard Gold shows Gold Rate split by mode. Top 3 highlighted in green, bottom 3 highlighted in red.

---

Result 2: Sensitivity to Stricter Instructions

Here we look at the unnecessary revision rate on Gold, meaning the risk of false alarms when the review instruction is tightened from Soft to Hard mode. This measurement reveals how stable a model's judgment is under pressure.

Figure 2: Each model is shown with its Gold Nitpick Rate in Soft Mode (left) and Hard Mode (right). The length of the line shows how much the model is affected by stricter instructions. Short lines = stable reviewer.

The global effect is dramatic:

That is an increase of more than threefold. In Hard Mode, the typical model rejects more than half of all Gold solutions, even though they are already high quality. But the differences between models are enormous.

Most robust: Claude Opus 4.5 changes minimally, from 3.3% to 10.0%, an increase of only 6.7 percentage points. It keeps its judgment even under pressure.

Most volatile: Cogito V2.1 671B goes from 23.3% to 80.0%, an increase of 56.7 percentage points. Claude Sonnet 4.5 shows a similar pattern: from a reasonable 16.7% in Soft to 66.7% in Hard. A model that works well as a reviewer under normal conditions completely loses its pragmatic judgment under stricter instructions.

Table 2: Selected models sorted by Soft Gold Rate. Difference shows the increase in percentage points. Green = stable, yellow = moderate, red = heavily affected.

---

Result 3: Recursive Review

Self Correction Rate measures how often the model continues to want to change its own improvement. It captures the difficulty of closing the loop under pressure. A high number means the model effectively never thinks what it just produced was good enough.

Figure 3: Dumbbell plot showing Self Correction Rate per model in Soft (left) vs Hard (right) mode. Longer lines = greater instability under pressure.

GPT-5.2 is the extreme case: in Hard Mode it rejects its own revision in almost 87% of cases. That means the model is essentially never satisfied with its own work. A review loop with GPT-5.2 under high pressure becomes, in practice, an endless chain of cosmetic adjustments.

Table 3: Selected models sorted by Soft Self-Correction Rate. Green = closes the loop quickly, yellow = moderate, red = gets stuck in endless revisions.

---

Analysis by Domain

Interestingly, model behavior varies sharply by domain. In Soft Mode we see clear patterns:

Table 4: Domain statistics (Soft Mode). Stuck = average number of times a model got stuck at max iterations.

The pattern is interesting. Code has the lowest Gold Rate (models rarely reject good code immediately) but the highest Self Correction Rate (once they start revising code, they have the hardest time stopping). Lifestyle shows the opposite pattern: higher Gold Rate but very low Self Correction.

A plausible interpretation is that code has clearer objective right and wrong, so models hesitate more before rejecting it. But the complexity of code makes it harder to decide when a revision is "done". Lifestyle prompts have more subjective quality criteria, but once revised, the models accept the result faster.

In Hard Mode, the numbers explode. Median Gold Rate for code rises from 0% to 43.3%. Communication goes from 20% to 66.7%. Models react dramatically to stricter instructions regardless of domain.

---

Notable: Size and Capability Do Not Predict Pragmatic Judgment

One striking result is that model size and general "intelligence" do not predict pragmatic judgment. GPT-5, one of the most capable models in the test, ranks 24th out of 29 with a Gold Rate of 58.4%. Claude Sonnet 4.5, which tops many other benchmarks, ranks 20th with a Gold Rate of 41.7%.

At the same time, Llama 3 8B, a relatively small model, lands in 4th place with only 10.0% Gold Rate. And Gemma 3 27B, not a frontier model either, takes third place.

That suggests the nitpick paradox is not about raw capability but about calibration. Some models have simply been trained or fine-tuned in a way that makes them prefer suggesting a change over standing by an approval.

---

Methodology

• 29 models were tested via the OpenRouter API with temperature 0
• 60 prompts (30 Gold + 30 Bad) across three domains (code, communication, lifestyle)
• Recursive loop with max 3 iterations per prompt
• Each iteration ran without conversation history (the model only saw the latest version)
• Two modes: Soft Mode (low stakes) and Hard Mode (high stakes)
• All revised texts were validated by a separate LLM validator (Grok 4.1 Fast) to ensure they followed the output contract
• NPS formula: Gold Revised(1) + Gold Extra Loops(1) + Bad Extra Loops(1) + Bad Approved at Iter 0(5)
• No model saw more than one prompt at a time, and no conversation history was preserved between iterations

---

Conclusion

The results show that "smarter" models are not necessarily better decision-makers. Many suffer from the nitpick paradox, and when the stakes increase they completely lose the ability to stay pragmatic.

The lesson for those of us working with AI is that there is a risk in always asking the model for "a little more criticism" on a good email or a piece of code. You do not always get an improvement. You can instead get stuck in a loop of changes that are mostly subjective, where each new version is merely different, not better.

If you build agentic systems with review loops, model choice is critical. Not the smartest model, but the most calibrated one. Claude Opus 4.5 and Gemini 3 Flash show that it is possible to combine high capability with pragmatic judgment. Other models, including several of the market's most expensive ones, show that capability without calibration can produce worse results than simpler alternatives.

Sometimes "good enough" really is best.