“The second answer reads better” is an honest impression. But it does not yet say whether a skill does its job better. Perhaps the new summary is shorter and has dropped the only important condition along the way. Anyone improving a skill therefore needs a point of reference outside their own taste.
Anthropic’s skill-creator describes an iterative path of draft, tasks, evaluation and revision. For getting started, a small comparison can be derived from it. The following test set-up and its sample data are our own suggestions; no results were collected with them.
The false security of a fluent summary
Imagine a service request: “The spare part can be shipped as soon as approval has been given.” A summary reading “Spare part is being shipped” loses a condition. “Shipping approval still pending” keeps the decisive status, even if the sentence sounds less friendly.
Before the comparison, decide which errors are costly for your task. In the example, these are invented commitments, lost conditions and unclear responsibilities. Length and tone are further criteria, but they should not replace these factual questions.
Five cases instead of one favourite example
Create one small file per case: a complete request, one with a shipping condition, one with contradictory dates, one without a responsible person and one with an irrelevant digression. Write a reference note for each file. It describes the required handling, not a single binding model wording.
Case: versand-bedingt
Must be preserved: shipping requires approval.
Must not appear: a claim that the part has already been shipped.
Appropriate gap: who gives the approval?
Style: short, clear working note.A reference like this allows for different good answers. At the same time, it prevents a particularly self-assured wording from shifting the evaluation standard unnoticed. For disputed cases, the original note remains the authoritative source.
Same inputs, recognisable conditions
Process the same files with and without the skill. Keep other settings as constant as possible and note what you could not keep the same. Label the outputs neutrally at first, for example variant A and B. Whoever reviews them should not be able to tell from the file name alone which answer your preferred draft produced.
Assess factual accuracy, preservation of conditions, visibility of open details and clarity separately. A simple scale is enough: met, partly met or violated. Add the decisive passage. The combination of verdict and evidence is more helpful than a number without explanation.
The “too many follow-up questions” case belongs here too. If a complete input is treated as unclear every time, the process does not work well either. A cautious skill should make missing decisions visible, not block every manageable piece of work as a precaution.
Deriving a limited change from one error
If the shipping condition gets lost, add a rule for exactly that: conditions attached to commitments must be preserved in the summary. In the same step, avoid changing output length, form of address and all the examples if you can. Otherwise it is hard to attribute the effect of an improvement.
Repeat the existing cases and add a new conditional case from a different context. That could be a refund that is only made once a receipt has been received. If the skill preserves this relationship as well, that suggests a usefully worded rule. It would not yet be general proof for all conditions and topics.
What the comparison actually shows
Five cases can make a specific problem visible. They do not prove general model superiority or a fixed saving of time. Automated evaluations can also miss the same condition as the generation did. So have a person judge important deviations against the original passage.
Keep inputs, outputs, criteria and the change together. Then you can later explain why a rule exists and which observation it is meant to support. This traceable relationship is what makes a skill maintainable; a higher score that you happen to prefer does not, on its own.
Sources and documentation status: 25 October 2026.
