The appointment was changed correctly. Even so, the voice agent interrupts several times, only understands the name at the third attempt and chooses an inappropriate form of address. Was the task completed successfully? Factually, perhaps. As a conversation, it was considerably weaker. A single success rate can hide this difference.
Sierra introduced tau-i18n on 21 October 2026. The benchmark extends tau-Voice to Spanish, Brazilian Portuguese, Hindi, Korean and Mandarin and keeps the tasks and success criteria comparable. Task completion, interaction and the linguistic quality of the output are considered separately. This is current news with a transferable evaluation question, not a universal ranking of all voice agents.
A correct outcome only answers the first question
Our own example is an invented rebooking. A person wants to move an appointment from one day to another. At the outcome level, you can ask: was the right appointment changed for the right person? Were additional conditions of the request taken into account?
A friendly tone cannot make up for a wrong record. Conversely, a correct record does not automatically make the path to it pleasant. The two statements do not contradict each other. They look at different parts of the same use.
For an evaluation of your own, the expected outcome should therefore be fixed before the conversation. Otherwise it is easy to adjust the yardstick afterwards to whatever the agent has just done. A vague task produces an equally vague assessment of success.
Turn-taking is a usability question in its own right
People pause, correct themselves and add details. An agent can answer too early and thereby miss exactly the restriction that was still to come. That is not the same as a wrong translation. It concerns the timing of the conversation.
In the rebooking example, the person first states the desired day and then adds a time window. If the agent already responds with a finished decision after hearing the day, the conversation becomes unnecessarily laborious. Even a short sound of acknowledgement is not necessarily a new main instruction.
For a traceable observation, you therefore note the specific point: who was speaking, what information was still outstanding and why did the response interrupt the flow? “Sounded bumpy” is an impression. A specific premature change of speaker is a finding you can work on.
Linguistic quality is more than recognisable words
Form of address, expression and pronunciation have to suit the language and situation in question. With names and numbers, unclear output can make the task directly harder. A sentence that is translated in a formally correct way can at the same time sound unnatural or inappropriately polite.
This cannot be judged reliably from English back-translations alone. For specific results in other languages, you need people who can assess the expression in the actual language. The task should stay comparable across the language versions without reducing every linguistic convention to the same word order.
The exact language variant is particularly important. The benchmark names Brazilian Portuguese; this does not automatically say anything about European Portuguese. Each additional language or variant is a specific object of investigation, not an interchangeable counter.
An evaluation sheet of your own prevents mixing things up
The following template is a teaching schema of our own, not a tau-i18n output format. It deliberately contains no results yet:
Case ID:
Expected change:
Outcome: correct / incorrect / unclear
Interaction: specific observation + point in the conversation
Language: expression, form of address, pronunciation
Basis of assessment: language and the actual recordingApply the three levels to the same recording when you test for yourself later. Record the factual decision separately from the linguistic judgement. A language error can be important without already meaning that the task was completed incorrectly. Equally, a natural voice can mask a factual error.
The Sierra publication comes from the provider of the benchmark and looks at specific systems and tasks. We do not recreate any foreign-language audio, measurements or passed runs of our own here. The insight lies in the more precise question to ask about a conversation: did the right outcome come about, did the person remain meaningfully involved and was the linguistic output suitably understandable? Only this separate view shows which improvement is actually needed.
Sources and status
Research and documentation as of 25 October 2026. tau-i18n: Evaluation across languages and beyond task completion.
