Speech decisions

A transcript keeps the caller's words and loses how they said them. "Fine, go ahead" can be calm agreement or barely held anger, and a caller can ask for a person in a way speech recognition doesn't catch. With speech decisions on, the agent also listens to the caller's voice on every turn and answers a few questions about it — How does the caller sound? Are they asking for a person? What are they doing? — from the audio itself.

What it hears can:

• tell the agent before it replies, so it can slow down for a frustrated caller or offer a person at once;
• move a flow agent to its next step as soon as the voice makes it clear, and take any path that waits for what was heard;
• start a lookup early, while the caller is still talking, so the answer is ready when the agent needs it;
• and is kept on the call, in Call History and the export.

It works for prompt agents and flow agents, on phone calls and in-app calls. It is off until you turn it on.

Turn it on

• Prompt agent: open the agent, choose Speech decisions in the list of settings, switch on Hear how the caller sounds, choose what to listen for and choose Save speech decisions.
• Flow agent: in the flow builder, open Settings › On the call. Speech decisions is at the end of the tab. It is saved with the builder's Save, with the rest of your changes.

When you switch it on with nothing chosen, five questions are filled in for you: Tone of voice, Asks for a person, What the caller is doing, Agrees or disagrees and Urgency. Add or remove any of them.

A live agent's change goes for approval like any other change, where your workspace reviews them. Speech decisions are set in the dashboard; the API doesn't include them yet.

What it listens for

Choose up to 12 questions. Each one is answered on every turn the caller speaks.

Ready-made questions

Speech decisions never ask about the caller's gender.

Your own questions

Choose Add your own question and fill in:

• Question — about the caller, in plain words, as you would ask a colleague who heard the call: Is the caller asking for a refund? Up to 300 characters.
• Answer type — Choose one (one answer from a list), Yes or no, or Scale (a list in order, from the lowest to the highest).
• Answers — for Choose one and Scale, 2 to 10 answers of up to 40 characters. Each can have a short description (up to 80 characters) that helps tell it apart from the others, such as pro — the pro plan.
• Short name — filled in from the question's words. It names the question in event names and in your call data: lower-case letters, numbers and _, starting with a letter, up to 32 characters. It can't be the name of a ready-made question (tone, wants_human, …).

How sure it must be

Only act when at least … sure sets how sure an answer must be before anything acts on it: from 50% to 99%, 80% to start with. A higher figure means fewer answers, and surer ones. Every answer is still recorded on the call with how sure it was.

Telling the agent

With Tell the agent what you hear on (the default), before each reply the agent is told what was heard in the caller's voice on that turn, with how sure it was — for example Tone of voice: frustrated (92%) — and what any matching rule says. Only answers at or above your threshold are passed on. The note is for that reply only: it is never added to the transcript and is never taken for the caller's words.

You can still tell the agent in its prompt how to use what it hears, such as If the caller sounds frustrated, apologise once and keep your answers short.

Rules

A rule acts when an answer is heard: When a question is an answer, then one or more of:

• Tell the agent — a line of up to 300 characters, added to the note for that reply: The caller wants a person. Offer to transfer them now.
• Also take the path waiting for (flow agents) — an event name of your own, sent into the call when the rule matches (below). Lower-case letters, numbers and . _ : -, up to 64 characters.
• Look up early — one of the agent's tools that only read data, started as soon as the rule matches (below).

A rule can have its own Only when at least … sure; left empty, it uses the setting above it. An agent can have up to 20 rules. A rule must name one of the questions being asked; if you remove the question, the rule is marked until you change or remove it.

Lookups while the caller is still talking

An agent that has to look something up — an order, a balance, an appointment — usually waits until the caller has finished, then asks for the lookup and waits for it. Start read-only lookups while the caller is still talking lets that start earlier:

Only lookups that read data are ever started early — API tools whose method is GET, never one that changes anything — and only when everything the lookup needs is already known on the call (variables sent with the call, or values collected on it). Nothing is made up to fill one in.

When the caller finishes, the guess is checked. If it is confirmed, the agent uses the result instead of waiting for the lookup. If it isn't, the result is thrown away and never reaches the agent.

Flow agents

Choosing the next step from the voice

Choose the next step from the caller's voice when at least … sure is on for flow agents, at 85% to start with. On a step whose paths are decided by the AI, the caller's voice is also asked where the conversation should go next. When it is at least that sure of one path, the call takes it at once, without waiting for the decision model to think. Otherwise the decision model decides, as it does today.

Rule, Event and Always paths work as before and are checked first.

Paths that wait for what was heard

With Use in flow paths on (the default), every answer the agent is sure of is sent into the call as an event named after the question's short name and the answer:

voice.wants_human.yes
voice.call_intent.callback
voice.tone.frustrated

The panel lists every name your questions can send, to copy, and the event names your rules add. To act on one, add an event path: on a step that talks (an AI reply or a Fixed line), add a path, under When to take it choose Event, and paste the name. When the agent hears that answer while the call is on that step, the call takes the path. Each event is sent once per turn.

Each answer is also kept on the call as a value named voice_ and the short name — voice_tone, voice_wants_human — so a Rule path can check it, and a line can say it as {voice_tone}.

Advanced

Folded under Advanced:

In Call History

Open a call to see How the caller sounded:

• across the top, the answer heard most often for each question, with how sure it was: Tone of voice: frustrated · 92%;
• then each turn that was heard, beside the caller's line from the transcript, with what was heard on it. On a flow agent, a turn also shows the step the call moved to — Step chosen from the voice or Step chosen by the agent — and a lookup started early says how long it took and whether it was used, not needed or discarded. Hover over any of them for more.

In the list of calls, a call shows its most telling answer beside its status. The export has a Caller's voice column with the same summary, such as Tone of voice: frustrated; Asks for a person: yes.

Good to know

• The call's transcript always comes from the agent's own speech recognition. Speech decisions add to it and never change it.
• Answers are about how the caller sounds on each turn. They are a signal for the agent, not a record of what the caller said.