Showing, not just saying
Point at an element on screen, and draw components instead of sentences — UI highlighting and generative UI.
An agent that can only talk is doing half the job. Two things let it use the screen the customer is already looking at.
Pointing at something
"Tap Continue" is useless if the customer cannot find Continue.
That is the whole integration. Registering the name tells the agent the element exists and can be pointed at; the boolean tells you when it is being pointed at.
The SDK does not draw the highlight. It does not own your design language, and a ring that looks right in one app looks broken in the next. You get a boolean and style it the way the rest of your app would — a border, a pulse, a scroll-into-view.
Highlights expire after six seconds. One still lit after the conversation moved on is worse than none: the customer keeps looking at the wrong thing.
Scoped to the component. Unmounting deregisters it, so the agent is never offered a target on a screen the customer has left.
The one dashboard step
Declare highlight_element on the agent, under Integrations & Tools,
alongside your other actions:
You do not implement it. The SDK answers it: it checks the name against what is actually registered on the current screen and refuses with the real list when the agent invents one — so a wrong guess produces a useful sentence rather than a dead action.
Turning one on
Four things have to be true before a component can appear, and one of them fails silently if your agent is older.
Saving is what pushes a component into an agent’s config — a live agent reads a snapshot of its tools, not your database. Components are deliberately never written to a phone agent, so an embed with no web-call variant is skipped and the dashboard tells you which.
Load the content from your API
Leave it blank and the component is static. Fill it in and it is called the moment the agent reaches for the component, and the JSON object it returns is merged over what you authored — so the layout stays yours and the items, prices, images or times are live.
| Called from | Our servers, not the customer's device — so a key in a header is not shipped to the app |
| Must be | https, reachable from the internet. Private addresses and the cloud metadata endpoint are refused |
| Timeout | 8 seconds, and only on a channel that can draw — a phone call never pays it |
| If it fails | The component still renders what you authored. A slightly emptier card beats an agent that went quiet |
Ask your API for what the customer said
A component was called the same fixed way whatever the customer said, so it could show a catalogue but never the right part of one. Write a placeholder into the URL, a query value or the body, and it becomes a parameter the model fills from the conversation:
"I want a new SUV under twenty lakh" then reaches your API as
max_price=2000000&type=SUV. The editor lists every parameter it found under
the URL, so you can see what the model will be asked for.
| Where they work | The URL, any query value, and the body at any depth |
| Where they do not | Headers, on purpose — a placeholder there would let the model choose an authorization value |
| Required? | Never. A customer who has not named a budget still sees the default list, rather than the model stalling or inventing one |
| Left blank | A query parameter with nothing to fill is dropped, not sent empty — ?max_price= asks something different from not asking |
| Safety | A value cannot re-point the request: it is encoded on substitution, and the finished URL is checked again before the call goes out |
Is it actually on the agent?
There are two copies of a component and only one is callable: the library in the dashboard, and the snapshot inside the agent's own config that the runtime reads. They used to look identical from outside. Embed & Widgets → On screen now reads the snapshot back and says one of three things:
- The agent can call
…— it is on the agent; mention it in the prompt. - Nothing — press Save selection to push. Until then the agent cannot show a component however it is asked, and will answer in words or reach for something else.
- No web-call variant — components are never written to a phone agent. Open the agent, save it once to create one, and come back.
A component is a tool the model calls, so it needs a tool name and a reason to call it. Refer to it in the prompt the way you would any other tool — by what it does, not with any special syntax. An agent that was never told it exists will answer in words and reach for whatever else it has.
Drawing components
Sometimes a card, a set of choices or a three-field form serves the customer better than another sentence. The agent decides that itself and sends one.
A product list, with pictures
Reading three cars and their prices out loud loses the customer by the second
one. product_list is the same three as rows they can see and tap, and it is
the kind most worth drawing in your app.
The contents are authored in Embed & Widgets, not in your app, and may be filled at call time from your own API — so a catalogue that changes weekly needs no release. The same component reaches a website embed and a native app: one definition, both surfaces.
A new product list starts already filled, with sample rows and sample pictures, so you can see the shape before you replace it.
Pictures
imageUrl takes three forms, and all three work on both surfaces:
| Form | Example | Notes |
|---|---|---|
| Absolute | https://cdn.yours.com/creta.jpg | What you will use in production |
| Root-relative | /samples/car-blue.png | Resolved against the agent's own origin before it reaches a device |
| Data URI | data:image/png;base64,… | Fine for something small; it travels with every send |
The middle one is the one that used to break. A path with no host is an
ordinary image on a web page and nothing at all on a phone — React Native's
Image renders an empty box and says nothing, which looks exactly like an
author who forgot to set a picture. The SDK resolves it against the agent's
origin on the way in, so what you author in the dashboard is what the app
draws.
Keep them small and roughly 4:3. A row draws the picture at about 64pt, so a 2000px photograph costs the customer their data and buys nothing.
The kind travels untranslated, so a component you add tomorrow reaches a browser and a phone without either being taught about it. Without the returning sentence the customer taps a card, sees a tick, and the agent carries on as though nothing happened.
One library, chosen per agent
Components belong to the workspace, not to an agent: a lead form is the same lead form whichever agent shows it, and a copy per agent is how five slightly different versions of "Share your details" end up in production.
Which of them an agent may SHOW is per agent, under Agent → Embed & Widgets → On screen. Every component you give an agent becomes a tool it can call, so a car catalogue on a loan agent is not just clutter — it is a tool a model will eventually reach for, and tokens on every turn until it does.
Pick nothing and the agent gets every enabled component, including ones added later. That is the default and it is right for a workspace with one agent. Tick even one and the agent sees only what is ticked. Untick them all and the agent draws nothing and answers in words, which is a real choice and not the same as having picked nothing at all.
The selection is applied in both places it has to be: what the SDK is told, and what the runtime registers as tools. Saving stores it; Sync to agent config is what hands it to the runtime.
The kinds the dashboard ships
lead_form, plan_comparison, info_card, calendar_slots, quick_actions,
product_list, blank. The SDK relays kind verbatim, so an agent may also
send one of your own — handle what you draw and fall back for the rest.
submit takes two things, and both matter
data | the structured answer your backend wants |
summary | a sentence for the conversation |
Send only the first and the customer taps a card, sees a tick, and the agent carries on as though nothing happened. The sentence is what moves the conversation.
Handle the kinds you know, fall back for the rest
widget.fallbackText is what the agent would have said instead. Render it for
any kind you do not draw — a component you cannot show should still move the
conversation forward, and a renderer that guesses produces something worse than
words.
Dismissing
dismiss() is local only. A customer closing a card has not answered the
question, so the agent is not told; it will ask again in words, which is right.
Both work on either transport. On rtc they travel the control channel
beside screen context and actions — for the same reason: they must arrive
whole and in order.

