When to use
- A new Specialist Agent is going to go live gradually (for example, "activated at 30%")
- You want to compare the outcome of conversations with and without a specific Specialist Agent
- You want to audit the performance of a Specialist Agent
- A conversation Replay did not work and you want to understand why
About this article
A newly published Specialist Agent usually does not go live for 100% of conversations all at once — it is activated gradually, like an A/B test, so that you can compare the outcome before expanding to all conversations.
How it works
How the activation percentage works
When a specialist is under gradual activation (e.g., 30%), each new conversation is drawn, a single time, at the moment it starts, to fall into the group with the specialist active or into the control group (without the specialist). This draw applies to the entire conversation, from start to finish.
:::info The draw is per conversation, not per customer. This means the same customer may fall into the group with the specialist in one conversation and into the control group in a different conversation — there is no rule that always places the same person in the same group. :::
How to compare the group with the specialist and the control group
The comparison is done by looking at the general A/B test metrics for the project (for example, retention) separately for the group with the specialist active and for the control group, over the same period. This shows whether the specialist is actually improving the outcome before increasing the activation percentage.
Auditing a specialist's performance
Today the audit is done conversation by conversation: the audit/tracing screen of each conversation shows whether the Supervisor triggered a Specialist Agent in that conversation and the reasoning behind the decision. To evaluate a specialist, the recommended path is to review a sample of the conversations in which it was triggered during the period — there is no ready-made aggregate report today that filters by a single specialist.
Why Replay sometimes does not work
Replay lets you reprocess the response of an old conversation with a new configuration, in order to compare the outcome. This feature is not available when the original response involved a Specialist Agent (that is, when the Supervisor delegated the conversation to a specialist in that message). Replay today knows how to reproduce the response of a single agent — it does not yet reproduce the Supervisor's delegation flow to a specialist.
:::warning If Replay fails for this reason, it is not an error: it is a known limitation for conversations handled by a Specialist Agent. The conversation remains available normally in the audit/tracing — only Replay itself is unavailable for this case. :::
Notes
- For the step-by-step on creating, linking, and publishing a Specialist Agent, see "How to create, edit, and publish a Specialist Agent".
- To understand how the Supervisor decides to trigger a specialist, see "What Specialist Agents are and how the Supervisor decides who responds".