Getting to know ClaudIA and the audit process
1) How ClaudIA works
ClaudIA is Cloud Humans' generative AI, designed to deliver a more humanized and agile customer service;
It uses Large Language Models (LLMs), trained on millions of texts from the internet, books, and articles;
Its use is personalized and restricted to the specific company it is serving;
It has the freedom to adapt content, as long as it respects the company's tone of voice and policies;
When it receives a request, ClaudIA analyzes the customer's question and consults its knowledge base, called IDS;
It performs a semantic analysis and ranks the best contents to use in the conversation.
****To check the content base the AI is using, you need to access the Cloud Humans Hub through this link here (if you
don't have access yet, just let us know in the project's main channel).
After logging in, click "Conteúdos" (Contents) in the side menu, as shown in the image below:
This will open the contents screen, where you can perform both an exact search for the information and a semantic search
— simulating the way the customer might phrase their question. The contents with the highest probability of a correct
answer will appear at the top.
To test a conversation from start to finish, you can use the Playground option, also available in the Hub through the
side menu.
2) ClaudIA's errors during conversations
It is important to highlight that no matter how complete the content available for the AI to handle conversations may
be, it will always have an error % associated with its performance. This happens for several reasons, including:
- Ambiguous user question: the customer may often phrase their question in a way that is not very clear, causing
ClaudIA to use an incorrect content or escalate due to lack of information;
- Outdated content: this covers cases of changes in the company's internal policy, links, or processes without
updating the content in the Hub;
- Missing content: new product launches, promotions, and even bugs or slowness must be added as content so ClaudIA has
access to the new information;
- Lack of real understanding: as humanized as ClaudIA may be, it works by identifying statistical patterns between
words and ideas, which can be subject to errors
Here at Cloud Humans we consider that ClaudIA can make 3 types of errors, which can be classified during the audit
process:
- Type 1 errors (minor): correct information, but some detail was missing that would have made the experience more
complete;
- Type 2 errors (moderate): partially correct information, ClaudIA omitted or included some content unnecessarily;
- Type 3 errors (severe): completely incorrect information, N2 cases that ClaudIA did not transfer, etc;
What we observe on average in projects with good AI management is that the overall error rate (adding up all 3) does not
exceed 20%, with at most 5% of type 3 errors. This can be a good benchmark for those who are starting the audit process
and need an initial target.
Regarding the number of tickets that need to be audited, it will depend on the volume of the project in question. Here
is a monthly suggestion, considering a 99% confidence level and a 5% error rate:
- From 20k to 40k tickets per month: 1000 monthly audits;
- From 10k to 20k tickets per month: 800 monthly audits;
- From 5k to 10k tickets per month: 600 monthly audits;
- Up to 5k tickets per month: 400 monthly audits;
3) Starting the audits
An essential part of ClaudIA's continuous improvement happens through audits of the answered tickets. This step is
fundamental to identify missing content in the base and to correct possible incorrect answers and/or transfers during
the conversation.
To do this, access our Hub through this link here (if you don't have access yet, just let us know in the project's main
channel).
Note: If you are at the stage of auditing false tickets, don't forget to remove the "Identificador: Cliente"
(Identifier: Customer) filter, shown in the image below:
The most recently closed tickets are at the top. We recommend always starting the audits with them, since they provide a
more up-to-date view of the quality of ClaudIA's service.
Filters can also be added to audit a specific sample (e.g., by tag); however, if you are just starting your ClaudIA
quality process, we recommend avoiding this segmentation and auditing tickets randomly, covering both N1 and N2 cases.
To start the audit, just click the icon in the "Ação" (Action) column, as illustrated in the image below:
When you open the ticket you will find some information, such as:
- Projeto (Project): the project name; if your company has more than one ClaudIA, it is indicated here;
- ID Helpdesk: the ticket ID in your Helpdesk, in case you want to check the conversation internally;
- ID Cloudchat: the ticket ID within the Hub, the same one present in the URL of the open ticket;
- Criado em (Created at): date and time when the conversation started;
- Resolução (Resolution): N2 covers all cases in which ClaudIA ended the conversation by transferring it to a human.
N1 covers all cases in which ClaudIA ended the conversation on its own.
- TAG: tag applied by ClaudIA at the end of the conversation.
- Número de interações (Number of interactions): sum of the messages sent by the customer and by ClaudIA
- Teste A/B (A/B Test): if a performance test is running, it will be indicated here.
Below this information you will find the interactions that took place during the conversation between the customer and
ClaudIA. To check the rationale used by the AI and the list of contents returned in its search, you can click the i icon
next to the messages:
This way it is possible to analyze what led ClaudIA to use a certain content and make the necessary adjustments if the
answer is incorrect or not applicable to the scenario in question:
To adjust ClaudIA's answer, you need to click the thumbsdown option and indicate the problem with the answer (this data
will be reflected in your metrics dashboard, also available in the Hub):
After that, some adjustment needs to be made so that in the next conversation ClaudIA responds as expected. Here are
examples of possible adjustments:
- Should have used another section to answer:
Case: The customer's question was clear and ClaudIA made a wrong content selection;
Fix: Change the title to make the answer more relevant; to do this, use the pencil next to the section. Then
indicate that this section is the correct one and click "Enviar" (Send) to save your adjustments, and scroll up to
complete the audit.
- The answer was incomplete:
Case: Claudia omitted part of the answer's content;
Fix: We can indicate it with [] in the Title so that it never omits information in that specific content.
- Should have clarified before answering:
Case: The customer's question was ambiguous and ClaudIA did not do a triage before providing the answer;
Fix: Add content asking which details you would like the customer to provide.
- Hallucination:
Case: ClaudIA went completely outside the content, made up information, etc;
Fix: Report it in the official Cloud Humans channel for support with the analysis.
- Should have escalated due to missing content:
Case: ClaudIA did not have the information and retained the customer;
Fix: In this situation, the most appropriate action is to add a new N2 content so that the transfer happens, and
faster, in the next conversation. To do this, click "Nenhuma das opções" (None of the options) to create new
content.
- Problem with the IDS content:
Case: ClaudIA chose the right content, but it was outdated;
Fix: In this situation, the most appropriate action is to add a new N2 content so that the transfer happens, and
faster, in the next conversation. To do this, click "Nenhuma das opções" (None of the options) to create new
content.
Extra: ClaudIA did not trigger the "Fluxo Controlado" (Controlled Flow): If a flow is published in the Fluxo
Controlado but for some reason that section is not being triggered, it is worth checking the section title and
adapting the keywords so the content gains more relevance and priority in ClaudIA's selection. If you want to know
more, we recommend this article about how ClaudIA and Fluxos Controlados work together.
Note: if you are not sure which adjustment to make, or if it is a severe hallucination case, ask for support through
the project's official channel so one of our specialists can help with the necessary changes.
4) ClaudIA's transfer reasons
There are some triggers that can make ClaudIA transfer a conversation. Knowing each of them and understanding which one
is hurting ticket retention the most helps in the strategy to increase the AI's retention.
The transfer reason will always appear in the internal note, at the end of the conversation, as shown in the example
below:
Here are ClaudIA's transfer reasons and their explanations:
- "Claudia usou conteúdo N2" (Claudia used N2 content): among the contents returned for Claudia to answer, she chose
to use an N2 content to answer the customer;
- "Claudia detectou ação de transferência" (Claudia detected a transfer action): the customer was transferred after
Claudia sent a message indicating that she would transfer the conversation to an agent/support;
- "Cliente Pediu Humano" (Customer Asked for a Human): the customer insisted on speaking with a human and Claudia
escalated
- "Claudia pediu detalhes e não entendeu a solicitação do cliente" (Claudia asked for details and did not understand
the customer's request): Claudia asked the customer clarifying questions until reaching the limit of those questions
— indicates a need for content adjustment
- "Claudia não sentiu confiança na resposta" (Claudia did not feel confident in the answer): Claudia answered without
confidence 2 or 3 times in a row — indicates a need for content adjustment
- "Claudia não resolveu o problema do cliente" (Claudia did not solve the customer's problem): after Claudia asked if
she could help with anything else, the customer still had questions about the same problem.
- "Interação superior ao limite de mensagens" (Interaction above the message limit): the conversation between the
customer and Claudia exceeded the maximum number of messages allowed
- "Repetição do mesmo Fluxo Controlado" (Repetition of the same Controlled Flow): to ensure the customer does not keep
entering the same Controlled Flow several times, they are escalated before the flow is triggered a second time
- "Fluxo Controlado Transferiu" (Controlled Flow Transferred): the customer was transferred within a Controlled Flow
- "Erro na chamada do Fluxo Controlado" (Error calling the Controlled Flow): there was some error in the Controlled
Flow during the conversation
- "Escalou por receber anexo" (Escalated after receiving an attachment): Claudia escalated the customer after they
sent an attachment (image, pdf)
- "Escalado de maneira forçada" (Escalated in a forced way): an agent took the ticket away from Claudia
- "Escalou fora do horário de atendimento humano" (Escalated outside human service hours): Claudia escalated because
there was no human service available (Human Office Hours)
- "Claudia ficou mais de 10 minutos sem responder (timeout)" (Claudia went more than 10 minutes without responding —
timeout): Claudia escalated after taking more than 10 minutes to respond to the customer's last message
