Home Improving ClaudIA Even More How to Improve Claudia's Quality

How to Improve Claudia's Quality

Last updated on Aug 12, 2025

In this tutorial, you will learn how to analyze the quality of Claudinha's responses (CSAT), detailing available metrics on the dashboard, the impact of tags and sections, and ticket auditing. It also addresses common errors in Claudia, their causes and solutions, helping to optimize responses and improve the customer experience.

Below, we have two videos and subsequently a textual explanation to improve Claudia's quality:

Best practices for auditing, CSAT, and Analysis of audits:

Best Practices in Auditing

1. What should I analyze when auditing a ticket?

  • Check if the AI's response is correct and complete.

  • If the response is correct, record a positive vote.

  • If there is an error, identify the cause:

    • Did the AI use the wrong section?

    • Was the response incomplete?

    • Was a qualification step needed before responding?

  • To audit the messages, follow this explanation of the options:

    • 1. Should have used another section to respond
      The AI used an incorrect information source and should have consulted a more appropriate or relevant section of the available content to respond correctly to the customer.

      2. The response was incomplete
      The AI partially answered the request, failing to address important points or those requested by the customer in the original message.

      3. Needed to clarify before responding
      The AI misinterpreted or guessed the customer's intention without first seeking clarification, which compromised the accuracy of the response.

      4. Should not have clarified
      The AI asked an unnecessary clarification question when there was already sufficient information to formulate a useful response.

      5. Hallucination
      The AI included information that was not present in any of the available sources, inventing data or making unsupported claims.

      6. Should have clarified differently
      The AI tried to clarify the customer's doubt, but the way it asked the question was confusing, generic, or poorly directed, making it difficult to continue the conversation.

      7. Should have escalated due to lack of content
      The AI did not have enough basis in the current content to respond correctly, and therefore should have forwarded the service to another flow or to a human.

      8. Problem with IDS content
      The content used by the AI (IDS information) was incorrect, outdated, or poorly structured, harming the quality of the response.

      9. Other
      Issues outside of common patterns, such as technical errors, inappropriate language, confusion in service logic, or unexpected AI behavior.

2. How to identify an incomplete response?

  • Compare the given response with the available content.

  • If the correct section was used, but the response did not address all necessary information, mark the error as incomplete response.

  • Use the prompt led section to avoid this problem in the future.

3. What to do if the AI does not have the necessary content?

  • Check if there is a relevant section for the doubt.

  • If not, create new content to cover this case.

  • Record the error as "missing content" in the error detail.

4. How to audit numerical responses or calculations?

  • Confirm if the AI used the correct section to perform the calculation.

  • If the calculation is correct, mark the response as valid.

  • If the AI could have used another more appropriate section, adjust the audit.

5. How to know if the AI should have transferred a service to N2?

  • Check if the AI had all the necessary information to resolve the issue within N1 level.

  • If the response was satisfactory and complete within N1, there is no need for transfer.

  • Otherwise, the audit should indicate that the AI should have forwarded it to N2.

6. How to handle cases of service abandonment?

  • If the service flow worked correctly and the audit did not point out errors, record the case as correct flow.

  • If there are improvement points, note them for future adjustments.

7. What to do if the AI has no information on a topic?

  • If the customer asks about something not in the knowledge base, record the error as "missing content" in the error detail.

  • If relevant content exists but was not used, adjust the section title to ensure better information retrieval.

8. How to deal with generic or uninformative responses?

  • If the AI answered something vague or did not fully clarify the customer's doubt, ideally, it should ask for more information before responding.

  • If necessary, adjust the content to include more specific questions and improve the contextualization of the response.

9. When should I modify a content title?

  • If the AI did not find a relevant section, but a similar content exists, it may be necessary to adjust the title to facilitate retrieval.

Follow the best practices for content creation and maintenance available at this link.

Quality Analysis of Claudinha (CSAT)

1. What is the quality analysis of Claudinha?

The quality analysis aims to evaluate the performance of Claudinha's responses, identifying improvement opportunities based on the adjustments we can make and the improvements you can also implement.

2. How to view CSAT data on the dashboard?

On Claudinha's dashboard, you can access:

  • CSAT overview: a broad analysis of the quality of responses.

  • Comparison between humans and Claudinha: comparison between responses given by Claudinha and tickets that were escalated for human assistance.

  • CSAT by tag: Claudinha's performance on different topics.

  • Historical data for the last five weeks: allows tracking the evolution of response quality over time.

  • Table view: aggregated display by tag, showing the total number of responses and the respective CSAT.

4. How to analyze the impact of tags and Eddie on CSAT?

There is a specific visualization on the dashboard that allows understanding if a particular EDDIE or Tag is negatively impacting the CSAT. This helps identify if a specific content is harming the quality of responses.

5. How to understand the impact of the sections used on CSAT?

We can cross-reference the sections used in conversations with customers against the CSAT results. This allows us to check if a particular section is associated with a negative CSAT, helping identify which contents are negatively affecting the customer experience.

6. How to visualize overall CSATs?

CSATs are categorized into:

  • Positive ("goods" 4 and 5)

  • Negative and neutral (1, 2, and 3)

Tickets from each category can be analyzed to understand if Claudinha performed below expectations or if the customer was dissatisfied with the response.

7. What to do after analyzing CSAT?

After this analysis, we can audit the tickets to identify the main reasons for Claudinha's errors and act on them. This audit will be addressed in the next step of Claudinha's continuous improvement.

Auditing and Quality of Claudia

1. What can we learn from auditing tickets?

Auditing CSAT tickets and other tickets allows us to identify errors and areas for improvement in Claudia. With this analysis, we can reduce failures and optimize the responses provided by the AI.

2. How to visualize Claudia's overall quality?

On the general quality tab of the dashboard, it is possible to see:

  • The total percentage of errors from Claudia

  • Classification of errors (Error 1, 2, or 3)

  • Analysis of correct and incorrect tags

  • Topics and tags that most negatively impact

The provided links allow access to error details by selected tag.

3. How to identify contents that generate the most errors?

The "error percentage by content" section of the dashboard allows visualizing which contents are more prone to generating errors in Claudia. This information is useful for prioritizing corrections.

4. What does the "breakdown by message sent" mean?

This section details the errors made by Claudia in generating messages. For each response, Claudia receives a set of 15 to 20 contents (TopK) and, based on them, constructs the response.

Errors can occur in different ways:

  • Correct section outside TopK: the ideal response was not in the received set.

  • Wrong choice: Claudia received the correct section but incorrectly chose another.

  • Incomplete response: Claudia used the right response but omitted critical parts.

5. How to analyze errors by section used?

The table presents:

  • Analyzed ticket

  • Section used in the response

  • Correct section that should have been used

  • Description of errors made

6. What can errors indicate?

  • Confusion between similar sections: Sections with similar titles may hinder Claudia's correct choice.

    • Solution: Review the titles of the sections following best practices for content maintenance and creation available at this link or use the Prompt Led Section functionality available at this link to better specify the use of each section.

  • Incomplete responses: When Claudia omits critical information.

    • Solution: Adjust the sections or use Prompt Led Section to reinforce the importance of complete content.

  • Sections do not appear in TopK: The correct content is not among the 15 to 20 most relevant.

    • Solution: Adjust titles so they appear better in certain customer searches.

7. How do these insights help in improving Claudia?

With the detailed analysis of errors and utilized contents, it is possible to:

  • Improve the accuracy of responses

  • Reduce classification errors

  • Ensure more complete responses

  • Adjust titles to improve indexing and relevance

  • Use the Prompt Led Section functionality to enhance the choice of correct sections

This way, we can optimize Claudia's performance and enhance the customer experience.

Access the mind map of best practices here!