# How to fine-tune your AI product advisor

- URL: https://www.chaterimo.com/en/blog/how-to-fine-tune-ai-product-advisor/
- Language: en
- Type: Guide
- Author: David Langr
- Published: 2023-12-27
- Updated: 2026-10-20

> Improve your AI chatbot's answers after setup: knowledge base, instructions, AI model, testing, answer analysis, sales rules and measuring progress.

## Key findings

- Fine-tuning an AI product advisor in Chaterimo means improving its knowledge base, its instructions and its settings; the language model itself is not retrained.
- Start with the knowledge base: an answer can only be as correct as the products, pages and texts the chatbot finds, so add what is missing and delete what is outdated.
- **Analyze why AI answered like this** under an answer in Test Chatbot or Chat History names the likely cause, such as a data gap, an instruction problem or a model limitation, and suggests a fix.
- Test every change on the same list of real questions in Test Chatbot, and check the sources each answer was built from.
- Measure progress with the share of answers customers flag, the clicks on product cards and links, and the tickets handed over to your team.

To fine-tune your AI product advisor in Chaterimo, you find the answers that need work, find out why the chatbot gave them, and then fix the knowledge base, the instructions or the model, in that order. Fine-tuning here does not mean retraining the language model: you change your own data, instructions and settings, and Chaterimo trains your knowledge base again. This guide is for owners and managers of online stores whose AI chatbot is already live; if yours is not, start with [your first AI chatbot in 4 steps](https://www.chaterimo.com/en/blog/create-chatgpt-chatbot-chaterimo-4-steps/). Repeat the loop whenever real conversations show a new problem and after every larger change in your store.

## Steps

### 1. Collect the answers that need work

Real questions from your customers show what to improve. Open **Chatbot** → **Chat History**: it lists the conversations from your store, newest first. Search with **Search messages with any keyword** for the topics that matter to you, for example a product line, “delivery” or “return”.

Let customers tell you which answers did not help. In **Chatbot** → **Chat Settings & Embed**, tick **Enable message flagging** in the appearance settings and save. A flag button then appears next to each AI answer in the chat window; translate its **Flag button tooltip** into the language of your chat. In Chat History, **Show only flagged sessions** lists the conversations with an answer marked **Flagged by user**. A conversation the customer reported is marked **Reported**.

Two more sources:

- **Tickets**: the conversations the chatbot handed over to your team show what it could not finish.
- **Your own team**: the questions customers still send by e-mail or ask by phone.

Chat History keeps a conversation for 30 days after its last message. Select **Export to CSV** if you want to keep a record. From all of this, write down 20–30 real questions together with the answer you expect. This list is your test set for step 6.

### 2. Find out why the chatbot answered that way

Under each answer of the chatbot in Chat History and in Test Chatbot, select **Analyze why AI answered like this**. The analysis reads the conversation, the chatbot settings and the knowledge-base content the chatbot used, and it searches your knowledge base and product catalogue again. It only reads: it changes nothing in your data or settings.

The result names the main cause with a confidence (**High**, **Medium** or **Low**), lists the **Evidence**, and gives a **Recommended fix** with **Who should handle this**: **Your team** or **Chaterimo support**.

| Cause in the analysis | What it means | Where to fix it |
|---|---|---|
| Data gap | The knowledge base does not contain the answer. | Add the information (step 3). |
| Instruction problem | The chatbot role led the chatbot to this answer. | Change the chatbot role (step 4). |
| Configuration problem | The chatbot uses the wrong data, for example the product feed of another store or another language. | Check which feeds and integrations the chat widget uses (step 3). |
| Model limitation | The right information was available, but the model ignored or misused it. | Try another model or search flow (step 5). |
| Platform issue | Something failed on Chaterimo's side, for example training or search. | Select **Report to Chaterimo team**. |
| Working as intended | The answer was correct or reasonable from your data. | Nothing; **Tips to improve (optional)** may suggest extra content. |

The analysis is automated and can be wrong, and the app says so under every result. Use it as a starting point and check the evidence.

The analysis runs on your own API key, or on Chaterimo's keys if your plan uses Chaterimo API keys. The number of analyses per hour is limited. If you analyse the same answer again shortly afterwards, the app shows the earlier result (**Loaded previous analysis**).

### 3. Fix the knowledge base

The knowledge base is everything the chatbot searches before it answers: products, web pages, files and texts. Fix it first, because neither instructions nor a better model can correct an answer that is missing from your data.

**Add what customers ask about.** Typical gaps are shipping and payment, returns and complaints, warranty, contact details and opening hours, size charts, care instructions and compatibility. Open **Knowledge Base** → **Get Data** and add each topic as its own entry: paste text in **Text Input** (each save is a separate entry), import a page in **Website Scraping** → **Scrape a single link**, or upload a document in **File Upload**. Give each entry a clear name, so you can find and update it later. Write tables as one line per row, with the unit or currency in each line.

**Remove what misleads.** Old promotions, old shipping prices, duplicate pages and pages without useful text (cart, login, search results) make answers worse. Open **Knowledge Base** → **Knowledge Base**, filter by type (**Text data**, **XML feeds**, **Files**, **Scraped URLs**, **Products**), open the entry and select **Delete**. For a sitemap that is imported every day, open **Filters** and list the paths under **Do not import these pages from the sitemap.**, for example `/cart/*`. Product pages (`/products/*`) are excluded by default when your products come from a store integration or a product feed. When you scrape pages, enter the classes of menus and footers under **Advanced** → **HTML classes to ignore**, so their text does not end up in every entry.

**Improve your product data.** The chatbot recommends products by what your product data says. From a product feed (XML), Chaterimo reads the title, description, brand, price, product type, availability, image link, product details (`g:product_detail`) and recommended accessories (`recommended_accessories`). Put every parameter customers ask about, such as size, material, power or compatibility, into the description or the product details; a parameter that appears only in a product image is invisible to the chatbot. Then:

- Keep prices and availability current: turn on **Download this feed again every day and retrain** for the feed. Store integrations (Shopify, Shoptet, Upgates, Eshop-rychle) update your products by themselves, so improve the texts in your store administration.
- Set **Select Language of your XML feed** to the language of the feed.
- If you run several language versions or stores, open each feed in the Knowledge Base and choose under **Which chat widgets can use this feed?** which chat widget may use it. A feed attached to the wrong chat widget is what the analysis calls a configuration problem.
- Add products that are not in any feed in **Get Data** → **Individual Products**.

**Explain your order statuses.** If customers ask about their orders, open **Get Data** → **Order Details** and explain what each order status means and how long shipping usually takes. The chatbot uses this text only when a customer asks about an order by its number. The order status lookup itself needs a connected store (Shopify, Shoptet, Upgates, PrestaShop or Eshop-rychle) and **Enable order fetching via API** in Chat Settings & Embed (a paid feature).

**Keep pages current.** On the entry of a scraped page that changes often, tick **Scrape this page again every day and retrain**. Sitemaps you subscribed to are checked every day at 3:00 AM for new, changed and removed pages.

### 4. Write clear instructions in the chatbot role

The chatbot role is the set of instructions the model gets with every question (its system prompt). It says who the chatbot is and how it behaves, not what it knows. Edit it in **Chatbot** → **Chat Settings & Embed** → **Chatbot role**, or in **Knowledge Base** → **Get Data** → **Chatbot Role (Instructions)**, where the app also explains how to write it. **Copy as Markdown** there copies a brief you can paste into ChatGPT, Claude, Gemini or Grok to draft the role with their help.

The app recommends 15–40 lines in five parts:

1. **Who you are**: the chatbot's personality in two or three sentences.
2. **How you communicate**: tone, language and length of answers.
3. **Rules**: conditional rules such as “share X only when Y”.
4. **Escalation**: when to offer to pass the customer to a person.
5. **Examples**: two or three ideal questions and answers.

A short role in that shape can look like this (replace the parts in brackets):

```text
You are the product advisor of [store name]. You are friendly, practical and save the customer's time.
Answer in two or three short sentences; give details only when the customer asks for them.
Recommend at most three products and say in one sentence why each one fits.
Share our contact details only if you cannot solve the question.
If the customer is upset, acknowledge it first, then solve the problem.
If the knowledge base does not answer the question, offer to pass it to our team.

Example:
Customer: [a real question from your Chat History]
You: [the answer your best salesperson would give]
```

Say what the chatbot should do rather than listing what it must not do; the app notes that long lists of prohibitions make a chatbot hesitant. Know the limits of the role:

- **Instructions cannot add knowledge.** Prices, products and policies belong in the knowledge base, where they are searched and kept up to date. A price written into the role is soon outdated.
- **Longer is not better.** A long role makes the model inconsistent; half a page works better than three pages.
- **Keep the chatbot on your data.** The switch next to **Chatbot system role** is on by default: the chatbot then answers from your instructions and your knowledge base. When it is off, the model also uses its general knowledge, which can lead to inaccurate or off-topic answers. Keep it on for a store.

**Reset to Default** restores the default role. A hand-over creates a ticket only when **Enable Ticketing System** is on in **Account Settings** → **Tickets Settings**, so write the escalation rule to match.

### 5. Choose the AI model and search settings

Change the model when the analysis keeps reporting a model limitation, or when answers are too slow or cost too much. Set it in **Chat Settings & Embed** → **AI model type**. To compare models on your own data, open **Test Chatbot** → **Compare Models**: one question goes to a model from OpenAI, Anthropic, Google and xAI at the same time, with your chatbot's settings, and the four answers appear side by side. Each model can take 15–30 seconds. What to compare is explained in [how to choose an LLM for customer support](https://www.chaterimo.com/en/blog/how-to-choose-llm-for-customer-support/), and the [AI benchmark](https://www.chaterimo.com/en/ai-benchmark/) shows how current models score on e-commerce questions.

The model list depends on your plan: plans with Chaterimo API keys offer a shorter list, plans with your own API key (BYOK: you use your own key and pay the model provider directly) offer every model.

Two more settings decide how the chatbot works with your data:

Search flow
:   **Advanced RAG** is the default and fits routine product questions, FAQs and customer support. RAG (retrieval-augmented generation) means the chatbot first searches your knowledge base and gives the passages it found to the model together with the question. **Agentic search v2** lets the AI choose its own search steps and check the products it found; the app recommends it for complex products with many variants and parameters, where customers ask about compatibility. Its answers take longer and use more tokens. After you switch to it, Chaterimo prepares a catalogue index, usually within a minute; until then the chatbot uses the standard search.

Reasoning effort
:   On plans with your own API key, models that support it show **Reasoning effort**: how much the model reasons before it answers. **none** is the fastest and cheapest. Plans with Chaterimo API keys do not show this setting.

Select **Save changes**. Model, role and search flow apply to the next answer; they need no training.

### 6. Retrain and test on Test Chatbot

After you change the knowledge base, open **Knowledge Base** → **Knowledge Base** and select **Train chatbot** (or **Retrain**). Entries waiting for training are marked **Not trained**. During the trial and on paid plans, new and changed data is also trained automatically every night, so training by hand only makes the change take effect sooner. Until training finishes, Test Chatbot warns that answers may be incomplete.

Then open **Chatbot** → **Test Chatbot** and ask the questions from your test set:

1. Select **New conversation** before each question, so earlier messages do not change the answer. If you have several chat widgets, choose the one to test under **Chat widget**.
2. Check the answer against the answer you expect: facts, products, tone and length.
3. Open **Sources** under the answer. It lists the products, pages, files and texts the answer was built from; select one to see its **Source Details**. If the right source is missing, fix the data (step 3). If it is there and the answer is still wrong, look at the role (step 4) or the model (step 5).
4. For an answer you cannot explain, select **Analyze why AI answered like this** (step 2).

Test Chatbot runs in testing mode: it creates no tickets, sends no ticket e-mails and its conversations do not appear in Chat History.

### 7. Tune the proactive sales agent

The sales agent sends a proactive message: after a delay on the pages you choose, the chat window opens by itself with a message for the visitor. Rules are in **Chatbot** → **Sales Assistant**, on the Growth, Evolve, Advanced and Masterful plans; the number of active rules and AI messages per month depends on the plan ([pricing](https://www.chaterimo.com/en/pricing/)).

Each rule has a **Match Type** (**All pages**, **Only these pages** or **All except these pages**), **Page Patterns**, a **Delay (seconds)** and a **Message Type**:

AI Generated
:   The AI writes the message from your **AI Instructions** and the page the visitor is on. It uses the AI model and language of the chat widget the rule belongs to and counts against your monthly AI messages.

Direct Message
:   Your own **Message Text**, shown at once and without AI cost. Direct messages are unlimited on all plans.

The app's own sales agent guide suggests starting with a delay of 30–60 seconds and using AI messages on specialised pages such as product pages, the cart and the checkout, not everywhere. Each rule fires once per visitor session. Change one thing at a time and compare the results of the rule, as described below.

## How to check it works

Compare the same measurements before and after each change, and change one thing at a time, so you know what helped.

| What to measure | Where to find it | A better result |
|---|---|---|
| Correct answers in your test set | Your own list, asked in Test Chatbot | More answers correct and complete |
| Flagged answers | **Dashboard** → **Flagged Messages - 30 days**: flagged answers, all AI answers and the rate in % (shown when message flagging is on) | A lower rate |
| Clicks on product cards and links | **Dashboard** → **Link Clicks - 30 days** and **AI Link Engagement**: clicks, AI messages with links and the click-through rate | A higher click-through rate |
| Hand-overs | **Tickets**: conversations the chatbot handed over | Fewer tickets about questions the knowledge base should answer |
| Proactive messages | **Sales Assistant** → **Stats** of a rule: **Total triggers**, **Message Visibility**, **Reply Rate**, **Customer Sentiment**, **Performance by Page URL** | More visitors who see the message reply |

The Dashboard cards cover the last 30 days, so compare them month by month. Write the date of each change next to your results.

## Troubleshooting

The chatbot says it does not know, although the answer is on your website
:   The page is not in the knowledge base or is not trained yet. Search for it in the Knowledge Base; if it is missing, add it with **Scrape a single link**, then select **Train chatbot**. In Test Chatbot, **Sources** shows whether the page is used.

The chatbot gives an old price or wrong availability
:   The product data is outdated. Turn on **Download this feed again every day and retrain** for the feed, or correct the product in your store administration if a store integration imports it. A feed with an old `g:sale_price` shows that price, because Chaterimo prefers it over `g:price`.

The chatbot mentions a finished promotion or old shipping prices
:   An old page or text is still in the knowledge base. Delete it, or add its path to **Do not import these pages from the sitemap.**, then retrain.

The chatbot answers questions unrelated to your store or adds details that are not in your data
:   Check that the switch next to **Chatbot system role** is on, and add a rule to the chatbot role about what the chatbot does when the knowledge base has no answer.

“Your chatbot is still being trained. Responses may be incomplete or inaccurate.”
:   Training is still running. Wait until the Dashboard shows “All available data trained and ready.” and test again.

“You have reached the answer analysis limit. Please try again later.”
:   The number of answer analyses per hour is used up. Try again later and analyse the answers that matter most first.

“No usable provider API key is configured for this organisation.”
:   The analysis needs an API key. Add your own key in **Account Settings**, see [how to get an OpenAI API key](https://www.chaterimo.com/en/blog/how-to-openai-api-account/), or choose a plan with Chaterimo API keys.

“Your data exceeded your plan limit. Please upgrade your plan to continue.”
:   The knowledge base is larger than your plan allows. Delete data the chatbot does not need, such as duplicate pages, or choose a larger plan on the [pricing page](https://www.chaterimo.com/en/pricing/).

“AI model temporarily overloaded” with a Gemini model
:   The app notes that Google's API is sometimes unstable. Switch the chatbot to a GPT or Claude model in **AI model type**.

Questions about an order get a general answer without the order details
:   The order status lookup is off or no store is connected. Turn on **Enable order fetching via API** in Chat Settings & Embed and connect your store in **Integrations**.

## Next step

Compare models on real e-commerce questions in our [blind rating of AI models for customer support](https://www.chaterimo.com/en/blog/best-ai-for-customer-support/) before you switch. If you want more models or reasoning settings, set up your own API key with the guide for [OpenAI](https://www.chaterimo.com/en/blog/how-to-openai-api-account/), [Anthropic](https://www.chaterimo.com/en/blog/how-to-anthropic-api-account/) or [Gemini](https://www.chaterimo.com/en/blog/how-to-gemini-api-account/). For the questions an AI should hand over to people, read [how to set up AI live chat on your website](https://www.chaterimo.com/en/blog/how-to-set-up-ai-live-chat-on-your-website/).

**Does the chatbot learn from conversations by itself?**

No. It answers from your knowledge base and your instructions, and conversations are not added to the knowledge base. To teach it something, add the missing information in Get Data and train the chatbot.

**How often do I need to retrain the chatbot?**

During the trial and on paid plans, new and changed data is trained automatically every night. Select Train chatbot in the Knowledge Base when a change should take effect sooner. Changes to the model, the chatbot role or the search flow need no training.

**Can the chatbot answer only from my own data?**

Yes. Keep the switch next to Chatbot system role on, which is the default. The chatbot then answers from your instructions and knowledge base and does not add general knowledge of the model.

**Does the answer analysis cost anything?**

It uses AI tokens like any answer. With your own API key you pay the model provider; on plans with Chaterimo API keys it runs on Chaterimo's keys. The number of analyses per hour is limited.