# Best AI for customer support: 12 models rated blind

- URL: https://www.chaterimo.com/en/blog/best-ai-for-customer-support/
- Language: en
- Type: Research
- Author: David Langr
- Published: 2026-06-14
- Updated: 2026-10-19
- Data period: Round 4, snapshot of 30 August 2026

> ChatGPT, Claude, Gemini and Grok on 45 real e-commerce support questions, scored blind by an AI judge model. Edition 2026 Q3.

## Key findings

- In round 4 (30 August 2026) the three ChatGPT 5.6 models lead with **71.7**, 69.8 and 69.7 points out of 100.
- Of the models that can be selected in Chaterimo on 19 October 2026, **Grok 4.1 Fast** scored best (65.2) and answered in 2.2 s on average.
- **Accuracy** is the weakest criterion for every model: the best reached 55.4, while tone scored between 81 and 90 for all twelve.
- Average response times range from **1.3 s** (Gemini 3.5 Flash-Lite) to **49.7 s** (Grok 4.6).

## Models tested

- ChatGPT 5.6 Sol (2026-08-30)
- ChatGPT 5.6 Luna (2026-08-30)
- ChatGPT 5.6 Terra (2026-08-30)
- ChatGPT 4.1 mini (gpt-4.1-mini, 2026-08-30)
- ChatGPT 4.1 (gpt-4.1, 2026-08-30)
- Claude Sonnet 5 (2026-08-30)
- Claude Opus 5 (2026-08-30)
- Claude Haiku 4.5 (2026-08-30)
- Gemini 3.7 Flash (2026-08-30)
- Gemini 3.5 Flash-Lite (gemini-3.5-flash-lite, 2026-08-30)
- Grok 4.1 Fast (grok-4-1-fast-non-reasoning, 2026-08-30)
- Grok 4.6 (2026-08-30)

Which language model answers e-commerce customers best? In round 4 of the Chaterimo CX-LLM Benchmark, 12 models answered the same 45 real customer-support questions and were scored blind by an AI judge model. The three ChatGPT 5.6 models led with 69.7 to 71.7 points out of 100. Every model scored lowest on accuracy, the share of facts it got right.

## Methodology

The questions come from real customer-support conversations in online stores that use Chaterimo. Every model receives the same system prompt, the same knowledge-base context and the same question. Their answers are labelled anonymously (Answer A, Answer B …) and scored by a judge model. In round 4 the judge was Gemini 3.1 Pro (preview) from Google, which is not one of the 12 tested models, so no model scored its own answer. Round 1 used judges from OpenAI, Anthropic, Google and xAI, rounds 2 and 3 judges from OpenAI and Google.

Each answer gets a score from 0 to 100 on six criteria. The overall score is their weighted average:

Accuracy (30%)
:   Are the facts about products, prices and policies correct?

Relevance (20%)
:   Does the answer address what the customer asked?

Completeness (15%)
:   Does it contain everything the customer needs to decide?

Helpfulness (15%)
:   Does it move the customer towards a purchase or a solution?

Tone (10%)
:   Is it friendly and professional?

Conciseness (10%)
:   Is it short without leaving out anything important?

Public scores use only questions that every compared model answered, so no model profits from an easier mix. Round 4 has 45 such questions and 540 blind evaluations. The benchmark also measures the average time to a complete answer.

## Results

![Horizontal bar chart of the overall score of the 12 models in round 4, from 71.7 to 56.8 points out of 100](https://chaterimo.fra1.cdn.digitaloceanspaces.com/blog/best-ai-for-customer-support/images/01-overall-score-56b03e36f3.svg "Overall score of the 12 models in round 4, weighted average of six criteria, 0–100. Source: Chaterimo CX-LLM Benchmark, round 4, 30 August 2026.")

| # | Model | Provider | Overall | Accuracy | Tone | Avg. response |
|---:|---|---|---:|---:|---:|---:|
| 1 | ChatGPT 5.6 Sol | OpenAI | 71.7 | 55.4 | 86.0 | 3.5 s |
| 2 | ChatGPT 5.6 Luna | OpenAI | 69.8 | 52.7 | 88.0 | 2.8 s |
| 3 | ChatGPT 5.6 Terra | OpenAI | 69.7 | 51.3 | 89.7 | 3.2 s |
| 4 | Grok 4.1 Fast | xAI | 65.2 | 49.0 | 82.1 | 2.2 s |
| 5 | Claude Haiku 4.5 | Anthropic | 65.1 | 45.7 | 85.4 | 4.0 s |
| 6 | ChatGPT 4.1 mini | OpenAI | 63.9 | 41.6 | 85.2 | 2.9 s |
| 7 | Claude Sonnet 5 | Anthropic | 63.6 | 40.3 | 85.9 | 7.8 s |
| 8 | Gemini 3.7 Flash | Google | 63.6 | 35.4 | 89.3 | 10.1 s |
| 9 | Grok 4.6 | xAI | 63.0 | 41.0 | 81.1 | 49.7 s |
| 10 | Claude Opus 5 | Anthropic | 61.2 | 37.4 | 87.8 | 11.4 s |
| 11 | ChatGPT 4.1 | OpenAI | 61.2 | 34.4 | 85.3 | 2.3 s |
| 12 | Gemini 3.5 Flash-Lite | Google | 56.8 | 32.8 | 81.3 | 1.3 s |

Scores 0–100, higher is better; for response time lower is better. The CSV under "Download the data" has all six criteria for every model. Source: Chaterimo CX-LLM Benchmark, round 4, 30 August 2026.

## Which models lead?

The three ChatGPT 5.6 models take the first three places with 69.7 to 71.7 points. Grok 4.1 Fast follows with 65.2 and Claude Haiku 4.5 with 65.1. The last model, Gemini 3.5 Flash-Lite, is 14.9 points behind the first, so in this round the choice of model made a clear difference.

## Which of these models can you use today?

Providers replace model versions every few months. On 19 October 2026, four of the twelve models in round 4 can be selected in Chaterimo: Grok 4.1 Fast (65.2), ChatGPT 4.1 mini (63.9), ChatGPT 4.1 (61.2) and Gemini 3.5 Flash-Lite (56.8). The others have been replaced by newer versions, which were not part of round 4.

ChatGPT 4.1 mini is the most stable model of the benchmark. It took part in all four rounds and scored between 60.8 and 63.9 each time.

## Why is accuracy the lowest score?

Accuracy checks whether the facts in an answer are correct: products, prices, stock, delivery and return policies. Even with the same knowledge-base context, the best model reached 55.4 points and the weakest 32.8. Tone, by contrast, scored between 81 and 90 for every model.

![Horizontal bar chart of the accuracy score of the 12 models in round 4, from 55.4 to 32.8 points out of 100](https://chaterimo.fra1.cdn.digitaloceanspaces.com/blog/best-ai-for-customer-support/images/02-accuracy-28ea7d8ac8.svg "Accuracy score of the 12 models in round 4, 0–100. Source: Chaterimo CX-LLM Benchmark, round 4, 30 August 2026.")

For a store this means that a fluent, friendly answer is not proof of a correct one. Check the facts in your chatbot's answers before you trust it, and keep the catalogue and policies it reads complete.

## How fast do the models answer?

Gemini 3.5 Flash-Lite answered in 1.3 s on average and Grok 4.1 Fast in 2.2 s. Grok 4.6 needed 49.7 s, Claude Opus 5 11.4 s and Gemini 3.7 Flash 10.1 s. A customer waiting in a chat window notices every second, so speed belongs to the decision as much as the score.

## How do the four rounds compare?

Each round uses a different set of questions, so results across rounds show a trend, not a controlled comparison. Claude Haiku 4.5 scored 64.3, 49.1, 53.5 and 65.1 in rounds 1 to 4; such swings come mostly from the question mix. Compare models within one round.

## Limitations

- The judge is an AI model, not a person. Its scores follow the criteria above, but it can have blind spots.
- In round 4 a single judge model from Google scored all answers, and two of the tested models also come from Google. The next round uses judges from OpenAI, Anthropic, Google and xAI.
- 45 questions are a small sample of what customers ask.
- The results apply to the model versions and the date of round 4. Newer versions can score differently.
- The questions come from stores that use Chaterimo; other industries can rank models differently.

**Which AI is best for e-commerce customer support?**

In round 4 of the Chaterimo CX-LLM Benchmark (30 August 2026), ChatGPT 5.6 Sol scored highest with 71.7 out of 100. Of the models selectable in Chaterimo on 19 October 2026, Grok 4.1 Fast scored best with 65.2.

**Is ChatGPT or Claude better for customer support?**

In round 4 the ChatGPT 5.6 models scored 69.7 to 71.7, the Claude models 61.2 to 65.1. The difference comes mostly from accuracy; all models scored similarly on tone.

**How is the benchmark scored?**

A judge model scores each anonymous answer from 0 to 100 on accuracy, relevance, completeness, helpfulness, tone and conciseness. In round 4 the judge was Gemini 3.1 Pro from Google, which is not one of the tested models.