Series Chaterimo CX-LLM Benchmark ยท Round 6

Live results

AI Model Benchmark

ChatGPT, Claude, Gemini and Grok answer the same real questions from online store customers. AI judges score every answer blind, without knowing which model wrote it.

Updated

Key findings

  1. Current leader: ChatGPT 6.1 Sol with an average score of 75.2 across 15 shared questions and 165 blind evaluations.
  2. Fastest answers: Gemini 3.5 Flash-Lite, 1.5 s on average.
  3. Best model of each provider: ChatGPT 6.1 Sol (75.2), Gemini 3.8 Flash (65.0), Claude Haiku 5.5 (64.8), Grok 4.7 (62.3).
Latest round
Round 6
Tested on
Questions
15
Models
11
AI judges
1
Contents
  1. Results of the latest round
  2. Score Breakdown
  3. Round history
  4. Average across all rounds
  5. How It Works
  6. Frequently Asked Questions

Results of the latest round

All models of the round, including retired ones. Show current models only

  • OpenAI
  • Anthropic
  • Google
  • xAI
Fig. 1Overall score in round 6, weighted average of six criteria, 0โ€“100. Source: Chaterimo CX-LLM Benchmark, round 6, Oct. 21, 2026.
Leaderboard of AI models ranked by blind evaluation scores on shared e-commerce customer service questions.
# Model Provider Overall Score Avg Response
1 ChatGPT 6.1 Sol OpenAI 75.2 5.3 s
2 ChatGPT 6 Luna OpenAI 66.9 4.1 s
3 Gemini 3.8 Flash Google 65.0 11.0 s
4 Claude Haiku 5.5 Anthropic 64.8 2.6 s
5 Claude Sonnet 5.5 Anthropic 64.4 4.3 s
6 ChatGPT 4.1 miniretired OpenAI 63.1 3.1 s
7 Claude Opus 5.5 Anthropic 62.8 7.5 s
8 Grok 4.7 xAI 62.3 38.8 s
9 ChatGPT 4.1retired OpenAI 54.5 3.7 s
10 Gemini 3.5 Flash-Lite Google 52.8 1.5 s
11 Grok 4.1 Fast xAI 51.0 2.4 s

Scores are 0โ€“100, higher is better; for response time, lower is better.

Score Breakdown

Per-criterion benchmark scores showing how each model performs on accuracy, relevance, completeness, helpfulness, tone, and conciseness.

Model
ChatGPT 6.1 Sol 72.0 79.7 64.7 66.7 88.3 91.7
ChatGPT 6 Luna 50.7 81.3 63.3 55.7 84.7 91.3
Gemini 3.8 Flash 56.0 75.3 61.3 56.3 78.3 76.0
Claude Haiku 5.5 49.3 81.3 61.3 53.0 80.0 85.7
Claude Sonnet 5.5 48.0 77.0 68.0 52.3 86.0 79.7
ChatGPT 4.1 miniretired 45.3 76.0 63.0 53.0 85.0 84.3
Claude Opus 5.5 39.0 77.7 73.0 55.7 87.3 75.0
Grok 4.7 42.0 77.0 65.7 51.0 85.7 82.7
ChatGPT 4.1retired 31.0 71.7 51.0 41.7 87.3 82.0
Gemini 3.5 Flash-Lite 40.3 58.7 49.3 41.0 80.7 73.0
Grok 4.1 Fast 32.7 67.7 46.7 36.7 71.0 80.3

Round history

Each round uses a different set of questions and judges, so compare models only within one round.

Round Tested on Questions Models AI judges Winner
Round 6 15 11 1 ChatGPT 6.1 Sol 75.2
Round 5 44 11 1 ChatGPT 6 Luna 66.4
Round 4 45 12 1 ChatGPT 5.6 Sol 71.7retired
Round 3 30 12 2 ChatGPT 4.1 mini 62.7retired
Round 2 23 12 2 ChatGPT 4.1 mini 60.8retired
Round 1 28 14 4 ChatGPT 5.4 69.5retired

Average across all rounds

Note

Rounds differ in questions and judges, so this average is not a fair comparison: a model that took part only in easier rounds ranks higher. Use the latest round to choose a model.

Weighted average across 6 rounds. Models with more evaluations weigh more heavily.
# Model Provider Overall Score Rounds Total evals
1 ChatGPT 6.1 Sol OpenAI 75.2 1/6 15
2 ChatGPT 5.6 Solretired OpenAI 71.7 1/6 45
3 ChatGPT 5.6 Lunaretired OpenAI 69.8 1/6 45
4 ChatGPT 5.6 Terraretired OpenAI 69.7 1/6 45
5 ChatGPT 6 Luna OpenAI 66.6 2/6 59
6 Claude Haiku 5.5 Anthropic 64.8 1/6 15
7 Claude Sonnet 5.5 Anthropic 64.4 1/6 15
8 Gemini 3.7 Flashretired Google 63.6 1/6 45
9 ChatGPT 5.4 miniretired OpenAI 63.1 3/6 218
10 Grok 4.6retired xAI 63.0 1/6 45
11 Claude Opus 4.6retired Anthropic 63.0 1/6 84
12 Claude Sonnet 4.6retired Anthropic 62.5 3/6 218
13 ChatGPT 5.4retired OpenAI 62.2 3/6 137
14 ChatGPT 6 Solretired OpenAI 61.8 1/6 44
15 ChatGPT 4.1 miniretired OpenAI 61.4 6/6 322
16 Claude Opus 5retired Anthropic 61.2 1/6 45
17 Gemini 3.1 Flash-Literetired Google 60.2 1/6 112
18 Grok 4retired xAI 59.6 1/6 112
19 ChatGPT 4.1retired OpenAI 59.3 6/6 322
20 Claude Haiku 4.5retired Anthropic 59.1 5/6 307
21 Grok 4.1 Fast xAI 59.1 4/6 216
22 Gemini 3.8 Flash Google 58.2 2/6 59
23 Gemini 3.1 Pro Previewretired Google 57.9 3/6 137
24 Claude Sonnet 5retired Anthropic 57.7 2/6 89
25 Claude Opus 4.7retired Anthropic 56.4 3/6 131
26 ChatGPT 5.5retired OpenAI 56.4 2/6 106
27 Grok 4.7 xAI 56.1 2/6 59
28 Grok 4.20retired xAI 55.5 1/6 84
29 Grok 4.3retired xAI 54.8 2/6 106
30 Gemini 3.1 Flash-Literetired Google 54.6 2/6 106
31 Claude Opus 5.5 Anthropic 54.4 2/6 59
32 Gemini 3 Flashretired Google 54.2 1/6 112
33 Gemini 3.5 Flash-Lite Google 53.7 3/6 104
34 Gemini 3.5 Flashretired Google 47.6 2/6 106

How It Works

  1. Real Questions. Selected from actual production customer service conversations in e-commerce.
  2. Same Prompt. All models receive the identical system prompt, knowledge base, and question.
  3. Blind Evaluation. Evaluators see only 'Answer A', 'Answer B' โ€” they don't know which model wrote it.
  4. Cross-Evaluation. Top-tier models from each provider evaluate answers. No model judges its own response.

Round 6 was scored by one AI judge: Gemini 3.1 Pro Preview.

Scoring Criteria

Each answer is scored 0-100 on six criteria with the following weights:

Accuracy (30%)
Are the facts about products, prices and policies correct?
Relevance (20%)
Does the answer address what the customer asked?
Completeness (15%)
Does it contain everything the customer needs to decide?
Helpfulness (15%)
Does it move the customer towards a purchase or a solution?
Tone (10%)
Is it friendly and professional?
Conciseness (10%)
Is it short without leaving out anything important?

To keep the comparison fair, public scores are calculated only from questions answered by every model included in the selected comparison set. That prevents newer or retired models from benefiting from an easier question mix.

Frequently Asked Questions

What is this AI benchmark measuring?

This benchmark measures how well leading AI models handle real customer service tasks for online stores. It focuses on practical support quality โ€” accuracy, helpfulness, tone, and conciseness โ€” rather than coding, math, or generic reasoning tests.

Which AI model is best for customer service?

The best model depends on your store, language mix, product complexity, and speed requirements. This page shows which models currently perform best in our blind benchmark, helping you shortlist candidates for your own live testing.

ChatGPT vs Claude vs Gemini โ€” which is better for ecommerce support?

Each provider has strengths. ChatGPT models tend to be fast and widely supported. Claude models often excel at nuanced, context-heavy responses. Gemini models offer strong multilingual capabilities. Grok models provide competitive performance at lower latency. Check the leaderboard above for the latest blind comparison.

How does the blind evaluation work?

Every model receives the identical question, system prompt, and knowledge base. Their answers are then labeled anonymously (Answer A, Answer B, etc.) and scored by top-tier AI judges from each provider โ€” OpenAI, Anthropic, Google, and xAI. No model evaluates its own response, eliminating self-evaluation bias.

Can I use this benchmark to choose an AI chatbot for my Shopify or ecommerce store?

Yes. The questions come from real production conversations in online stores, including Shopify, Shoptet, WooCommerce, and others. Use the leaderboard as a starting point, then test top models with your own product catalog and brand tone before going live.

How should I use these benchmark scores?

Use the leaderboard as a decision aid, not as the only deciding factor. Start with the highest-ranked models, then test them on your own knowledge base, brand tone, and response speed requirements before rolling out in production.

How often is this benchmark updated?

We add new models as providers release them and periodically expand the question set with fresh real-world scenarios. When a new model is added, it is tested on the same shared questions as all existing models to keep the comparison fair.

chaterimo logo

Copyright ยฉ Chaterimo

about-icon