Series Chaterimo CX-LLM Benchmark ยท Round 6
Live results
AI Model Benchmark
ChatGPT, Claude, Gemini and Grok answer the same real questions from online store customers. AI judges score every answer blind, without knowing which model wrote it.
Key findings
- Current leader: ChatGPT 6.1 Sol with an average score of 75.2 across 15 shared questions and 165 blind evaluations.
- Fastest answers: Gemini 3.5 Flash-Lite, 1.5 s on average.
- Best model of each provider: ChatGPT 6.1 Sol (75.2), Gemini 3.8 Flash (65.0), Claude Haiku 5.5 (64.8), Grok 4.7 (62.3).
- Latest round
- Round 6
- Tested on
- Questions
- 15
- Models
- 11
- AI judges
- 1
Contents
Results of the latest round
Only models you can choose in Chaterimo today. Include retired models
- OpenAI
- Anthropic
- xAI
| # | Model | Overall Score | Avg Response |
|---|---|---|---|
| 1 | ChatGPT 6.1 Sol | 75.2 | 5.3 s |
| 2 | ChatGPT 6 Luna | 66.9 | 4.1 s |
| 3 | Gemini 3.8 Flash | 65.0 | 11.0 s |
| 4 | Claude Haiku 5.5 | 64.8 | 2.6 s |
| 5 | Claude Sonnet 5.5 | 64.4 | 4.3 s |
| 6 | Claude Opus 5.5 | 62.8 | 7.5 s |
| 7 | Grok 4.7 | 62.3 | 38.8 s |
| 8 | Gemini 3.5 Flash-Lite | 52.8 | 1.5 s |
| 9 | Grok 4.1 Fast | 51.0 | 2.4 s |
Scores are 0โ100, higher is better; for response time, lower is better.
Score Breakdown
Per-criterion benchmark scores showing how each model performs on accuracy, relevance, completeness, helpfulness, tone, and conciseness.
| Model | ||||||
|---|---|---|---|---|---|---|
| ChatGPT 6.1 Sol | 72.0 | 79.7 | 64.7 | 66.7 | 88.3 | 91.7 |
| ChatGPT 6 Luna | 50.7 | 81.3 | 63.3 | 55.7 | 84.7 | 91.3 |
| Gemini 3.8 Flash | 56.0 | 75.3 | 61.3 | 56.3 | 78.3 | 76.0 |
| Claude Haiku 5.5 | 49.3 | 81.3 | 61.3 | 53.0 | 80.0 | 85.7 |
| Claude Sonnet 5.5 | 48.0 | 77.0 | 68.0 | 52.3 | 86.0 | 79.7 |
| Claude Opus 5.5 | 39.0 | 77.7 | 73.0 | 55.7 | 87.3 | 75.0 |
| Grok 4.7 | 42.0 | 77.0 | 65.7 | 51.0 | 85.7 | 82.7 |
| Gemini 3.5 Flash-Lite | 40.3 | 58.7 | 49.3 | 41.0 | 80.7 | 73.0 |
| Grok 4.1 Fast | 32.7 | 67.7 | 46.7 | 36.7 | 71.0 | 80.3 |
Round history
Each round uses a different set of questions and judges, so compare models only within one round.
| Round | Tested on | Questions | AI judges | Winner |
|---|---|---|---|---|
| Round 6 | 15 | 1 | ChatGPT 6.1 Sol 75.2 | |
| Round 5 | 44 | 1 | ChatGPT 6 Luna 66.4 | |
| Round 4 | 45 | 1 | ChatGPT 5.6 Sol 71.7retired | |
| Round 3 | 30 | 2 | ChatGPT 4.1 mini 62.7retired | |
| Round 2 | 23 | 2 | ChatGPT 4.1 mini 60.8retired | |
| Round 1 | 28 | 4 | ChatGPT 5.4 69.5retired |
How It Works
- Real Questions. Selected from actual production customer service conversations in e-commerce.
- Same Prompt. All models receive the identical system prompt, knowledge base, and question.
- Blind Evaluation. Evaluators see only 'Answer A', 'Answer B' โ they don't know which model wrote it.
- Cross-Evaluation. Top-tier models from each provider evaluate answers. No model judges its own response.
Round 6 was scored by one AI judge: Gemini 3.1 Pro Preview.
Scoring Criteria
Each answer is scored 0-100 on six criteria with the following weights:
- Accuracy (30%)
- Are the facts about products, prices and policies correct?
- Relevance (20%)
- Does the answer address what the customer asked?
- Completeness (15%)
- Does it contain everything the customer needs to decide?
- Helpfulness (15%)
- Does it move the customer towards a purchase or a solution?
- Tone (10%)
- Is it friendly and professional?
- Conciseness (10%)
- Is it short without leaving out anything important?
To keep the comparison fair, public scores are calculated only from questions answered by every model included in the selected comparison set. That prevents newer or retired models from benefiting from an easier question mix.
Frequently Asked Questions
What is this AI benchmark measuring?
This benchmark measures how well leading AI models handle real customer service tasks for online stores. It focuses on practical support quality โ accuracy, helpfulness, tone, and conciseness โ rather than coding, math, or generic reasoning tests.
Which AI model is best for customer service?
The best model depends on your store, language mix, product complexity, and speed requirements. This page shows which models currently perform best in our blind benchmark, helping you shortlist candidates for your own live testing.
ChatGPT vs Claude vs Gemini โ which is better for ecommerce support?
Each provider has strengths. ChatGPT models tend to be fast and widely supported. Claude models often excel at nuanced, context-heavy responses. Gemini models offer strong multilingual capabilities. Grok models provide competitive performance at lower latency. Check the leaderboard above for the latest blind comparison.
How does the blind evaluation work?
Every model receives the identical question, system prompt, and knowledge base. Their answers are then labeled anonymously (Answer A, Answer B, etc.) and scored by top-tier AI judges from each provider โ OpenAI, Anthropic, Google, and xAI. No model evaluates its own response, eliminating self-evaluation bias.
Can I use this benchmark to choose an AI chatbot for my Shopify or ecommerce store?
Yes. The questions come from real production conversations in online stores, including Shopify, Shoptet, WooCommerce, and others. Use the leaderboard as a starting point, then test top models with your own product catalog and brand tone before going live.
How should I use these benchmark scores?
Use the leaderboard as a decision aid, not as the only deciding factor. Start with the highest-ranked models, then test them on your own knowledge base, brand tone, and response speed requirements before rolling out in production.
How often is this benchmark updated?
We add new models as providers release them and periodically expand the question set with fresh real-world scenarios. When a new model is added, it is tested on the same shared questions as all existing models to keep the comparison fair.
