AI Model Benchmark

Real-world e-commerce customer service scenarios, blind evaluation by multiple AI judges

This benchmark compares ChatGPT, Claude, Gemini, and Grok on real e-commerce customer service questions. It is designed for teams choosing the best AI model for support chat, help desk automation, and AI sales-assistant workflows.

Current leader: ChatGPT 4.1 mini with an average score of 62.7 across 30 shared questions and 656 blind evaluations.

30 questions evaluated 656 evaluations performed Last updated: Jun 05, 2026

Overall ranking across all snapshots

Weighted average across 3 snapshots. Models with more evaluations weigh more heavily.

Cross-snapshot weighted-average ranking; rounds column shows how many snapshots each model participated in.
# Model Provider Overall Score Rounds Total evals
1 ChatGPT 5.4 mini OpenAI
63.1
3/3 218
3 Claude Sonnet 4.6 Anthropic
62.5
3/3 218
4 ChatGPT 5.4 OpenAI
62.2
3/3 137
5 ChatGPT 4.1 mini OpenAI
62.2
3/3 218
6 ChatGPT 4.1 OpenAI
60.3
3/3 218
10 Claude Haiku 4.5 Anthropic
58.1
3/3 218
11 Gemini 3.1 Pro Preview Google
57.9
3/3 137
12 Claude Opus 4.7 Anthropic
56.4
3/3 131
13 ChatGPT 5.5 OpenAI
56.4
2/3 106
15 Grok 4.3 xAI
54.8
2/3 106
16 Gemini 3.1 Flash-Lite Google
54.6
2/3 106
18 Gemini 3.5 Flash Google
47.6
2/3 106

Latest round — Jun 05, 2026

Leaderboard of AI models ranked by blind evaluation scores on shared e-commerce customer service questions.
# Model Provider Overall Score Avg Response
1 ChatGPT 4.1 mini OpenAI
62.7
4.4s
2 Claude Sonnet 4.6 Anthropic
61.0
8.3s
3 ChatGPT 5.4 mini OpenAI
59.9
3.6s
4 ChatGPT 5.5 OpenAI
58.3
7.5s
5 Grok 4.3 xAI
57.7
4.7s
6 ChatGPT 4.1 OpenAI
57.2
4.4s
7 Claude Opus 4.7 Anthropic
56.4
11.5s
8 Gemini 3.1 Flash-Lite Google
55.5
1.7s
9 Claude Haiku 4.5 Anthropic
53.5
4.9s
10 Gemini 3.1 Pro Preview Google
53.1
15.4s
11 ChatGPT 5.4 OpenAI
52.2
8.5s
12 Gemini 3.5 Flash Google
50.5
9.5s

Score Breakdown

Per-criterion benchmark scores showing how each model performs on accuracy, relevance, completeness, helpfulness, tone, and conciseness.
Model Accuracy (30%) Relevance (20%) Completeness (15%) Helpfulness (15%) Tone (10%) Conciseness (10%)
ChatGPT 4.1 mini 48.0 74.1 59.2 54.0 82.6 82.1
Claude Sonnet 4.6 44.0 74.5 59.6 51.7 83.0 79.0
ChatGPT 5.4 mini 44.6 70.8 56.6 51.3 80.9 80.9
ChatGPT 5.5 40.0 71.2 57.5 48.0 82.8 79.6
Grok 4.3 42.9 70.5 49.4 47.3 78.1 84.0
ChatGPT 4.1 38.1 72.1 55.8 47.9 82.0 76.3
Claude Opus 4.7 36.3 70.5 56.2 45.2 83.6 78.6
Gemini 3.1 Flash-Lite 40.3 65.7 54.8 45.5 83.2 69.4
Claude Haiku 4.5 33.2 67.4 52.3 42.8 81.6 76.0
Gemini 3.1 Pro Preview 47.4 62.3 39.6 42.3 75.8 65.3
ChatGPT 5.4 32.5 66.7 57.2 37.0 79.3 70.8
Gemini 3.5 Flash 43.6 61.2 41.3 39.0 71.2 60.6

How It Works

Real Questions

Selected from actual production customer service conversations in e-commerce.

Same Prompt

All models receive the identical system prompt, knowledge base, and question.

Blind Evaluation

Evaluators see only 'Answer A', 'Answer B' — they don't know which model wrote it.

Cross-Evaluation

Top-tier models from each provider evaluate answers. No model judges its own response.

Scoring Criteria

Each answer is scored 0-100 on six criteria with the following weights:

Accuracy 30%
Relevance 20%
Completeness 15%
Helpfulness 15%
Tone 10%
Conciseness 10%

To keep the comparison fair, public scores are calculated only from questions answered by every model included in the selected comparison set. That prevents newer or retired models from benefiting from an easier question mix.

Results over time

Each round uses a different set of questions, so trends are indicative, not a controlled comparison.

Round-by-round average scores (all models)
Model Round 1Round 2Round 3
Claude Haiku 4.5 64.349.153.5
Claude Opus 4.6 63.0——
Claude Opus 4.7 65.051.156.4
Claude Sonnet 4.6 66.754.561.0
Gemini 3 Flash 54.2——
Gemini 3.1 Flash-Lite —53.455.5
Gemini 3.1 Flash-Lite 60.2——
Gemini 3.1 Pro Preview 61.251.953.1
Gemini 3.5 Flash —43.850.5
ChatGPT 4.1 63.257.157.2
ChatGPT 4.1 mini 62.660.862.7
ChatGPT 5.4 69.548.752.2
ChatGPT 5.4 mini 65.960.659.9
ChatGPT 5.5 —53.958.3
Grok 4 59.6——
Grok 4.1 Fast 58.4——
Grok 4.20 55.5——
Grok 4.3 —51.257.7

Frequently Asked Questions

This benchmark measures how well leading AI models handle real customer service tasks for online stores. It focuses on practical support quality — accuracy, helpfulness, tone, and conciseness — rather than coding, math, or generic reasoning tests.

The best model depends on your store, language mix, product complexity, and speed requirements. This page shows which models currently perform best in our blind benchmark, helping you shortlist candidates for your own live testing.

Each provider has strengths. ChatGPT models tend to be fast and widely supported. Claude models often excel at nuanced, context-heavy responses. Gemini models offer strong multilingual capabilities. Grok models provide competitive performance at lower latency. Check the leaderboard above for the latest blind comparison.

Every model receives the identical question, system prompt, and knowledge base. Their answers are then labeled anonymously (Answer A, Answer B, etc.) and scored by top-tier AI judges from each provider — OpenAI, Anthropic, Google, and xAI. No model evaluates its own response, eliminating self-evaluation bias.

Yes. The questions come from real production conversations in online stores, including Shopify, Shoptet, WooCommerce, and others. Use the leaderboard as a starting point, then test top models with your own product catalog and brand tone before going live.

Use the leaderboard as a decision aid, not as the only deciding factor. Start with the highest-ranked models, then test them on your own knowledge base, brand tone, and response speed requirements before rolling out in production.

We add new models as providers release them and periodically expand the question set with fresh real-world scenarios. When a new model is added, it is tested on the same shared questions as all existing models to keep the comparison fair.
chaterimo logo

Copyright © Chaterimo

about-icon