52% is the best score an AI model reached on 100 questions drawn from 14 manufacturers’ catalogs. It got 46 fully right. Modern Distribution Management reports the benchmark, from Reshape Automation, which sells industrial AI technology and wrote and graded the questions itself.
What the article says
Modern Distribution Management’s Mike Hockett, writing on October 7, reports a benchmark from Reshape Automation that put 100 questions from 14 manufacturers’ catalogs to general-purpose AI models, making 1,500 API calls. With web search, GPT-6 Astra reached 52% and Claude 40.5%. Without tools, GPT-6 Astra, Claude and Gemini scored 12% to 14%; Gemini was tested without tools only. Scores include half credit for partially correct answers, and the leading result held 46 fully correct answers out of 100. An estimated 36.5% of incorrect answers supplied a specific fact, figure or code without qualifying the response. MDM notes that Reshape, “which sells industrial AI technology, wrote and graded the questions without third-party review.”
Why it matters at the top of the house
The questions came from manufacturers’ own catalogs. On their own the models scored 12% to 14% on them; with the web to search, the best reached 52%, with 46 answers fully right. More than a third of the wrong answers came as a specific fact, figure or code with nothing to mark the doubt. The benchmark is a vendor’s own and has had no outside review, so it is a first reading and not a settled measure.
Source: Modern Distribution Management, “Study: AI Models Top Out at 52% on Industrial Parts Questions,” by Mike Hockett, October 7, 2026. Research: Reshape Automation. The Marketing Table shares the link and a note on why it matters; the article and its copyright belong to the publisher.
Leave a Reply