Claude’s language gap does not close in English

The pages

96 local pages of the November 2026 AIGROW Index, drawn at random with a fixed seed: 16 in each of Italian, German, French, Spanish and Portuguese, and 16 in 15 US cities whose names are written the same in Italian and German. Every group has the same 16 kinds of business, one page each.

The questions

Each page asks the four questions of its November page: one for the best of the trade in the city, and three for a concrete need. The questions in English name each city by its English name. Each assistant was asked each question 3 times in every arm, 10,944 answers in all.

The arms

On the 80 pages in the five languages, each question was asked three ways: in the city’s language with the instruction in English, as the Index asks it; in the city’s language with the instruction in that language too; and in English. On the 16 US pages it was asked in English, in Italian with the instruction in English, in Italian with the instruction in Italian, and in German with the instruction in English.

The instruction

Every question goes with the four sentences the AIGROW scan and the Index send as the system instruction: “You are answering an ordinary consumer who is choosing a supplier. Name the specific businesses you would actually recommend, in order. Use real business names only. If you do not know any for this question, return an empty list. Do not hedge, do not explain your limitations, do not suggest how to search.” The same four sentences in each of the five languages are listed below.

The models

gpt-6-sol, gemini-3.1-flash and claude-sonnet-5, asked through their APIs between 14:58 and 15:21 UTC on 9 November 2026, exactly as the November Index asked them: ChatGPT and Gemini at temperature 0.3 with low reasoning effort, Claude at its default temperature without extended thinking, each answering with a list of up to eight businesses.

From memory

No answer used web search, and no assistant was told where the person asking was. Every answer was a new conversation.

An empty answer

An answer is empty when its list names no business. Names were grouped by business with the method the Index uses, so two spellings of one business count once, and each group was looked up in the page’s November ranking.

Statistics

Every empty-answer rate carries a 95% Wilson interval. A change between two arms is pooled over the same pages, and its 95% interval comes from 2,000 resamples of whole pages. Name overlap is the share of businesses two arms both named on a page, averaged over the pages where either arm named someone.

The instruction in Italian

Stai rispondendo a un normale consumatore che deve scegliere a chi rivolgersi. Indica in ordine le attività precise che consiglieresti davvero. Usa solo nomi di attività reali. Se per questa domanda non ne conosci nessuna, restituisci un elenco vuoto. Non metterti al riparo con cautele, non spiegare i tuoi limiti e non suggerire come cercare.

The instruction in German

Du beantwortest die Frage eines ganz normalen Verbrauchers, der einen Anbieter auswählt. Nenne der Reihe nach die konkreten Unternehmen, die du wirklich empfehlen würdest. Verwende nur echte Firmennamen. Wenn du für diese Frage keine kennst, gib eine leere Liste zurück. Antworte ohne Vorbehalte, erkläre nicht deine Grenzen und schlage nicht vor, wie man suchen kann.

The instruction in French

Tu réponds à un consommateur ordinaire qui choisit un prestataire. Cite dans l’ordre les entreprises précises que tu recommanderais vraiment. N’utilise que des noms d’entreprises réels. Si tu n’en connais aucune pour cette question, renvoie une liste vide. Ne prends pas de précautions oratoires, n’explique pas tes limites et ne suggère pas comment chercher.

The instruction in Spanish

Estás respondiendo a un consumidor corriente que está eligiendo un proveedor. Nombra por orden los negocios concretos que recomendarías de verdad. Usa solo nombres de negocios reales. Si no conoces ninguno para esta pregunta, devuelve una lista vacía. No pongas reservas, no expliques tus limitaciones y no sugieras cómo buscar.

The instruction in Portuguese

Vai responder a um consumidor comum que precisa de escolher um fornecedor. Indique por ordem as empresas concretas que realmente recomendaria. Use apenas nomes reais de empresas. Se não conhecer nenhuma para esta pergunta, devolva uma lista vazia. Não faça ressalvas, não explique as suas limitações e não sugira como pesquisar.

One day

All answers were collected on 9 November 2026. The models change, and the same questions asked another week may come out differently.

Answers from memory

The assistants had no web search and no location. An assistant that searches the web, as the consumer apps often do, may name businesses it would not name from memory.

The answer form stays English

The description of the list the assistants fill in is in English in every arm, so an arm with the instruction in another language is not wholly in that language.

Sixteen pages a language

16 pages per language and per group, so the figures for one language carry wide intervals. The tables print each with its interval.

Translated wording

The five translations of the instruction say the same thing in different words, so a change with the instruction is a change of wording as well as of language. Claude moved one way in Italian and the other way in German and French, which suggests the wording counts.

Two askings differ

The same question asked twice in the same language also names different businesses, and the study did not measure that baseline, so a name overlap below 100% is not all due to the language.

Missing answers

6 of the 10,944 questions got no list from Gemini even after two retries, and they are left out of every rate.

Different cities

The pages in each language are in that language’s cities, and the US pages are in large US cities. A gap between countries can come from the size or the fame of the cities as much as from the country.

Empty is not wrong

An empty answer can be the honest one, and a list can name businesses that do not exist or have closed. The study does not check the names; the share found in the November ranking is a partial check only.

Competing interests

The author founded AIGROW, which publishes the Index and sells services that aim to improve how businesses appear in AI answers. The study had no external funding.

• Accounting software in Spain
• Accounting software in Germany
• Accounting software in Austria
• Accounting software in Poland
• Hair salons in Rome
• Hair salons in Naples
• Hair salons in Florence
• Veterinarians in Florence

• Gyms in Turin
• Plumbers in Rome
• CRM software in Italy
• Accounting software in Italy
• Project management software

Download the data as CSV

The CSV holds the counts and rates behind every table, by language, arm, assistant and question. The answers themselves stay with AIGROW Labs.