Havas Media titled a corresponding publication “Can AI replace the media agency?” It would therefore be quite surprising if the accompanying study, conducted by the Havas Group’s media subsidiary, were to arrive at a resounding “Yes” as the answer. Not entirely surprisingly, the “Havas AI Media Quality Index” test revealed sobering results. According to the findings, agency work is still indispensable.
Havas Media Germany has conducted extensive studies of specific media plan designs using the “Havas AI Media Quality Index” (HAI-Q), which the agency aims to establish as its own AI benchmark for media planning. This research is driven by the increasing use of AI systems like ChatGPT or Claude by marketing professionals who want to develop media strategies, review budget decisions, or question agency recommendations.
The new benchmark is designed to measure how well AI models can solve real-world media planning tasks. Unlike traditional AI tests, which primarily assess language comprehension or general knowledge, the HAI-Q focuses on practical questions, according to Havas Media in Frankfurt. These include strategic media planning, budget allocation, target group analysis, and the calculation and interpretation of reach, GRP, and impact metrics.
For the initial evaluation, Havas developed a test set with 35 tasks tailored to the German market. The aim was to reveal the difference between linguistically convincing answers and actual professional competence.
The agency found the results sobering. According to their findings, current Large Language Models only answer typical media planning questions correctly in about half of the cases on average. The tested systems showed significant weaknesses, particularly in data-driven decisions and numerical calculations.
In a direct comparison, the “Agentic Media Machine” developed by Havas achieved 33 out of a possible 35 points, resulting in a hit rate of 94.3 percent. According to the benchmark, GPT-5 achieved 16 correct answers (45.7 percent), Claude Sonnet 4.5 six (17.1 percent), and GPT-40 four (11.4 percent).
Florian Teschner, Head of Data & AI at Havas Media Germany and responsible for the project, says: “Many answers seem convincing at first glance. That’s precisely where the danger lies.” Marketing managers often receive recommendations that appear strategically plausible, but whose data or conclusions are not necessarily reliable. “Our benchmark shows that there is still a significant difference between a good-sounding answer and a sound media recommendation.”
This becomes particularly evident in reach calculations, target group models, or budget decisions. While large language models often deliver solid results for open-ended strategic questions, the quality drops significantly as soon as concrete data analyses are required.
Havas explicitly states that the results are not a criticism of AI technologies. Rather, the agency sees them as evidence of the need for specialized systems. “The exciting question is not whether AI will replace the media agency,” says Teschner. “The real question is: What kind of AI can actually improve media decisions?” Generalist models can be valuable sparring partners, but for reliable planning results, specialized systems, proprietary data sources, and industry-specific expertise are required.
According to the agency, the Agentic Media Machine developed by Havas is based on a further development of the “ Meaningful Media Machine ” which was introduced in 2023. The system combines language models with media data, planning logics and expert knowledge and is now available to the network’s employees.
With the HAI-Q, Havas plans to regularly compare the performance of various AI systems in the marketing and media sector. Further test questions, additional model comparisons, and an expansion to international markets and other marketing disciplines are planned. Havas itself is currently accelerating its engagement with AI within its agency network.
Teschner sees the future of AI-supported media planning primarily in the combination of various technologies and tools: “The future does not belong to individual models. It belongs to intelligently orchestrated systems that combine data, technology, and human expertise. That is precisely where we see the next stage of development in AI-supported media planning.”
