Can AI Count to 4 in 2026? Testing Perplexity, RTX 5090 Local Models, and DGX Spark

Published 05/05/2026

Description

ai benchmark, llm benchmark, llm comparison, ai model comparison, can ai count, counting benchmark, prompt following, llm reasoning, ai reasoning test, local llm, local ai models, perplexity ai, perplexity models, ollama, rtx 5090, dgx spark, nvidia dgx spark, llm test, ai evaluation, model accuracy, real world llm test, open source llm, cloud vs local llm, gemma, qwen, llama, nemotron, kimi, glm, name filtering test, ai fails simple task,samayas, stef Description: One year later, I reran my simple but revealing AI benchmark with a fresh lineup of current models. The task sounds trivial: take a list of names and keep only the ones with exactly 4 characters. But as this test shows, even modern models can still fail at counting, filtering, prompt following, and basic reasoning when the prompt is simple and the answer should be obvious. 0:00 - Introduction 0:9 - Description 1:04 - The List 1:33 - Online GPT 5.4 2:02 - Online GPT 5.4 Thinking 2:21 - Online Gemini 3.1 Pro Thinking 2:54 - Online Claude Sonnet 4.6 3:14 - Online Claude Sonnet 4.6 Thinking 3:43 - Online Nemotron 3 Super 4:40 - Online Kimi K2.6 5:07 - Online Kimi K2.6 Thinking 5:19 - Local Gemma 4 26b 5:55 - Local GLM 4.7 Flash 6:11 - Local Nemotron 3 Nano 6:49 - Local Qwen 3.5 27B Claude 4.6 Opus Reasoning 8:05 - Local Seed OSS 36b 10:17 - Local Qwen3 coder next 10:49 - Local LLama 4 scout 17b 11:13 - DGX GPT OSS 128b 11:45 - DGX LLama 4 Scout 17B 16E 12:03 - DGX Nemotron 3 Super 12:56 - DGX Gemma 4 31b 13:49 - DGX Qwen 3.6 35b a3b 14:18 - Results 16:21 - Conclusion In this updated version, I compare: Online models accessed through Perplexity Local models running on an RTX 5090RTX 5090 Models running on a DGX SparkDGX Spark The final results side by side This benchmark is easy to understand, quick to reproduce, and surprisingly useful for spotting where LLMs overthink, miscount, hallucinate structure, or simply ignore the instructions. If you are interested in AI evaluation, local L