I love LLMs, I hate hype. A developer's reality check on AI capabilities.

A product manager is evaluating AI tools for their team. They read the vendor page, watch the demo, and come away convinced the model can "reason through complex business problems autonomously." Three weeks into the trial, it hallucinates a competitor's pricing, confidently cites a document it was never given, and refuses to make a judgment call without being pushed. The demo was real. The expectation it set was not.

George Hotz published a post this week that cuts through that exact problem. The title is blunt: "I love LLMs, I hate hype." The argument is not that current models are bad. It is that the marketing layer around them actively obscures what they are good for, which makes them harder to use well. That distinction is worth taking seriously, because it changes how you evaluate tools, choose models, and set expectations with the people you work with.

The number that keeps getting misread

The 80/20 framing comes up constantly in honest assessments of LLM performance, and it is worth sitting with what it actually means. A model that handles 80% of a coding task well is not "nearly there." It is a tool that requires a skilled person to catch and fix the remaining 20%, which often contains the load-bearing logic. If the person reviewing the output cannot recognize what is wrong, the 80% is not a win - it is a liability with a confidence veneer on top.

Double that number and think about what changes. An LLM that handled 95-99% of tasks correctly would transform deployment decisions in a real and measurable way. At that level, the review overhead drops low enough that you could route large categories of work through it with minimal supervision. The gap between 80% and 95% is not a 15-point improvement. It is the difference between a tool that accelerates an expert and a tool that can operate without one nearby.

Halve the number instead. At 40% reliability, the same tool becomes slower than doing the work manually, because you spend more time verifying than you would have spent producing. Some categories of AI-assisted work are already in this zone. Long-form factual research, multi-step agentic tasks with external tools, anything requiring accurate memory across a session longer than a few thousand tokens - these regularly fall below the threshold where the assistance pays for itself in time saved.

The honest framing is not "AI is good" or "AI is bad." It is: which specific tasks sit in the high-reliability zone for this specific model, and which do not. That question almost never appears in vendor marketing.

Two takes on whether the hype matters

Skeptic: The hype is just marketing. Engineers read past it, teams calibrate after a few weeks of use, and the market corrects. Why is this worth analyzing?

Counter: Because calibration has a cost. Every team that over-deploys based on inflated expectations burns engineering time, loses stakeholder trust, and sometimes ships something that damages users. The "market corrects" framing assumes the correction is cheap. It is not. A six-week integration built on a model capability that does not hold up in production is not a learning experience - it is a write-off.

Skeptic: But the underlying models are improving at a real and meaningful pace. GPT-5.6 and Claude Sonnet 5 are meaningfully better than what existed two years ago. The hype is premature, not wrong.

Counter: Premature hype applied to a capability that does not exist yet still causes the same downstream damage. And "better than two years ago" is not the same as "good enough for the use case you are about to commit to." The benchmark numbers improve. The specific failure mode you will hit in production usually does not appear in any benchmark. That is the gap the hype fills with false confidence.

A decision tree for evaluating any AI tool claim

Start here: is the capability claim specific or generic? If the marketing says "understands your business context" or "reasons through complex problems," that is a generic claim with no testable prediction. Move on or test it yourself with a task that would actually fail if the claim were false.

If the claim is specific - "extracts structured data from PDFs with high accuracy," "generates working SQL from natural language" - ask whether there is a benchmark or third-party evaluation behind it. If yes, check whether the benchmark conditions match your actual use case. Benchmarks run on clean, well-formatted inputs. Your data is probably not that.

If you cannot find an independent evaluation, run one yourself before committing. Take ten real examples from your actual workload. Not curated examples, not edge cases you already know will fail - representative samples. Score the output on your own criteria. If the accuracy is above the threshold where it saves you time, proceed. If it is below, stop.

If the tool costs money and you are evaluating at the API level, look at the pricing. Claude Sonnet 5 at $3.00/$15.00 per million tokens and Gemini 3.5 at $1.50/$9.00 are both reasonable for high-volume text tasks where the accuracy holds. At those prices, a failed evaluation costs very little. A six-week integration that fails in week five costs a great deal more.

If the task involves persistent memory, multi-step tool use, or autonomous decision-making over a long session, apply extra skepticism regardless of what the demo showed. These are the categories where the gap between demo and production is widest. If you cannot test a realistic version of the full workflow before committing, treat the capability as unconfirmed.

If someone on your team says "the model is almost there, we just need to tune the prompts," ask them to define what "there" looks like and how they will know when they have reached it. Vague improvement trajectories are how integration projects stay unfinished for months.

Why the hype exists and why it is structurally hard to fix

The incentives are not mysterious. Labs need capital, and capital flows toward audacious claims. But the mechanism that makes hype durable is less obvious than the incentive structure suggests.

LLMs are impressive on first contact in ways that are hard to dismiss. A model that produces fluent, confident, structurally coherent text about almost any topic feels more capable than it is, because fluency and accuracy are not the same thing and they are hard to disentangle quickly. The first demo is almost always good. Failure modes emerge with volume, edge cases, and sustained use - exactly the conditions that do not appear in a sales cycle.

This creates a gap that hype fills naturally. The product manager who saw the demo is not lying when they report it was impressive. The engineer who hit the wall three weeks later is not wrong either. Both are reporting real data. The problem is that the data from the first contact is systematically more visible than the data from sustained deployment, so the prior in most organizations is set too high.

There is also a status dynamic. Admitting that a heavily hyped tool underperformed in your specific use case feels like admitting you did not know how to use it. So the correction often happens quietly, at the team level, without surfacing to the broader conversation that sets organizational priors. The hype propagates. The recalibration does not.

Tools like Cursor and Claude Code have moved the needle in clear and practical ways for developers who work within the constraints of what they are actually good at: generating boilerplate, refactoring bounded functions, explaining unfamiliar codebases. The developers who report the most value from these tools are almost always the ones who have the clearest mental model of where they break. That is not a coincidence. It is what calibrated use looks like. For more on the practical limits of current models in production environments, the gap between benchmark and deployment reality is a recurring theme.

Matching the tool to the task

User type / use case	Best fit	Why it wins
Developer writing boilerplate and unit tests	Claude Sonnet 5 or GPT-5.6 via a coding assistant	High reliability on bounded, well-defined tasks; pricing supports volume use
Analyst summarizing long documents	Gemini 3.5 or Claude Sonnet 5	Strong long-context handling; competitive cost at $1.50-$3.00 input per million tokens
Team evaluating autonomous agents for multi-step workflows	Start with a constrained pilot, not a production deployment	Agentic failure modes are hard to predict without real-workload testing; no current model is reliably autonomous at scale
Content team producing high-volume drafts	Claude Haiku 4.5 or Gemini 2.5 Flash	Low per-token cost ($0.30-$1.00 input) suits volume; quality is sufficient for draft-level work with human review
Researcher needing factual accuracy on specialized topics	Human verification pipeline, model as first draft only	No current model has reliable factual accuracy on specialized or recent information; treat output as a starting point, not a source
Budget-conscious team exploring AI for the first time	GPT-5 at $1.25/$10.00 per million tokens	Solid general capability at a price point that keeps evaluation costs low while you build a real picture of what the model can and cannot do