GLM 5.2 outperforms Claude on cybersecurity benchmarks. New contender reshapes code security comparisons.

A security engineer is triaging a batch of flagged code samples, trying to determine which AI model will catch the most real vulnerabilities without flooding the queue with false positives. They run the same set of malicious snippets through two models. One of them, a Chinese open-weights model that few engineers on their team have ever heard of, keeps winning. That is the situation Semgrep's researchers found themselves in when they ran GLM 5.2 against Claude on their internal cybersecurity code benchmarks.

Why this benchmark result deserves skepticism before action

Before treating this as a signal to swap your security tooling, it helps to understand what benchmark wins usually mean in practice: not much, or at least not what they appear to mean at first glance.

Semgrep built their Mythos benchmark specifically around cybersecurity code analysis. That is a narrow slice of what a model does in a real security workflow. A model that scores well on pattern-matching malicious code samples in a controlled evaluation can still fail badly at writing clear remediation guidance, handling edge cases in real repositories, or reasoning across multiple files with shared context. Benchmark tasks are, almost by definition, cleaner and more bounded than production work.

There is also the question of who benefits from a headline like this. GLM 5.2 is a model from Zhipu AI, a Beijing-based company. Results that position it above Claude on a high-profile security benchmark attract attention and downloads. That does not make the results wrong, but it does mean they deserve the same scrutiny you would apply to any vendor-adjacent evaluation. Semgrep's researchers are credible and their methodology appears transparent, but one benchmark run, however well designed, is not a deployment decision.

Finally, Claude is not a security-specialized model. Comparing a general-purpose model to a benchmark built around cybersecurity code patterns, and then declaring a winner, is a bit like benchmarking a Swiss Army knife against a scalpel and declaring that scalpels are better tools. The framing matters as much as the number.

The number that actually anchors this

The win is domain-specific. Semgrep's Mythos benchmark focuses on cybersecurity code tasks: identifying vulnerabilities, classifying malicious patterns, and reasoning about attack surfaces in code. GLM 5.2 outperforming Claude in this context means it is better calibrated for this category of problem, not that it is a superior model across the board.

What would it mean if GLM 5.2's margin were twice as large? It would suggest the gap is architectural or training-data-driven rather than noise, which would make the result more reproducible and more actionable. What if the margin were half as wide? It would land well within the variance you would expect from prompt phrasing, temperature settings, and evaluation sample size, meaning the result would tell you almost nothing.

The source article does not publish a specific numeric margin, so the honest read is: GLM 5.2 won on this benchmark, the win is in a specialized domain, and the magnitude is not independently confirmed. That is not nothing, but it is also not a migration plan.

For teams whose entire AI workload is security code analysis, this result is worth investigating seriously. For teams using Claude for a mix of tasks that happens to include security review, it changes very little.

How to run your own comparison before drawing conclusions

If your team does security code analysis at scale, the right response to this result is not to switch models and not to ignore it. It is to run a structured comparison on your own workload. Here is how to do that without building a benchmark from scratch.

1. Pull 20 to 30 real code samples from your own triage history. Include a mix of true positives (confirmed vulnerabilities) and false positives (flagged but clean code) so you can measure precision and recall separately, not just accuracy.
2. Write a single system prompt that represents how you actually use the model. Do not optimize it for either model. Use the same prompt for both runs.
3. Run each sample through GLM 5.2 via its API and through whichever Claude version you currently use in production. Log the raw responses, not just pass/fail scores.
4. Score each response against your ground truth labels. Track four numbers: true positives, false positives, true negatives, false negatives. Calculate F1 for each model, not just accuracy.
5. Read 10 of the false positive responses carefully. Accuracy numbers can look similar while the failure modes are completely different. A model that confidently misidentifies clean code is a different kind of problem than a model that hedges on real vulnerabilities.

Verification test: Take one confirmed vulnerability from your sample set and one confirmed clean snippet. Submit both to each model with no additional context beyond the system prompt you wrote in step 2. If the model correctly classifies both and provides a coherent explanation for each, your prompt is usable for the broader evaluation. If it fails on either, fix the prompt before running the full batch.

What the Semgrep team actually said

"We have Mythos at home" - Semgrep research team, referencing the internal benchmark that produced the GLM 5.2 result. (Source)

The headline is a riff on the "we have X at home" meme, which is self-aware about what they are claiming: not that GLM 5.2 is objectively better, but that their internal benchmark, built for their specific use case, surfaces a model that works better for them than the flagship options everyone defaults to. That framing is actually more useful than a straight benchmark comparison, because it is honest about the scope.

This is the correct way to think about model selection for specialized tasks. The general leaderboards, whether MMLU, HumanEval, or public coding benchmarks, are built around tasks that approximate general capability. A security team at a company like Semgrep has a narrow, well-defined problem space. Building an internal evaluation against that problem space and running candidates through it is exactly the right methodology. The finding that a less prominent model wins on a specialized task should surprise no one who has spent time evaluating models for production use.

The broader implication is not "stop using Claude." It is "stop assuming the default choice is optimal for your specific task." That applies to code-focused tools as much as it does to raw model selection. Claude Code may still be the right choice for general development workflows even if GLM 5.2 turns out to be better at vulnerability classification. The tasks are different enough that the comparison barely overlaps.

For more on how open-weights models are performing against the closed-source incumbents on specialized tasks, the Qwen 3 analysis covers similar territory from a different angle.