One-sentence summary
The Artificial Analysis Cyber Index measures agentic cyber-defense ability by averaging three equally weighted sub-evaluations; on October 24, 2026, DeepSeek-V4.1-Flash-1017 ranked first among publicly available models with a score of 57.
Background
In September 2026, Artificial Analysis launched the Cyber Index (version 1.0, running since September 2026) to measure large language models on agentic cybersecurity defense work: finding vulnerabilities in a codebase, reproducing and validating them, and patching them without breaking existing functionality.
On October 24, 2026, Artificial Analysis published a new leaderboard (UTC October 23, 2026, 17:23): DeepSeek-V4.1-Flash-1017 (max) ranked first among publicly available models with a composite score of 57%. The previous leaderboard top was shared by Grok 4.7 (xhigh) and MiMo-V2.6-Pro at 56.
Scope and Design Principles
The Cyber Index defines a narrow evaluation boundary. The official methodology lists what it does not cover: incident response, writing new code without introducing vulnerabilities, targets without source access (compiled software or live servers), and exploit realization.
Every task starts with source-code access, mirroring how a security engineer audits an application. No task asks a model to turn a vulnerability into a working exploit. Cases where a model or provider declines a task on safety grounds are recorded separately and excluded from the score.
All models run on the same open-source agent harness, Stirrup, with identical prompts and tools within each evaluation, in isolated sandboxes with no internet access.
The Three Equally Weighted Sub-Evaluations
The Cyber Index composite is an equally weighted average of three evaluations, each at one-third weight.
CWE-Bench-AA is Artificial Analysis' implementation of Collinear AI's CWE-bench, with 120 held-out tasks covering all ten OWASP Top 10 (2025) categories across C/C++, Go, Java, JavaScript/TypeScript, Python, and Rust. Each task mirrors a real security audit: the agent receives a repository checkout and a single audit instruction, with a 2-hour limit. Grading uses a deterministic verifier confirming the exploit no longer works and legitimate behavior still works, with a headline pass@1.
DeepsecBench-AA is an implementation of Vercel's DeepsecBench, measuring how well models find vulnerabilities in application code. Tasks use real open-source application code pinned to a pre-fix commit; agents submit findings within up to 500 turns. Findings are compared against an expert-verified golden set, judged by GPT-5.6 Sol (high), with a headline F2 score (recall weighted above precision), repeated three times and reported as the median.
CyberGym-E2E-AA implements CyberGym-E2E from Berkeley RDI, covering 131 memory-safety vulnerabilities from C/C++ open-source projects (filtered from a 920-vulnerability, 139-project dataset, one task per project). Agents locate the vulnerability, write a proof-of-concept input that triggers a sanitizer crash, and submit a patch within 90 minutes. Scoring runs four cumulative stages: PoC crashes the unpatched build, the patch fixes the crash, patched functionality tests pass, and a diagnostic stage checks the ground-truth fix. A task scores 1 only when stages 1-3 pass, with a headline pass@1.
Scoring and Safety Refusals
The composite is an equally weighted average of the three sub-evaluations. All sub-evaluations run on the Stirrup harness with the same prompts and tools in sandboxes without internet access. Safety refusals are recorded separately from scores, so declining defensive work does not inflate or deflate capability scores.
The leaderboard also reports cost per task and safety blocks for joint cost-capability assessment. Artificial Analysis maintains a separate "trusted-access" category (for example, GPT-6 Sol Daybreak Blue), where models are evaluated through vendor-exclusive access and ranked apart from publicly available models.
Leaderboard Evolution and Latest Results
After the Cyber Index launched in September 2026, the public-model head score stayed at 56: Grok 4.7 (xhigh) and MiMo-V2.6-Pro tied for first, GPT-6 Luna (Max) at 53, and GLM-5.3-Flash at 50. Independent analysis (al-ice.ai, September 29, 2026) noted large-scale refusals of defensive work by some frontier models, with no composite score above 56%.
On October 24, 2026, Artificial Analysis published a new leaderboard: DeepSeek-V4.1-Flash-1017 (max) became the first among publicly available models with 57%, 1 point above the previous tied top. The model is the October 17, 2026 update of DeepSeek's V4.1-Flash (released September 10, 2026), a 552B-parameter MoE with 8B active input and 16B active output parameters.
Limitations and Usage Notes
The official methodology warns that results depend on implementation: Artificial Analysis' numbers are not directly comparable to those published by cwe-bench.com, Vercel, or Berkeley RDI. It does not cover incident response, no-source targets, or exploit realization. Pass rates reflect single-attempt results.
For model selection, the Cyber Index measures only one dimension of agentic cyber defense and should be combined with other capability benchmarks and cost data. When citing scores, specify the model version (for example, -1017), reasoning tier (for example, max), and publication date.
Data Sources and References
- Artificial Analysis Cyber Index methodology: https://artificialanalysis.ai/methodology/cyber-index
- Artificial Analysis Cyber Index leaderboard: https://artificialanalysis.ai/evaluations/artificial-analysis-cyber-index
- Artificial Analysis trusted-access models: https://artificialanalysis.ai/articles/trusted-access-models-cyber-index
- BAAI Hub (October 24, 2026, DeepSeek top report): https://hub.baai.ac.cn/?tag_id=1003
- DeepSeek official news (V4.1 Flash release): https://www.deepseek.com/en/news/deepseek-v4-1-flash/
- al-ice.ai analysis (September 29, 2026): https://al-ice.ai/posts/2026/09/artificial-analysis-cyber-index-refusals-patching-gap/