Software Engineering
Code development and debugging
320 datasets
last indexed 17h ago
Model Leaderboard
| No. | Model | Score |
|---|---|---|
| 1 |  Claude Sonnet 4.5 | 0.93 |
| 2 |  GPT-5 Codex | 0.87 |
| 3 | GPT-5.1 (high) | 0.87 |
| 4 | GPT-5.1 | 0.86 |
| 5 | Claude Sonnet 4 (thinking) | 0.86 |
| 6 | GPT-5 | 0.84 |
| 7 | Claude 3.7 Sonnet (thinking) | 0.75 |
| 8 | Claude Sonnet 4 | 0.75 |
| 9 | Claude Opus 4.1 | 0.72 |
| 10 | GPT-5 (high) | 0.71 |
| 11 |  Gemini 3 Pro | 0.68 |
| 12 | Grok 4 Fast (reasoning) | 0.65 |
| 13 | Kimi K2 Thinking | 0.55 |
| 14 | GPT-5 Mini | 0.54 |
| 15 | Gemini 2.5 Pro | 0.54 |
| 16 |  DeepSeek V3.1 | 0.52 |
| 17 | Qwen3 235B A22B | 0.52 |
| 18 | Claude 4.5 Sonnet (thinking) | 0.49 |
| 19 | Grok 4 | 0.49 |
| 20 | Qwen3 VL 235B A22B Thinking | 0.49 |
| 21 | Claude 3.7 Sonnet | 0.48 |
| 22 | Grok 4 Fast | 0.43 |
| 23 | Claude Haiku 4.5 | 0.43 |
| 24 | Kimi-K2-Instruct | 0.42 |
| 25 | Kimi K2 0905 | 0.42 |
| 26 | o4-mini (high) | 0.40 |
| 27 | Grok Code Fast 1 | 0.40 |
| 28 | Claude Opus 4 | 0.27 |
| 29 | MiniMax M2 | 0.26 |
| 30 | Qwen3 Next 80B A3B Thinking | 0.26 |
| 31 | Claude 3.5 Sonnet v2 | 0.24 |
| 32 | Gemini 2.5 Flash | 0.21 |
| 33 | Z.AI: GLM 4.5 | 0.19 |
| 34 | Qwen3 Coder 480B A35B | 0.18 |
| 35 | o3 (high) | 0.18 |
| 36 | gpt-oss-120b | 0.17 |
| 37 | DeepSeek-R1 | 0.13 |
| 38 | Z.AI: GLM 4.5 Air | 0.11 |
| 39 | GPT-5 Nano | 0.05 |
| 40 | gpt-oss-20b | 0.00 |
| Name | Organization | Best Model |
|---|---|---|
| LiveCodeBench Pro | University of Washington | Gemini 3 Pro Preview |
| LiveBench | New York University | Claude 4 Sonnet |
| SWE-Bench Pro | Scale AI | Claude 4.5 Sonnet |
| GSO | UC Berkeley | Claude-4.5-Sonnet |
| Terminal Bench | Terminal Bench | Gemini 3 Pro |
| SWE-Lancer | OpenAI | Claude 3.5 Sonnet |
| HumanEval | OpenAI | Codex-12B |
| Design2Code | Google DeepMind | GPT-4o |
| LiveCodeBench | UC Berkeley | O4-Mini (High) |
| AL-Bench | The Chinese University of Hongkong, Shenzhen | FastLog |
| HumanEval-Decompile | The Hong Kong Polytechnic University | Ghidra + LLM4Decompile-Ref-22B |
| SWT-Bench | ETH Zurich | GPT-5 |
| CodeAgentBench | Peking University | GPT-4-turbo |
| BigCodeBench | University of Illinois at Urbana-Champaign | Claude 3.7 Sonnet (20250219) |
| CodeElo | Alibaba Group | o1-mini |
| SWE-Dev | Shanghai Jiao Tong University | Claude-3.7-Sonnet-thinking |
| rSDE-Bench | Shanghai AI Laboratory | EvoMAC |
| PwP-Bench | Carnegie Mellon University | Claude 3.5 Sonnet |
| HPC Performance Optimization Benchmark | Oak Ridge National Laboratory | Codee |
| SWE-bench Multimodal | Stanford University | GPT-4o |
| DA-Code | Shanghai Artificial Intelligence Laboratory | GPT-4 |
| VADER | UC Berkeley | o3 |
| APPS | University of Illinois at Urbana-Champaign | GPT-Neo 2.7B |
| SWE-bench-Live | Shanghai Artificial Intelligence Laboratory | Qwen3-Coder-480B-A35B |
| SciReplicate-Bench | The Alan Turing Institute | claude-3-sonnet-20240229 |
| SecRepoBench | Google DeepMind | GPT-5 |
| EvalPlus | University of Illinois at Urbana-Champaign | O1 Preview |
| WebGen-Bench | The Chinese University of Hong Kong | WebGen-LM-32B |
| LeetCodeDataset | IEEE | DeepSeek-R1 |
| CyberSecEval | Meta | Llama-2-7b-chat |
| Multi-SWE-bench | Peking University | Gemini-2.5-Pro |
| REPOEXEC | FPT Software AI Center | DeepSeek-R1 |
| SciCode | University of Illinois at Urbana-Champaign | OpenAI o3-mini-low |
| Web-Bench | ByteDance | Web-Agent (Claude 3.7 Sonnet) |
| FEA-Bench | Peking University | DeepSeek-R1 |
| RustEvo2 | Sun Yat-Sen University | Claude-3.7-Sonnet |
| Copilot Arena | Carnegie Mellon University | deepseek-coder-fim |
| BigOBench | Meta | DeepSeek-R1 Llama 70B |
| CodeFlowBench | Shanghai University of Finance and Economics | GPT-4.1-mini |
| SWE-PolyBench | AWS AI Labs | Deepseek R1 |
| RepoBench | University of California, San Diego | Codex (code-davinci-002) |
| AutoCodeBench | Tencent | Claude Opus 4 (20250514) (Reasoning) |
| LongCodeBench | Stanford University | Claude 3.5 Sonnet |
| CoV-Eval | National Engineering Research Center for Software Engineering, Peking University | claude-3-sonnet-20240229 |
| SWE-QA | Shanghai Jiao Tong University | Claude 3.7 Sonnet |
| EffiBench | University of Waterloo | starcoder2-15b |
| GitTaskBench | Chinese Academy of Sciences | Claude 3.7 Sonnet |
| MigrationBench | Amazon | Claude-3.5-Sonnet-v2 |
| DS-1000 | University of Washington | Codex-002 |
| Sallm | University of Notre Dame | GPT-3.5 |
| TestGenEval | Carnegie Mellon University | GPT-4o |
| A.S.E | Chinese Academy of Sciences | Claude-3.7-Sonnet-20250219 |
| Vibe Checker | Google DeepMind | GPT 5 |
| CodeXGLUE | Sun Yat-Sen University | CodeBERT Baseline |
| CODEGUARD+ | University of Maryland | GPT-4-1106-preview |
| Commit0 | Cohere | Claude 3.5 Sonnet |
| ConvCodeWorld | Seoul National University | GPT-4-0613 |
| SecureAgentBench | Monash University | DeepSeek-V3.1 |
| TDD-Bench Verified | IBM Research | GPT-4o |
| HumanEval-V | Wuhan University | Claude 3.5 Sonnet |
| EvoCodeBench | Peking University | gpt-4 |
| Real-World Project Benchmark | Fudan University | DeepSeek-R1 |
| BaxBench | ETH Zurich | GPT-5 |
| ProjectEval | Harbin Institute of Technology | GPT-4o |
| InterCode | Princeton University | GPT-4 |
| CodeEval-Pro | Tsinghua University | Deepseek-R1 |
| ClassEval-T | Peking University | DeepSeek-V3 |
| RACE | Chinese Academy of Sciences | o1-mini-2024-09-12 |
| BinMetric | University of Science and Technology of China | GPT-4 |
| WebApp1K | ONEKQ Lab | o1-preview |
| Long Code Arena | Delft University of Technology | GPT-4 |
| TESTEVAL | University of Illinois at Urbana-Champaign | GPT-4o |
| CLEVER | California Institute of Technology | DeepSeek-R1 |
| PyBench | Tsinghua University | GPT-4o |
| Mercury | National University of Singapore | deepseek-coder-33b-base + DPO |
| ClassEval | Fudan University | GPT-4 |
| CWEval | Columbia University | gpt-4o-2024-08-06 |
| SWE-Compass | Nanjing University | Claude-Sonnet-4 |
| CodeCriticBench | Alibaba | GPT-OSS-120B |
| CanItEdit | Hugging Face | GPT-4 |
| M2RC-EVAL | University of Waterloo | DeepSeekCoder-6.7B |
| LoCoBench | Salesforce AI Research | Gemini-2.5-Pro |
| CodeCrash | The Chinese University of Hong Kong | DeepSeek-R1 |
| ExecRepoBench | Chinese Academy of Sciences | Qwen2.5-Coder-Instruct-C (7B) |
| SURGE | Tsinghua University | DeepSeek-V3 |
| EffiBench-X | Monash University | Gemini-2.5-Pro |
| DomainCodeBench | Sun Yat-Sen University | DeepSeekCoder-33B |
| CodeIF | National University of Singapore | Claude-3-5-Sonnet-20241022 |
| XCODEEVAL | Nanyang Technological University | CodeLlama-13b-Instruct |
| ParEval | University of Maryland | GPT-3.5 |
| AetherCode | ByteDance | o4-mini-high |
| EvoEval | University of Illinois at Urbana-Champaign | GPT-4-Turbo |
| DevEval | Alibaba Group | gpt-4 |
| SWA-Bench and SWEE-Bench | ETH Zurich | GPT-4O |
| CodeReviewQA | The University of Melbourne | Llama-3.1-70B-Instruct |
| DebugBench | Tsinghua University | gpt-4-0613 |
| CodeMind | University of Illinois at Urbana-Champaign | GPT-4-Turbo |
| ReCode | University of Chicago | CodeGen 16B mono |
| ArtifactsBench | Tencent | GPT-5 |
| ENAMEL | Qualcomm AI Research | HumanEval+ |
| DesignBench | The Chinese University of Hong Kong | GPT-4o |
| SEC-bench | University of Illinois at Urbana-Champaign | Claude 3.7 Sonnet |
| CodeClash | Stanford University | Claude Sonnet 4.5 |
| TestBench | Nanjing University | GPT-4 |
| RepoTransBench | Monash University | GPT-4o |
| DafnyBench | Harvard University | Claude 3 Opus |
| APEval | University of Science and Technology of China | GPT-4o |
| DeepCRCEval | Chinese Academy of Sciences | LLM-Reviewer |
| RustRepoTrans | Sun Yat-Sen University | DeepSeek-R1-0528 |
| CodeEditorBench | University of Waterloo | gemini-ultra |
| REval | Zhejiang University | GPT-4-Turbo (0125) |
| RepoClassBench | Microsoft | Llama3-70b |
| HumanEvalNext | Delft University of Technology | deepseek-coder |
| ProBench | Tianjin University | QwQ-32B-Preview |
| EDIT-Bench | Carnegie Mellon University | claude-sonnet-4 |
| ProjectTest | University of Illinois Chicago | GPT-o1 |
| CodeHaluEval | University of California, Santa Barbara | GPT-4 |
| CodeMMLU | FPT Software AI Center | GPT 4o |
| CodeLMSec | CISPA – Helmholtz Center for Information Security | CodeGen-6B |
| CodeTransOcean | University of Illinois at Urbana-Champaign | ChatGPT (gpt-3.5-turbo) |
| CODESYNC | Zhejiang University | GPT-4o |
| EVALPERF | University of Illinois at Urbana-Champaign | GPT-4 Turbo perf-CoT |
| VERINA | UC Berkeley | o4-mini |
| CRUXEval-X | Chinese Academy of Sciences | GPT-4o |
| CodeSmellEval | William & Mary | Mistral-7B-v0.3 |
| FrontendBench | ByteDance | o3-mini |
| SAFIM | Meta | Qwen2.5-Coder-32B |
| StackEval | Prosus AI | O1 Preview |
| VulDetectBench | Fudan University | ERNIE 4.0 |
| R2C2-Bench | Alibaba Group | DeepSeekCoder-6.7B (R2C2-Tuned) |
| CoderEval | Peking University | PanGu-Coder (300M) |
| CodeAssistBench | Amazon Web Service | ChatGPT 4.1 Mini |
| ICPC-Eval | Renmin University of China | o3-mini High |
| McEval | University of Waterloo | GPT-4o |
| SeCodePLT | University of Illinois at Urbana-Champaign | Claude-3.7-Sonnet |
| COFFE | Zhejiang University | Llama3.1 |
| ComplexCodeEval | Monash University | CodeLLaMa 34B |
| Stateful SWE | Carnegie Mellon University | Claude Sonnet 4 |
| DI-BENCH | Wuhan University | GPT-4o |
| GitChameleon 2.0 | Université de Montréal | Gemini 2.5 Pro |
| CoRe | Purdue University | GPT o3 |
| DSDBench | Tsinghua University | Qwen2.5-72B-Instruct |
| HumanEvalComm | University of British Columbia | DeepSeek Coder |
| SafeGenBench | ByteDance | o3 |
| SyncBench | University of Illinois at Urbana-Champaign | Claude-3.5-Sonnet |
| EditEval | National University of Singapore | ChatGPT (gpt-3.5-turbo-0613) |
| HackerRank-ASTRA | HackerRank | o1 |
| M2-EVAL | Beihang University | GPT-4o |
| DyCodeEval | Columbia University | Llama-3.1-8B-Instruct |
| CodeJudge-Eval | University of California, Santa Barbara | Claude-3.5-Sonnet |
| SR-Eval | Sichuan University | QW3-235 |
| FreshBrew | Gemini 2.5 Flash | |
| CodeArena | Monash University | DeepSeek-Coder |
| DSCodeBench | Chinese Academy of Sciences | GPT-4o |
| CodeMirage | University of Surrey | GPT-4 |
| CodePrefBench | University of Illinois at Urbana-Champaign | GPT-4o |
| ManyIFEval | the University of Tokyo | o3-mini (high) |
| SWE-Effi | The Chinese University of Hong Kong, Shenzhen | Qwen3-32B |
| Mostly Hard Python Problems | South China University of Technology | o1-preview |
| MDEVAL | Beihang University | o1-mini |
| VulnLLMEval | University of North Carolina at Charlotte | Llama3.1-8b |
| DafnyComp | HKUST | CLAUDE-3.5-SONNET |
| SWR-Bench | Northwestern Polytechnical University | Gemini-2.5-Pro |
| DependEval | Beihang University | DeepSeek-V3 (37/671B) |
| USEbench | National University of Singapore | USEagent |
| HLCE | City University of Hong Kong | o4-mini (High) |
| Scoring Verifiers | NVIDIA | o3-mini |
| DynaCode | University of Electronic Science and Technology of China | GPT-4o |
| REPOCOD | Purdue University | GPT-4o |
| RACodeBench | Fudan University | GPT-4o-mini |
| ECCO | Carnegie Mellon University | GPT-4o |
| NaturalCodeBench | Tsinghua University | GPT-4 |
| AutoAPIEval | Concordia University | ChatGPT (gpt-3.5-turbo) |
| CodeScope | University of Illinois at Urbana-Champaign | GPT-4 |
| SE Arena | Queens ’ University | GPT-4o |
| CodeSense | Microsoft | Claude 3.5 Sonnet |
| SWE-fficiency | Google DeepMind | Human Expert |
| ExKLoP | University of Manchester | Codestral-22B |
| MBXP | AWS AI Labs | CodeGen-mono 16B |
| HumanEval-NFR | Seoul National University | ARCHCODE (GPT-3.5-Turbo) |
| HumanEvo | Sun Yat-Sen University | GPT-4 |
| Copilot Evaluation Harness | Microsoft | GPT-4 |
| ARCADE | Google Inc. | PACHINCO |
| CoreCodeBench | Shanghai Jiao Tong University | Claude-3.7-Sonnet |
| SwingArena | UCLA | DeepSeek-V3 |
| KGym | Google DeepMind | GPT-4 Turbo |
| TCGBench | Wuhan University | Human |
| CLMEEval | National University of Defense Technology | CodeLlama (7B) |
| FullFront | University of Washington | Claude 3.7 Sonnet |
| RepoMasterEval | ByteDance | DeepSeek-Coder-Base 33B |
| COMPASS | Codility | O4-Mini-High |
| UA-Code-Bench | Odesa Polytechnic National University | OpenAI o4-mini medium |
| CoderUJB | Peking University | GPT-4 |
| NaturalCC | University of New South Wales | cpt-code M |
| SpecEval | Nanjing University | GPT-4 |
| CTF-Code | Fudan University | CTF |
| OBFUSEVAL | National University of Defense Technology | GPT-4-Turbo-0125 |
| OSVBench | Georgia Institute of Technology | Doubao-1.5-pro |
| CLOVER | Salesforce AI Research | GPT-4O |
| BioCoder | Google DeepMind | GPT-4 |
| AICoderEval | University of Toronto | AICoder (Llama-3-8B-Instruct w/ SFT) |
| SWE-MERA | ITMO University | DeepSeek-R1-0528 |
| IFEvalCode | Beihang University | ControlledCoder |
| FeatBench | Nanjing University | GPT-5 |
| LiveOIBench | University of Michigan - Ann Arbor | GPT-5 |
| VeriEquivBench | Shanghai Artificial Intelligence Laboratory | Human |
| GitChameleon | Monash University | Yi-1.5-Chat 34B |
| CodeIF-Bench | Beihang University | Claude-3.5-Sonnet |
| PythonSaga | IIT Gandhinagar | GPT-4 |
| Breakpoint | MIT | o4-mini |
| InteractScience | Shanghai Artificial Intelligence Laboratory | GPT-5 |
| CPP-UT-Bench | University of Wisconsin-Madison | TinyLlama-1.1B-Chat-v1.0 |
| MRG-Bench | Peking University | Claude 3.5 Sonnet |
| VersiCode | Monash University | GPT-4o |
| TransCoder-test-X | Peking University | ExeCoder |
| TestCase-Eval | Northeastern University | Human Expert |
| CodeMixBench | New York University | Phi-4 |
| Exec-CSN | Carnegie Mellon University | GPT-4 |
| CoCo-Bench | Tsinghua University | o1-mini |
| DOMAINEVAL | Chinese Academy of Sciences | GPT-4o-mini |
| Next Edit Prediction | The Chinese University of Hong Kong | Claude 4 Sonnet |
| GitGoodBench | Technical University of Munich | GPT-4o |
| CodeFuse-CR-Bench | Ant Group | Gemini 2.5 Pro |
| FeedbackEval | Sun Yat-Sen University | Claude-3.5 Sonnet |
| IdentityChain | IBM Research | GPT-4 |
| GBCV | Marian University | GPT-4o |
| DRCodePilot | Beihang University | DRCodePilot |
| OOP | Wuhan University | ChatGPT |
| MERA Code | ITMO University | GPT-4o |
| NoFunEval | Microsoft | GPT-4 |
| CODEMENV | South China University of Technology | GPT-4O |
| OEIS Benchmark | Los Alamos National Laboratory | o1-preview |
| FAUN-Eval | ByteDance | GPT-4o |
| JavaBench | Nanjing University | gpt-4o-2024-05-13 |
| TC-Bench | Central South University | Claude4 |
| PseudoEval | HKUST | Qwen32B |
| STEPWISE-CODEX-Bench | ByteDance | openai-o3 |
| LLM SAST Benchmark | Wroclaw University of Science and Technology | GPT-4.1 |
| DebugEval | Northeastern University | DeepSeek-Coder-V2 |
| mHumanEval | George Mason University | Claude-3.5-Opus |
| WebUIBench | Northwestern Polytechnical University | GPT-4o |
| SysMBench | Wuhan University | Qwen3-32B |
| CACP | GPT-4 (gpt-4-1106) | |
| TRACY | Chinese Academy of Sciences | Qwen2.5-Coder-14B-Instruct |
| buggy-HumanEval | Amazon Web Services | CODEGEN-16B-MONO |
| OSS-Bench | National University of Singapore | PHP (baseline) |
| L2CEval | Google DeepMind | gpt-4 |
| ScratchEval | HKUST | Gemini-1.5-Pro |
| PYMIGBENCH | Northeastern University | GPT-4o |
| CodeApex | Shanghai Jiao Tong University | GPT-4 |
| CRQBench | Google DeepMind | GPT-4 |
| TutorCode | University of Melbourne | GPT-4 |
| E2EDevBench | Peking University | Gemini-2.5-Pro |
| ThrowBench | CNRS | Qwen2.5 Coder Instr. |
| BICS | University of Waterloo | GPT-4o |
| MultiCodeIF | Sun Yat-Sen University | Claude-3-7-Sonnet |
| FPBench | Jilin University | DeepSeek-R1 |
| TypyBench | University of Toronto | CLAUDE-3.5-SONNET |
| Turbulence | Imperial College London | GPT-4 (t=0) |
| Code2Bench | Beihang University | Claude-Sonnet-4 |
| RunBugRun | University of Bordeaux | CodeT5 |
| PostcondGen | Nanjing University | Mistral-7B-Instruct |
| QCoder | the University of Tokyo | o3 |
| PromptSE | Nanyang Technological University | Qwen-1.5b |
| MLDebugging | Central South University | DeepSeek-V3 (72B) |
| ReDef | Jeonbuk National University | CodeBERT |
| RepoDebug | Beihang University | Claude 3.5 Sonnet |
| SWEDE | University of Pennsylvania | Claude-SONNET-3-5 |
| Vericoding Benchmark | MIT | claude-opus-4.1 |
| Defects4C | Tianjin University | CodeLlama-Instruct-7B |
| MINICODE | Cornell University | Claude Sonnet 4 |
| Python Programming Puzzles | Microsoft | davinci-codex (Long Prompt) |
| VJBench | York University | Codex |
| OpenCodeEdit | Sun Yat-Sen University | GPT-4 |
| CodeInsight | onepoint | CodeLLAMA 13B |
| RaCGEval | Nota Inc. | Gemma 7B |
| Geospatial Code Generation | INESC-ID | bigcode/starcoder2-7b |
| PRDBench | Shanghai Jiao Tong University | GPT-5 (Minimal) |
| CoverageEval | Microsoft | GPT-4 |
| PerfBench | Microsoft | Claude Sonnet 4 |
| UnLeakedTestBench | National University of Singapore | gemma-3-27b-it |
| HumanEval-Haskell | Delft University of Technology | UniXcoder (Fine-tuned) |
| StudentEval | Northeastern University | StarCoderBase |
| ExeDS | Beihang University | JuPyT5 |
| MT-Sec | University of Maryland, College Park | Claude Opus 4 (Thinking) |
| SolBench | Sun Yat-Sen University | Claude-3.5-Haiku |
| CoQuIR | Mohamed bin Zayed University of Artificial Intelligence | Voyage-code-3 |
| ACEOB | Shandong Normal University | CodeT5-small |
| VerifyThisBench | University of Toronto | o3-mini |
| VUL4C | Huazhong University of Science and Technology | VulnFix |
| RepairBench | KTH Royal Institute of Technology | o4-mini-2025-04-16-high |
| SeqCoBench | University of Edinburgh | qwen2.5-coder-instruct (32B) |
| ReCatcher | Polytechnique Montreal | DeepSeek-Coder-6.7B (Merged) |
| MultiOOP | Wuhan University | GPT-4o mini |
| Code Execution Simulation | University of Illinois Urbana Champaign | Gemini-2.5-Pro |
| SIMCODE | Queens ’ University | GPT-4.1 (FT) |
| Defects4J-Nl2fix | Microsoft | gpt-3.5-turbo |
| X-HumanEval-X | The Chinese University of Hong Kong | DeepSeek-Coder-33B |
| Educational Program Repair Benchmark | Aalto University | starcoder (3B) |
| Build-bench | Tsinghua University | GPT-5 |
| EVALOOP | NC State University | o3-mini-2025-01-31 |
| AppForge | Fudan University | GPT-5-High |
| UniCode | Peking University | o4-mini (high) |
| MacroBench | KAIST | GPT-4o-Mini |
| PECC | Humboldt-Universität zu Berlin | Claude 3 Haiku |
| TREAT | The Chinese University of Hong Kong | GPT-5 |
| JUGE | University of Washington | EVOSUITE |
| LoCaL | University of Virginia | CrystalBLEU |
| SimCopilot | Rice University | o3-mini (high) |
| CCrepairBench | National University of Defense Technology | llama3.3-70B |
| AutoGEEval++ | Wuhan University | o4-mini |
| RegMiner4APR | The University of Melbourne | ChatGPT-4o + Conversation |
| SWE-Sharp-Bench | Microsoft | GPT-5 |
| PACT | Yonsei University | Qwen3 |
| Assertion Messages | University of North Texas | Codestral-22B |
| Diff-XYZ | JetBrains Research | GPT 4.1 |
| BRACE | William & Mary | Seed-Coder-8B-Instruct |
| Code Comprehension Benchmark | IBM Research | Codestral-22B |
| ComErrFix-CGP-v1.1 | Technische Hochschule Ingolstadt | WARP-Full (CodeLlama-70B) |
| CryptoAPI-Bench | University of Wisconsin-Madison | CryptoGuard |