OpenAI Identifies Flaws in SWE-bench Coding Benchmark

OpenAI has revealed significant issues with the SWE-bench Verified coding benchmark, which is widely used to evaluate AI models. According to NS3.AI, the benchmark's reliability is compromised due to task contamination and training data leakage, allowing models to memorize solutions rather than genuinely solving tasks. OpenAI recommends transitioning to the more robust SWE-bench Pro and is working on developing new private evaluation methods to better assess AI coding capabilities.