Flaw And Order: Finding The Needle In The Haystack Of CodeQL Using LLMs
Running CodeQL's built-in queries on Redis gave me over 6,800 potential issues. Doable, maybe. But when I tried FFmpeg, I got over 51,000. That's way too much for me. And how many of those are real vulnerabilities? Probably around 0.01%. The sheer number of false positives makes static code analysis impractical - who wants to manually sift through tens of thousands of results just to find a few actual security flaws?
To fix this, we built an open-source tool that fuses CodeQL with an LLM-driven agent. This agent autonomously navigates the code, running targeted queries to extract only the relevant context. On top of that, we introduced Guided Questioning, an advanced reasoning technique that keeps the LLM focused, improving accuracy even for complex vulnerabilities.
Using this approach, we reduced false positives by up to 97% and uncovered more than a dozen real-world security issues in Linux, Apache, FFmpeg, Bullet3, Libvips, libretro, Linenoise, and other widely used open-source projects.
Speakers
Simcha Kosman
Senior Security Researcher, CyberArk
Simcha Kosman is a Senior Security Researcher at CyberArk Labs with over seven years of experience in vulnerability research. He discovered his first vulnerability at the age of 15 and earned a reward for it. Since then, Simcha has uncovered security flaws across a wide range of targets, including processors, embedded systems, and large-scale open-source projects, during his time at Rockwell Automation and Intel. His current work focuses on developing novel methods for vulnerability detection by combining static analysis, AI, and automation. He is especially passionate about using techniques like fuzzing, LLM-guided reasoning, prompt injection security, MCP security, and jailbreak research to push the boundaries of modern security.
