What Makes a Good LLM Agent for Real-world Penetration Testing?
A 2026 study of real-world LLM penetration testing agents and Excalibur, a difficulty-aware agent for end-to-end attack planning.
Feb 19, 2026
Excalibur
Excalibur is a difficulty-aware LLM agent design for real-world penetration testing. It couples typed tooling, retrieval-augmented security knowledge, task difficulty assessment, and evidence-guided attack tree search to reduce planning failures in multi-step penetration testing tasks.
Feb 19, 2026
PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing
An LLM-empowered automated penetration testing framework that leverages domain knowledge inherent in LLMs, achieving 228.6% task completion improvement over baseline GPT models.
Aug 14, 2024
PentestGPT
An LLM-empowered automatic penetration testing framework with 14k+ GitHub stars and 2.4k+ forks. PentestGPT is designed to automate penetration testing by leveraging the domain knowledge inherent in Large Language Models. It features a three-module architecture (Reasoning, Generation, and Parsing) that emulates human penetration testing workflows. Key Features: Multi-module agent design for reasoning, generation, and parsing Integration with multiple LLM backends and real-world security workflows Evaluation on CTF challenges and practical penetration testing targets 228.6% task completion improvement over baseline GPT models Recognition: Distinguished Artifact Award at USENIX Security 2024 Widely used open-source security research artifact with active community adoption
Aug 1, 2023