2025-10-21 Prompting the Priorities: A First Look at Evaluating LLMs for Vulnerability Triage and Prioritization

[ahouseholder]

Prompting the Priorities: A First Look at Evaluating
LLMs for Vulnerability Triage and Prioritization
Osama Al Haddad∗, Muhammad Ikram∗, Ejaz Ahmed†, and Young Lee∗
https://arxiv.org/pdf/2510.18508

Abstract - Security analysts are under growing pressure
to assess increasingly complex and high-volume vulnerabili-
ties. Recent advancements have shown that Large Language
Models (LLMs) offer a promising solution by automating key
aspects of this process through advanced natural language
understanding. In this study, we evaluated four state-of-the-
art LLMs—ChatGPT, Claude, Gemini, and DeepSeek—across
twelve prompting techniques (PT) to assess their effectiveness
in interpreting semi-structured and unstructured vulnerability
information. As a practical use case, we investigate each model’s
ability to predict decision points within the Stakeholder-Specific
Vulnerability Categorization (SSVC) framework, a widely used
method for guiding response actions. Specifically, we focus on
the four core SSVC decision points: Exploitation, Automatable,
Technical Impact, and Mission & Wellbeing.
We leveraged a sample of 384 real-world vulnerabilities from
the VulZoo dataset to evaluate the performance of each LLM
across twelve PTs, including One-shot, Few-shot, and Chain-of-
Thought. Over 165,000 LLM queries were issued to assess their
predictive accuracy. Model responses were evaluated using F1-
scores for each SSVC decision point (SDP), and weighted and
unweighted Cohen’s Kappa metrics for SSVC decision outcomes
(SDO). Our results show that Gemini consistently outperformed
the other models, demonstrating superior performance on three
out of four SDPs and yielding the highest number of correct
recommendations. PTs that included exemplars generally led to
improved accuracy, although all models exhibited difficulty with
certain SDPs. Notably, only DeepSeek achieved fair agreement
under weighted metrics, and all models showed a tendency to
over-predict risk levels.
Our results suggest that while LLMs cannot yet replace expert
judgment, certain LLM-PT combinations demonstrate moderate
efficacy in specific decision areas in the SSVC vulnerability
framework. When used appropriately, LLMs can serve as ef-
fective aids in vulnerability prioritization workflows, helping
security analysts respond more efficiently to emerging threats.

(emphasis added)