FEAT Integration Request: Jailbreak Template Collection for Enhanced Red Teaming.

[Arth-Singh]

Hello PyRIT team,

I would like to propose integrating my comprehensive collection of jailbreak templates as an extension to the existing attack templates in pyrit/datasets/jailbreak/templates/.

Background & Motivation

Through my research while being an AI Red Teamer, I've observed that the current jailbreak templates in PyRIT, while foundational, may not adequately challenge modern LLM safety measures and are bit outdated. This presents a gap in comprehensive red teaming capabilities where attack templates becomes the Heart of PyRIT.

Proposed Contribution

I have developed around 80+ jailbreak templates and still continue to develop attacks based on latest techniques, available at: https://github.com/Arth-Singh/Arth-Jailbreak-Templates ; I have also taken few of the existing attacks in pyrit and went ahead to enhance them.

Technical Details

• Format: All templates follow PyRIT-compatible YAML structure with standardized metadata

Preliminary Validation

Initial testing on GPT-4o has shown promising results, with several templates successfully eliciting responses to sensitive queries (e.g., illicit substance synthesis) that standard approaches fail to achieve.

Quantitative Evaluation Offer

If empirical validation data would support this integration, I am prepared to conduct comprehensive quantitative analysis over the weekend, including success rate measurements across multiple models.

I believe this contribution would significantly enhance PyRIT's red team capabilities and support more robust AI safety testing. I'm happy to discuss implementation details, provide additional documentation, or conduct any required validation studies.

Thank you for considering this contribution to the PyRIT project.

Best regards,
Arth Singh
LinkedIn: https://www.linkedin.com/in/arthsingh7in/

[Arth-Singh]

Currently the attacks has a prefix of (new_) in my repo, If there is a need of removing that for integration. I can surely do that.

[romanlutz]

Hi @Arth-Singh , thanks for opening the issue to discuss further 🙂 I think that's a fabulous idea.

Your YAML files already have everything in terms of structure that we'd need for PyRIT. You already have parameters, data type, source, authors, description, name, and value. I have only inspected a couple to make sure. When you open the PR I'll read all of them in detail because that will allow me to comment if necessary.

Thanks a ton for the contribution! Also, obviously feel free to update or add more. We're very much interested in contributions like this (or for other attacks) as discussed on LI.

[Arth-Singh]

Haha thanks @romanlutz ; I had edited the code to make sure it adapts the pyrit environment.

[romanlutz]

That explains a lot 🤣

[Arth-Singh]

Created a PR here ; #1254