🌐 Project Page | 🤗 Datasets & Models | 📄 arXiv Paper
Official repository for "KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards" (NeurIPS 2026).
Authors: Pengfei Li1,∗, Naufal Suryanto1,∗, Sicheng Zhang1, Muzammal Naseer1,2
1 Khalifa University · 2 University of Western Australia · ∗ Equal contribution
KaliBench evaluates natural-language-to-command translation on Kali Linux, with 8,504 verified query-command pairs spanning 1,642 sub-tools and 23 tool dimensions.
We release RedSage-K models and reproducible training pipelines for supervised fine-tuning (SFT) and reinforcement learning with runtime-free verifiable rewards (GRPO/RLVR).
| I want to… | Go to |
|---|---|
| Try a released model | Quick inference |
| Evaluate a model on KaliBench | Evaluation guide |
| Download the data or models | Dataset · Released models |
| Train models with SFT or GRPO/RLVR | Training guide |
| Construct data or inspect its format | Data construction · Reference |
Models are evaluated under three tool-knowledge settings:
| Mode | Information available to the model |
|---|---|
unrestricted |
The query only; the model selects the tool. |
restricted |
The query and a candidate set of allowed tools. |
hinted |
The query, the target tool, and its usage documentation. |
| Released data | Rows | Purpose |
|---|---|---|
| Training split | 3,504 | SFT and GRPO/RLVR |
| Test split | 5,000 | Held-out evaluation |
| Tool documentation | 2,809 | Sub-tool names and usage information |
See the dataset reference for schemas, examples, and coverage.
All three variants start from RedSage-Qwen3-8B-Ins. See the training guide to reproduce them.
| Model | Training |
|---|---|
| RedSage-K-SFT | Supervised fine-tuning (SFT) |
| RedSage-K-GRPO | GRPO with verifiable rewards |
| RedSage-K-SFT-GRPO | SFT followed by GRPO |
Try our best-performing released model, RedSage-K-SFT-GRPO, with Python 3.10 from the repository root:
pip install torch transformers accelerate
python demo/grpo_inference.py \
--query "List the network interfaces using ifconfig."See the inference guide for other model variants, local checkpoints, and generation options. For benchmark setup and scoring, follow the evaluation guide.
If you use KaliBench in your research, please cite:
@inproceedings{li2026kalibench,
title={KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards},
author={Pengfei Li and Naufal Suryanto and Sicheng Zhang and Muzammal Naseer},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems Evaluations and Datasets Track},
year={2026},
url={https://github.com/RISys-Lab/KaliBench}
}