Frozen VLM
Generate normally and retain the original policy on safe trajectories.
RESEARCH PROJECT · 2026
SafeRI
Recognition and Intervention for Token-Level
Safety Intervention in Large Vision Language Models
1 WHU · 2 UTokyo · 3 JD · 4 PKU · 5 Shanghai AI Lab · 6 SII · 7 SJTU · 8 HKU
* Equal contribution † Co-corresponding authors ‡ Project leader: Wenqi Shao
ABSTRACT
Safety alignment should be an on-demand intervention, not a permanent modification to every decoding trajectory.
Existing vision-language model safety methods typically modify behavior globally, perturbing safe and already-safe generations alongside risky ones. SafeRI monitors the evolving pre-token generation state with a lightweight recognizer. When unsafe drift is predicted, it activates a boundary-aligned LoRA module to redirect the following continuation; otherwise, the frozen backbone proceeds unchanged. Across safety and general-purpose benchmarks, SafeRI mitigates residual risk while preserving multimodal utility.
01 / METHOD
Recognize risk.
Intervene only then.
Recognizer
Read the current prefix state and estimate imminent unsafe drift.
Intervention
Open a renewable intervention window and steer the continuation safe.
Figure 1. SafeRI performs streaming recognition at every decoding step. The safety adapter remains dormant until risk emerges.
Token-level
The gate is updated from the current hidden state for the next decoding step, making intervention causal rather than retroactive.
Selective
Once risk subsides, the gate closes and generation returns to the frozen-backbone policy.
02 / RESULTS
Safety gains,
utility retained.
SafeRI improves the safety average on all evaluated backbones while keeping general multimodal capability close to the frozen base model.
| Backbone | Setting | Safety Avg. ↑ | Change |
|---|---|---|---|
| Qwen3.5-9B | Base | 86.56 | - |
| Qwen3.5-9B | SafeRI | 88.88 | +2.32 |
| Qwen3.5-4B | Base | 87.57 | - |
| Qwen3.5-4B | SafeRI | 88.28 | +0.71 |
| Qwen3.5-2B | Base | 86.54 | - |
| Qwen3.5-2B | SafeRI | 87.70 | +1.16 |
| Llama3.2-Vision-11B | Base | 83.49 | - |
| Llama3.2-Vision-11B | SafeRI | 84.27 | +0.78 |
Safety Avg. is the arithmetic mean over SPA-VL-test harm, AdvBench, HADES, XSTest, and MSSBench. Higher is safer.
SPA-VLAdvBenchHADESXSTestMSSBenchMMBenchMM-VetBLINK
03 / CONTRIBUTIONS
01We formulate post-alignment VLM safety hardening as a selective trajectory-control problem.
02We introduce streaming hidden-state recognition paired with a boundary-aligned LoRA intervention.
03We show that selective activation can reduce residual safety risk without globally shifting safe behavior.
CITATION
Paper and resources
coming soon.
Paper, code, and complete citation information will be added here upon release.
Follow the project →@misc{saferi2026,
title = {SafeRI: Recognition and Intervention for
Token-Level Safety Intervention in Large
Vision Language Models},
author = {Ma, Caoyuan and Gu, Tian and Liuwenpu and Xie, Weichu and Dong, Shuai and Xu, Yuqi and Zhao, Ji and Wang, Ziyue and Chang, Wenzheng and Wu, Taiqiang and Zhu, Yongfu and Shao, Wenqi and Wang, Zheng and Zheng, Yinqiang},
year = {2026}
}