Only GPT-OSS-120b is perfectly reliable in both directions (in our 3 re-runs of each setup). Most models that find the bug also false-positive on the fix, fabricating arguments about signed-integer bypasses that are technically wrong.
这一结果揭示了AI模型在识别已修复代码方面的局限性，许多模型虽然能检测漏洞，但错误地将已修复代码标记为仍有问题。这强调了在AI安全系统中需要额外的验证和人工审核层，以确保结果的准确性和可靠性。
Rocca, R., & Yarkoni, T. (2020). Putting psychology to the test: Rethinking model evaluation through benchmarking and prediction. PsyArXiv. https://doi.org/10.31234/osf.io/e437b
Grimm, V., Johnston, A. S. A., Thulke, H.-H., Forbes, V. E., & Thorbek, P. (2020). Three questions to ask before using model outputs for decision support. Nature Communications, 11(1), 4959. https://doi.org/10.1038/s41467-020-17785-2
Edelsbrunner, P. A., & Thurn, C. (2020, April 22). Improving the Utility of Non-Significant Results for Educational Research. https://doi.org/10.31234/osf.io/j93a2