SoK: On the Offensive Potential of AI

Summary

This paper presents a comprehensive systematization of knowledge (SoK) on the offensive potential of Artificial Intelligence (AI). The authors analyze 95 research papers, 38 InfoSec briefings, and survey 549 non-experts to provide a holistic understanding of the threats posed by AI.

Highlights

• AI can be used for malicious purposes, including cyberattacks and social engineering.
• The use of AI in offensive security is becoming increasingly prevalent.
• Non-experts are concerned about the potential misuse of AI, but lack a clear understanding of the concept.
• Experts identify several open problems in the field of offensive AI, including the need for better detection and mitigation techniques.
• The development of AI-powered countermeasures is crucial to defend against AI-driven attacks.
• The societal impact of offensive AI is significant, with potential consequences for democracy, economy, and human relationships.

Key Insights

• The offensive potential of AI is vast and can be used to launch sophisticated cyberattacks, including phishing and social engineering attacks.
• The use of AI in offensive security is not limited to technical attacks, but also includes social and psychological manipulation.
• Non-experts are concerned about the potential misuse of AI, but lack a clear understanding of the concept, highlighting the need for awareness and education.
• Experts emphasize the need for better detection and mitigation techniques to defend against AI-driven attacks.
• The development of AI-powered countermeasures is crucial to defend against AI-driven attacks, but also raises concerns about the potential for AI-powered countermeasures to be used for malicious purposes.
• The societal impact of offensive AI is significant, with potential consequences for democracy, economy, and human relationships, highlighting the need for a multidisciplinary approach to address the challenges posed by offensive AI.
• The use of AI in offensive security raises important ethical and moral questions, including the potential for AI to be used to manipulate and deceive individuals.

Mindmap

Citation

Schröer, S. L., Apruzzese, G., Human, S., Laskov, P., Anderson, H. S., Bernroider, E. W. N., Fass, A., Nassi, B., Rimmer, V., Roli, F., Salam, S., Shen, A., Sunyaev, A., Wadwha-Brown, T., Wagner, I., & Wang, G. (2024). SoK: On the Offensive Potential of AI (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2412.18442