AI-driven spear phishing attacks
Generative models can be misused to target individual users more efficiently by using personalized information [23]. Highly convincing automated fraudulent schemes can exploit the trust of victims by extracting sensitive data and making the deception more likely to succeed. For example, in LLMs, this misuse can be aided by jailbreaking techniques [178].
ENTITY
1 - Human
INTENT
1 - Intentional
TIMING
2 - Post-deployment
Risk ID
mit1193
Domain lineage
4. Malicious Actors & Misuse
4.3 > Fraud, scams, and targeted manipulation
Mitigation strategy
1. Deploy advanced, AI-native email security solutions that utilize behavioral analytics, communication pattern analysis, and real-time multimodal detection (text, voice, image) to identify hyper-personalized, tone-matched, and zero-hour phishing attacks that evade traditional keyword and reputation-based filters. Furthermore, mandate Multi-Factor Authentication (MFA) across all enterprise accounts to contain unauthorized access even if credentials are compromised. 2. Institute mandatory and continuous security awareness training for all personnel, emphasizing recognition of advanced AI-enabled social engineering tactics, including deepfake video/voice, subtle tone mismatches, and urgent, context-aware requests. This training must include real-world phishing simulations and establish a clear, non-punitive incident reporting framework. 3. Implement technical defenses at the Large Language Model (LLM) layer to mitigate jailbreaking and prompt injection, such as layered guardrails, input/output validation and sanitization, and adversarial techniques like 'LLM salting' to disrupt the efficacy and transferability of prompts designed to generate malicious or restricted content for attackers.