GenAI Security Fundamentals
GenAI introduces new risks beyond traditional application security, because a model treats everything in its context as text it might act on.
Overview
Traditional application security has a clear line: user input is untrusted, application code is trusted. GenAI blurs that line, because a model can’t reliably distinguish an instruction from data — both are just text in its context window.
Key Idea
This lesson is the entry point to the AI Security & Governance category — it frames the risk landscape that Jailbreaks, RAG Security, Agent Security, and the other lessons here each address one piece of.
Where It Fits
Untrusted Input
Model Context
Instructions and data blur together.
Model Output / Actions
Key Points
- Text-as-instructions problem
- A model can’t inherently tell a legitimate instruction from adversarial text embedded in retrieved content or user input.
- Expanded attack surface
- Every new capability — retrieval, tools, memory — adds a new place untrusted content can influence model behavior.
- Not solved by one control
- GenAI security is a category of related risks, each with its own mitigation, not one setting to turn on.
Interview Question
Why is GenAI security considered fundamentally different from traditional application security?
Traditional security assumes a clear boundary between trusted application logic and untrusted user input. A language model treats everything in its context — user input, retrieved documents, tool results — as just text, without a reliable way to distinguish an instruction from data, which opens risk surfaces like prompt injection that don’t map cleanly onto traditional input validation.
Explain It in 30 Seconds
GenAI security starts from the fact that a model can’t reliably separate instructions from data in its context — every new capability, like retrieval or tools, expands where untrusted content can influence behavior, which is why it needs a category of distinct mitigations rather than one control.