AI agents often rely on labels to separate users, tools, webpages, and internal reasoning, but models may still infer authority from the text itself rather than the role tag. This role confusion can help explain why prompt injection works, because malicious content inside tool outputs may be treated like legitimate instructions by the model. #ChatGPT #Claude #promptinjection
Keypoints
- AI conversations are organized by role labels such as user, assistant, and tool.
- Models may not fully trust those labels and can respond to how text sounds.
- Malicious instructions hidden in webpage or tool output can influence the agent.
- Prompt injection remains a problem because role boundaries can be confused internally.
- New probes can measure how strongly models represent text as different roles.
Read More: https://www.toxsec.com/p/cot-forgery-prompt-injection