Where have we seen untrusted data containing code executed by a software system?

we show that augmenting LLMs with retrieval and API calling capabilities (so-called Application-Integrated LLMs) induces a whole new set of attack vectors. These LLMs might process poisoned content retrieved from the Web that contains malicious prompts pre-injected and selected by adversaries. We demonstrate that an attacker can indirectly perform such PI attacks. Based on this key insight, we systematically analyze the resulting threat landscape of Application-Integrated LLMs and discuss a variety of new attack vectors.

SQL injection and Cross-Site Scripting (XSS) are both vulnerability classes where untrusted user input containing code is executed in a context beneficial to an intruder. This paper expands the active prompt injection field. It demonstrates how snippets of data from 3rd party sources can be embedded in an AI prompt and effectively hijack execution to impact other users.