Welcome to the Party (Data Scientists)
In Part 1, we briefly discussed classes of vulnerabilities unique to AI systems: adversarial attacks, data poisoning, model inversion, transfer learning robustness, and alignment errors. There is certainly an ever-expanding number of AI vulnerability classes, but that topic is thoroughly covered elsewhere [ref: MITRE ATLAS, Microsoft Security Bug Bar, etc.].
So now we get to the elephant in the room: What do we do about it? More specifically, how do we manage the security of AI systems in the context of FISMA, FedRAMP, and DoD Cloud SRG authorizations? The quick answer: Your data scientists now have operational cybersecurity responsibilities, and there’s no one else who can do it.
Many of the vulnerabilities described in Part 1 require an interpretation of the data being ingested into the AI system and the data being produced by it. In the context of FISMA/FedRAMP/DoD cSRG, that data is Federal or Mission data and it lies at the heart of what those programs are designed to protect. The arbitrary nature of prompts to Large Language Models (LLMs) and the inherent unpredictability of their outputs means that an engineer who understands the purpose and design of the system must serve as the oracle who judges the efficacy and appropriateness of the system’s performance. The AI oversight role must perform his or her duties within the boundary as they will be handling Federal/Mission data. They must handle this data within the context of a security need-to-know.
Conventional tools to identify and protect against sensitive data leakage are insufficient. Consider, for example, the “poem” attack described in Part 1 wherein Personally Identifiable Information (PII) was unintentionally disclosed to an unauthorized third party. Typical Data Loss Prevention (DLP) tools are designed to classify data tokens such as credit card numbers, email addresses, phone numbers, Social Security numbers, and so on. Applying DLP tools to the data leakage problem in LLMs is akin to building a taller fence to ward off a drone swarm. Why couldn’t an attacker simply use the same “poem” prompt with the additional condition that each character in the response be delimited by a carat character, or the word “l33th4x0r,” or even the Fibonacci sequence of integers mod 31337? DLP tools cannot identify sensitive data tokens under arbitrary encodings.
The in-boundary operational security data scientists must be actively and regularly reviewing prompts and responses while thinking like an attacker. As data scientists, they are capable of quickly spotting misbehaving AI, collecting sufficient data samples, and re-training the AI as needed. These data samples cannot be exported from the authorization boundary for further analysis and model correction as it would violate the very definition of what a boundary is. Furthermore, these individuals must remain current with public threat advisories and be on the lookout for “zero day” prompts, potentially available to nation-state purchasers of exploits.
AI service providers seeking or maintaining Federal authorization must consider that those responsible for creating the AI can no longer sit on the sidelines alongside Dev and QA teams. The tremendous power of AI naturally translates to a tremendous responsibility to ensure it, and the data it represents, remains secure.
Need help keeping your data secure? Get in touch with a 38North security expert today.
