Generative AI Security Risks Every Engineering Team Should Understand

One is the way that such systems process information. The instructions may be located in a user's account, in a document uploaded by the user, in an e-mail, in a website or another linked source. Malicious code may appear as regular text and be difficult to identify using traditional security measures.

The risk increases on systems that access sensitive information or critical systems. An incorrect response can just annoy a user, but one that discloses personal information, uses the incorrect API, modifies a record, or takes an unsafe action can actually do serious harm!

Therefore, security must be taken into account when developing the application. There must be clear controls on what data the system can have access to, what tools it can use, what actions it can take, and when it needs approval from a human.

This guide outlines the primary security threats developers must be aware of when using generative AI for security, their prevalence in applications, and mitigation strategies to minimize these threats.

Why Are Generative AI Systems Vulnerable

Traditional software is based on clear logic. When the system is generative, it can be used to process natural language and generate a distribution of output, rather than a single output, so the engineer can't predict all the outputs from code alone. NIST acknowledges that this is not the same as conventional software vulnerabilities and suggests that risks be managed throughout the lifecycle.

But, natural language gives rise to another input problem. A malicious instruction may be valid text, either directly entered by a user or via a document, email, or webpage. Prompt injection is a serious issue as per OWASP, since the responses may be affected by the instructions in the prompt, the information may be revealed, or the functionality may be altered.

The impact will be determined by the application's access. Relate it to internal data, APIs, or business tools, and a manipulated response can be a leak in data or an unauthorized action. This is the reason why it's not enough to ask if the system can create the wrong output; it's important to ask what it can do when it does.

The use of firewalls, authentication, static analysis, and input validation is still relevant, but not sufficient to cope with context-dependent behavior at runtime due to natural language context. Engineering teams must add more barriers, such as least-privilege access, permission-aware data access, validation of outputs, access to restricted tools, and human approval for high-impact actions. OWASP suggests a number of these controls to minimize the impact of prompt injection.

These shall be the decisions from the architecture in the first sprint. Ensure that the system has an understanding of what it can see, access to, and do before those capabilities go into production without consent from people.

Top Generative AI Security Risks for Software Teams

The primary security issue is that sometimes these systems provide false answers. It's because they're now placed between users, company information, use logic, APIs, and even business actions. The manipulated input or unreliable output can then escape from the chat window and impact another component of the software stack.

The risks engineering teams should consider are:

Prompt Injection Attacks

Prompt Injection occurs when someone inputs instructions into the system that are designed to alter the way the system is supposed to operate. It can be direct, such as including the instruction in the code or indirect by including it in text that the application reads later.

The risk becomes greater when the output generated is fed to another system without validation.

Consider something that can help you translate natural-language questions to PostgreSQL queries. A user might enter:

The bad consequence would produce SQL like this:

Always use a read-only database account for search use cases, permit which operations can be performed on the database, parameterize user-supplied values, validate database queries, and require approval before any write or destructive operation. OWASP also encourages limiting the access and privileges of models, in addition to just telling users to avoid prompt injection attacks.

Sensitive Data Leakage

A hacking attempt can still happen even with a secure database in place. This may occur when the application presents data that the user should not be receiving. A simplified path is represented as follows:

User request - application gets data - limited data goes into context - response includes the data - sent to the unauthorized user.

As an example, when a worker inquires about the salary of another department from an internal assistant. Before it queries the HR document, the search layer has already crossed an important boundary, if it does find the document.

The authorization should be done before the information goes into the context, not after it is responded to. Also take a look at what is being stored in prompts, logs, traces, and third-party services.

Insecure AI-Generated Code

Insecure code is code that is working as intended but is open to vulnerability due to unsafe input, authentication, permissions, dependencies, secrets, or configuration.

If a Python program is generating the query for a database straight from the user's input:

Insecure:

An attacker might be able to alter the SQL statement in an email.

The safer version does parameterized queries:

Secure:

The code generated should then be included in the same code review cycle as human code: testing, code review, static analysis, dependency checking, and secrets detection.

Data Poisoning and Model Manipulation

Data poisoning does not directly attack the UI, but rather goes against what the system believes to be knowledge. An intruder could alter training examples, manipulate knowledge, data, feedback, or documents fed into a knowledge assistant. This application can then access or acquire the altered information and exhibit wrong behavior subsequently.

For example:

When a document is compromised, it is registered as trusted content, fetched when requested later, and gives false context, which influences the response.

That's why it's important to consider data provenance. Control that allows for addition and changing of trusted sources, validation of new content, version history, and re-run of evaluations when important datasets change.

Hallucinations, Model Drift & Content Integrity Risks

A hallucination is a response that seems to make no sense or is made up. Drift is a phenomenon where the system's performance changes over time due to changes in models, data, prompts, user behavior, or other components of the system. Content integrity risk occurs when downstream treats unreliable information as reliable information.

The primary fault of a wrong answer is reliability. When another component acts on it, it becomes a security concern:

Wrong output - trusted by application - action is performed

Infrastructure recommendations, for example, shouldn't immediately result in a change to the production configuration. Make sure that consequential operations are encircled by approval gates and confirm security-sensitive outputs with trustworthy sources.

AI Supply Chain Vulnerabilities

Typically, a production application will rely on a whole ecosystem of models, open-source models, data sources, packages in Python or JavaScript, vector databases, plugins, APIs, containers, and more services.

Any of these components can be a weak link in your application.

A developer, for instance, may install a package that is similar to the name of a library that a coding assistant recommends. Similar risks can be posed by a broken model artifact or third-party integration.

Maintain a dependency list, rely on reliable registries, verify the provenance of models and datasets, identify important versions, examine packages and artifacts, and audit access from third parties.

Shadow AI and Ungoverned Usage

When a company relies on unauthorized generating tools, it's known as "shadow AI." Like, an engineer gives his private source code to a public coding helper, an employee uploads a client spreadsheet for analysis, or someone describes an anonymous contract using his own account. When that time arrives, the company might not have an idea of what information has escaped its environment, how it was handled, or how long it will be stored.

Teams must have approved tools and a set of guidelines on what type of data employees can share. It's usually easier to provide a safe alternative than to expect that an outright prohibition will deter unauthorized use.

Insecure Integrations & Access Control Exploits

Integrations define the distance of propagation of a failure. For instance, an internal assistant requires only to read the support ticket but can be used to create users, edit tickets, export customers, and delete information, thanks to its API credentials. Now, the degraded workflow is far more powerful than the task itself.

The user's authorization should not be determined by the model. Apply only the appropriate privileges to integrations; implement application and related service privileges; and, if available, give user-scoped or task-scoped credentials. This is the least privilege principle for software that is coupled to a model.

Governance, Privacy & Compliance Risks

Problems of governance occur when an organisation is unable to answer fundamental questions relating to the use of these systems in relation to information:

What data are allowed to be submitted by employees? Where in the United States is it made? Is it retained? Is it capable of use by the provider for training? In which areas is it processed? What users can view the logs? What is done when a person asks to be deleted?

These questions are particularly relevant when it comes to personal, financial, healthcare or other regulated information.

NIST's Generative AI Profile suggests that these concerns are addressed in the lifecycle of the system, from system governance to privacy, third-party risk, monitoring, and incident response.

Agentic AI & Autonomous Actions

It's the agents who increase the stakes, as the decisions they make can turn into real actions. It is important to note that there is a significant difference between software that will tell you:

“It is necessary to disable this account.” and software that accesses an API and simply turns off the account.

It extends to sending e-mail messages, permission changes, code execution, refunds, changes to infrastructure, and financial workflows. A wrong tool now, a prompt injection or a bogus conclusion could lead to an operational consequence.

Reversibility is a useful test for a good design:

Read - Recommend - Draft - Execute - Irreversible action

Avoid having steps that are stand-alone. The harder the action is to undo, the more restrictive the permissions, the more parameters to check and the more logging of calls to the tools, the more human approval to add.

How to Build a Generative AI Security Strategy

A good security plan is multi-layered, meaning if one prompt is compromised or output is incorrect, then it does not give direct access to sensitive data or the ability to run a privileged action. NIST also takes a generative-system risk-life cycle view, instead of a single application control.

Always implement Governance Prior to Development

Establish approved models and providers, acceptable data types, security review requirements, and ownership prior to development. If you have autonomous workflows, consider what actions are always going to need human participation.This will provide a distinction between what the system can recommend and what it can actually run.

Secure the Application Layer

Use prompts and retrieved content and generated responses as untrusted input. Validate input, separate external content from system instructions, and validate or sanitize output before sending it to a database, browser, API, shells, or other applications.

The same code review, code testing, SAST, dependency scanning, and secrets checks should be performed on generated code as well as human-written code. According to OWASP, if output isn't regulated, the model may be susceptible to SQL injection, XSS, SSRF, privilege escalation, and code execution.

Secure the Data Layer

Label sensitive data prior to using it in a model. Implement access controls, if needed, and anonymize or remove personal information if it is not needed for the task.

Above all, apply permissions prior to retrieval. If the user cannot access a document directly, he or she should not be able to gain access by having an assistant retrieve or summarize the document.

Limit Infrastructure and Integration Access

Protect APIs using authentication, authorization, rate limits, and scoped credentials. Only give functions and/or permissions needed for the job to the integration.

An assistant who can only read customer records, for instance, shouldn't have access to the credentials a customer may use to update and/or delete customer records. Overstated functionality, permissions, and autonomy are some of the primary causes of harmful agent actions.

Monitor Runtime Behavior

Security testing is not only needed prior to deployment but also necessary ongoing. Make model requests, fetch model data, call models, authenticate, deny access to models, modify model configuration, and take high-impact actions.

Those logs will be used to identify any activity that is unusual, policy violations, use of any tools not authorized, and volumes of requests that are abnormal. Throughout the system lifecycle, NIST's Generative AI Profile helps with ongoing measurement and risk management.

These controls should be implemented in the first development sprint - not after launch. The architecture should have the model as if it can be tweaked or it can make a mistake, while deterministic controls determine what it can see, what it can access, and what it can ultimately do.

Generative AI Security Checklist for Development Teams

Use this checklist for release gates, architectural reviews, and sprint planning. The objective is to detect the risky design decisions at an early stage and before they become production constraints.

Pre-Build

1. Identify the use case and the level of risk (information, record modification, execution of code, messages, transactions).

2. Prior to adding prompts, logs, or third-party services, categorize the data they will come into contact with as belonging to customers, source code, credentials, financial records, or PII.

3. Approve models, providers, and data flows: Make sure that a record is kept of the services that can process company information, where this data is stored, and if it can be kept or used for training.

4. Define the human-approval boundary: Determine which activities, especially those that have a major impact or are irreversible, should never be carried out without human assistance.

5. Do not only test the threat model of the chat interface, but document all the steps like prompts, retrieved documents, APIs, tools, model providers, databases, and downstream applications.

During Build

6. Test external content for direct and indirect prompt injection of test prompts, uploaded files, webpages, emails, repository content, and retrieved documents as untrusted external content. OWASP suggests the following measures: external content separation and limiting the effects of altered instructions.

7. Before using the output of any process, validate it. Avoid directly injecting model output without first doing schema verification, sanitization, or parameterization into SQL, HTML, shell commands, file paths, or APIs. In the list of the possible consequences of unsafe output handling, OWASP documents SQL injection, XSS, SSRF, privilege escalation, and code execution.

8. Run generated code through the normal security pipeline: Security code review, tests, SAST, dependency scanning, and secrets detection prior to merging and deployment of generated changes. If there is any possibility of coding agents being able to read repository files, issues, PR comments, and generated documentation should also be considered untrusted input.

9. Make all API and tool connections more robust. Apply authentication, scoped credentials, rate limits, timeouts and user-level authorization. Workflows with a read-only permission level should not have the ability to make changes or delete.

10. Add cases for using the tool incorrectly, not just how it is expected to be used.

Post-Launch

11. Track system operation: model calls, retrieval activity, tool run, unauthorized access, risky actions, and system configuration changes and log sensitive information.

12. Be alert to repeated blocked prompts, unexpected tools, abnormal data access, increasing volume of requests, rising inference cost and sudden drop in request output quality.

13. Audit permissions periodically, delete unused tools, expired credentials, too many toolboxes and removed integrations from tests. The unnecessary functionality, permissions, and autonomy are OWASP's major reasons for damaging agent behavior.

14. Convert actual failures to regression tests. All confirmed security issues should be represented as a repeatable test for the next time it occurs after a prompt, model, integration or configuration change.

15. Learn how to securely roll back, identify response ownership, isolate a compromised component, deactivate a tool, revoke credentials, and preserve evidence.

Generative AI Security Is a Development Problem, Not Just a CISO Problem

The engineering decision is the initial stage in the security process. The first step in security is the engineering decision. It is up to developers to decide which data can be accessed, which APIs and tools can be used, which permissions will be given, and what output or actions will need to be validated.

The policies set by security teams are useless if they cannot be supported with the right architecture, access controls, secure integrations, testing and monitoring. Embedding security architecture at the outset of development also ensures privacy and compliance obligations can be met with less effort and expense, before they become too costly to modify.

CodingCrafts enables companies to create production-ready AI solutions that are secure, from start to finish, throughout the application, data, integrations, and deployment.

Looking to develop a safe AI product? Discuss the possibility of moving from Architecture to Production with CodingCrafts.