AI Application Security: What Engineering Teams Need to Get Right
A customer uploads a document to an artificial intelligence-based application. One of those documents has a secret instruction inside that changes the behavior of the system. Such an instruction might lead to the application calling the wrong API, or to the disclosure of sensitive information (such as address, credit card number, social security number, etc.), or to the execution of an action that the user did not request.
That's where the complexity of applying AI to security arises. These applications are seldom used separately. They integrate models with company data, databases, APIs, SaaS tools, and business workflows. If one of these connections is flawed, a bad input could have an impact on other areas of the system.
When agents can do more than just answer questions, the risk escalates. If an application is able to send an e-mail message, update a customer record, run code, or trigger a workflow, it requires significantly more permissions and tool access control.
So it's not sufficient to just have the model. Engineering teams must ensure the security of data, application logic, integrations, credentials, tool calls, and run-time behavior around the data.
This guide clarifies where the security risks are in real-life AI applications, what controls are important at each stage, and how the engineering team can avoid a bad model decision turning into a production security incident.
What Is AI Application Security?
App security for artificial intelligence is about more than the protection of the model; it is the protection of the entire application. It includes information about how the user interacts with the system, what information it has access to, which APIs and tools it can use, how permissions are enforced, and what output is generated.
For instance, an assistant on the inside might be able to use a language model to respond to questions, fetch records from a knowledge base, and interface with a CRM. Getting that application means ensuring that users are able to get only the records they have access to, that the external content can't easily control the behavior of the tool, that generated output is checked before it is used by another system, and that connected tools have only the permissions they need.
It's more comprehensive than AI model security, which more directly targets things like model artifacts, training and fine-tuning data, model behavior and inference-layer attacks. Application-level protection concerns the interaction of the model with code, data, identities, APIs, tools, users and business systems.
Whether the model generates a poor response or follows a misguided instruction, the application around it should not let that model access any data that it shouldn't or take an action that it shouldn't perform.
Why AI Applications Create New Security Risks
Although traditional security measures are important, model-powered applications bring new avenues for attacks. They receive less predictable input, access to more data sources, and might have the ability to interact with other systems. This is an attack surface that goes beyond regular app code.
AI-Generated Code Can Introduce Vulnerabilities
Regretfully, even if generated code is valid, it may still be insecure. Examples of such issues are insecure database queries, sensitive credentials that are hard-coded, poor access control, sensitive defaults, and legacy or unconfirmed dependencies. OWASP’s cheat sheet suggests that code generated be treated as "production code" and be checked as per the standard security pipeline.
A coding assistant could write a query into a database such as:
# Vulnerable: user input is added directly to the SQL query
query = "SELECT * FROM users WHERE email = '" + user_email + "'"
cursor.execute(query)In this case the attacker can inject an SQL statement in the user_email variable, causing an SQL injection vulnerability.
A safer version will take the input and split it from the SQL command:
# Safer: parameterized query
query = "SELECT * FROM users WHERE email = %s"
cursor.execute(query, (user_email,))These examples illustrate various problems such as hard-coded secrets, command injection risk, and lack of access control.
The engineering rule is simple: generated code cannot be trusted unless it has been examined and tested. It's the only way to get it through code review, SAST, dependency and secret scanning, and security testing before it makes it into production. The more restrictive controls will be needed for coding agents that install packages, execute commands, and/or modify CI/CD files since a bad generation can impact multiple source files.
AI Applications Expand the Data Attack Surface
These programs can manipulate data from files, emails, web pages, databases, chat, and retrieved documents. Sensitive data and dangerous content might enter the workflow from any source.
For instance, there might be instructions in a retrieved document that affect the model. This is considered indirect prompt injection by OWASP. The document may be considered to be reliable for information, but instructions within the document should not be automatically expected to be reliable. Data origin, access permissions, and trust boundaries are all considered application security.
AI Agents Can Take Actions
The threat becomes quite different if a system is able to produce more than just text. An agent can come to assist you with:
Call APIs
Query databases
Send emails
Modify records
Execute code
Access SaaS platforms
Trigger business workflows
From there, it can be a true operation if the decision is bad. OWASP calls this a needless agency risk as systems are granted too much functionality, permissions, or autonomy. It provides a simple example: a database integration should not be given INSERT, UPDATE, and DELETE permissions if it is supposed to only read information.
Permissions should be specific to the operation, and not granted to the whole service. An agent who has to query an order doesn't have to have permanent write access to the order database. A customizable agent ID and a restricted scope of credentialing reduce the power of the manipulated/malfunctioned workflow.
Another risk that can easily be overlooked is that the agent can use the wrong tool even if no attacker is present. The description of a tool is commonly similar to the description of another and can lead to the selection of the wrong function or to the passing of wrong arguments to a planner. For this reason, sensitive tool calls should be monitored and checked against the user action in such a way that they do not run automatically based on the model's selection only.
Another reason for NIST's focus on agent security is that agents can choose functions based on the model output, argue with it, and follow up on the various steps.
This alters the fundamental security issue. The engineering team not only has to regulate the production of the model, but the application of the model in the surrounding application has to be controlled.
AI Application Security Threats That Matter in Production
The interactions between models, data, tools and downstream systems frequently pose production risks. The following threats are significant as they could carry over from a wrong step and impact actual data or operations.
Prompt Injection
Prompt injection: An attack that involves altering the normal operation of a model. The attack may be direct (from a user) or indirect (via a webpage, email, uploaded file, support ticket or retrieved document).
An invoice-processing agent might read a document with a hidden order to call up a connected tool, for instance. The document is valid document data but the instructions contained within it are not.
Prompt injection cannot be reliably mitigated through input filtering alone and more controls around high-impact actions should be implemented.
Data Poisoning
Data poisoning occurs if the training, fine-tuning or retrieval data alters the behavior of the system. May include false information, bias or vulnerabilities and backdoors. RAG systems' answers rely on external knowledge sources, adding to their exposure:
The system is broken if it takes one of the following paths: source - indexed content - retrieved context - response or action
The teams should be in charge of modification rights for knowledge sources, validation of important data, maintaining data provenance, and monitoring changes to indexed content.
See our RAG best practices for securing the retrieval layer.
Supply Chain Compromise
Additional artifacts such as packages, models, datasets, adapters, SDKs, container images and more third-party artifacts are part of the application supply chain. A damaged component has the potential to contain malicious code prior to the application's execution.
Real examples include:
Axios npm compromise (2026). The attackers exploited a maintainer's npm account and released fake versions of Axios that include a bogus dependency. TrendAI said the dependency installation caused a cross-platform remote-access trojan (RAT) to install. Interestingly, the malicious releases were present on npm without a matching release on the official GitHub repository, highlighting the need to always check for the right version number.
PyTorch nightly compromise (2022). It seems that Linux users who installed packages from PyTorch every night (via pip) were vulnerable to the malicious torchtriton package on PyPI. PyTorch stated that the broken dependency ran a malicious binary when it was being installed.
Malicious Hugging Face model (2024). JFrog researchers discovered a model that had a Pickle file they could load that would run the code, giving them a backdoor into the user's machine. This shows that one can have an executable model artifact as a dependency in the supply chain.
The engineering lesson is to regard models and model-related artifacts the same way as the production dependencies: take models from trusted sources, version them, check integrity, scan artifacts, check dependency changes, don't load untrusted models automatically.
Model Theft and Extraction
The model may be a valuable piece of intellectual property. An attacker can attempt to steal model files directly or repeatedly access a vulnerable endpoint to gain sufficient knowledge of its operation to replicate aspects of its operation. The effects can be the loss of proprietary technology, unauthorized copies, higher infrastructure costs, and disclosure of information on how the system works.
Ensure model artifacts are protected with appropriate access controls and encryption. Use authentication, rate limits, usage monitoring, and alerts for abnormal queries for hosted inference. The model repositories and deployment credentials should be treated like all other sensitive assets in production.
Sensitive Data Leakage Through Outputs
Sensitive information may be passed to the model via prompts, retrieved documents, conversation history, tool responses, and/or application context. If the controls on access to that information are compromised, it can resurface in an output to someone who shouldn't have seen it.
The risks are associated with PII, financial data, health data, credentials, legal data, confidential business data, and IP data. It is not an authorization layer to never reveal private information. A better idea would be to avoid letting information get into the model that isn't supposed to be there. This involves implementing permissions at the data source, reducing sensitive context, isolating tenants, masking data when complete values aren't needed, and verifying permissions prior to data retrieval.
Agent and Tool-Use Risks
Agents may have access to sensitive data, as well as be able to call APIs, modify databases, send emails, execute code or trigger workflows. Any error or manipulation gets worse with broad permissions.
The permissions should be associated with specific operations only. An agent with SELECT access should not have UPDATE or DELETE access by default. It is also best practice for agents to not gain developers' credentials or share service accounts.
Another risk is involved in the selection of tools. An agent may grasp the meaning of the user's request but execute the wrong function or arguments when there are multiple functions or arguments with similar functions. Sensitive calls ought to be inspected prior to execution.
A safer flow is:
User intent - tool selection - argument and permission validation - human approval (when necessary) - execution
The controls that can be applied to write, delete, payment, permission-change and code-execution operations should be more rigorous than those for read-only operations.
For more, see how to create AI agents.
Insecure Output Handling
If another component is going to process the data coming from a generated response, it should be considered untrusted data.
An application's use of model output in SQL queries, HTML, file paths, API parameters, or compiled code is the base of the problem. An attacker can affect the downstream system too if they target the generating system in an indirect way. Possible consequences are XSS, SSRF, SQL injection, privilege escalation, path traversal, and remote code execution.
As an illustration, this pattern is risky:
command = model_response
os.system(command)The model is making a decision as to what the operating system runs.
Rather, applications should assume a fixed structure and check it before doing anything else:
result = validate_tool_output(model_response)
if result.is_valid and result.action in ALLOWED_ACTIONS:
execute_approved_action(result)It's easy to understand: generation and authorization are two different things. A model can suggest an action or generate data for an action but deterministic application logic should determine if that output is valid and can proceed.
AI Application Security Lifecycle: Where Controls Belong
Security should be part of the application, from the design to the deployment. Monitoring can't give back the details of an application's execution if the application didn't know about them, and a control added at deployment cannot solve all problems.
The practical method is to determine what needs to be protected at each level and incorporate control into that stage.
| Stage | What to Secure | Key Controls |
|---|---|---|
| Design & Architecture | Trust boundaries, data flows, permission scoping | Threat modeling, least-privilege design, separation of agent identities |
| Data & Training | Datasets, fine-tuning data, retrieval sources | Provenance tracking, validation, access control, privacy checks |
| Model Supply Chain | Pre-trained models, adapters, dependencies | Trusted registries, version pinning, artifact scanning, hash verification |
| Development | Application code, prompts, tool integrations | Input/output validation in code, guardrails as application logic |
| CI/CD Pipeline | Build artifacts, model packages, configurations | Automated scanning, policy enforcement, SBOM/AI-BOM generation |
| Deployment | Inference endpoints, API permissions, credentials | Authentication, rate limiting, token limits, scoped API keys |
| Runtime & Production | Live model behavior, tool calls, data access | Execution tracing, anomaly detection, drift monitoring, audit logging |
| Incident Response | Compromised models, leaked data, manipulated agents | Credential rotation, tool isolation, model rollback, log preservation |
Design and run-time incident response are generally the areas with the largest gaps.
It is common for teams to give agents access to a whole service at design time, when they only require a few operations. Individual agents must be individual, not share service accounts or inherit human credentials from other agents. This reduces the effect from a compromised agent and makes it easier to determine what system did an action.
Standard API logs at runtime indicate that a request occurred, but they may not reveal which agent chose the tool, the arguments it produced, or the reason for the action. Execution traces are useful for probing unexpected behaviour and being able to attribute actions appropriately.
Model-driven systems must be taken into consideration for incident response as well. Groups should have the ability to stop the execution of a tool and revoke credentials, isolate integrations, roll back impacted components, and keep execution logs that can be followed up.
The basic idea is that permissions reside in architecture, controls are added at ingestion time, validation is performed in application logic and monitoring is performed at runtime. Security is more effective if each risk is managed at the point of entry into the system.
Best Practices for Securing AI Applications
The best way to deal with it is to imagine that the model may be wrong or that it may have been misled. When it does, security controls should need to be put in place to limit what occurs.
Enforce Security in the Application, Not Just the Prompt
It is possible to steer behavior by a system message like “never expose secrets” or “only execute safe commands,” but it is impossible to enforce security. Application code is a place where important decisions should be made.
Sample generated code might produce an unsafe SQL query:
# Insecure: user input becomes part of the SQL command
query = "SELECT * FROM users WHERE email = '" + user_email + "'"
cursor.execute(query)If the query includes user input, it is kept separate in a parameterized query:
# Safer
query = "SELECT * FROM users WHERE email = %s"
cursor.execute(query, (user_email,))This also extends to generated shell commands, HTML, file paths and API parameters: check model results before integrating with another system.
Similar controls should be used for dependencies. As in any supply-chain attack, package managers like npm have fallen victim. Approve the sources, version sharing, lock files, dependency checking, and code review prior to deployment if the coding agent can install packages.
Apply Least-Privilege Access to AI Agents
Permission levels are easily understood with coding assistants. There are various terminologies but access tends to become more independent:
Read/plan – Edit files – Execute with approval – Automatic execution within limits – Full access.
For instance, Claude Code offers approval modes from standard "approval behavior" to automatic "approval" and skipping approvals in a suitably restricted environment. Codex also offers programmable sandbox and approval restrictions.
Production agents should do the same: provide them with just the access they need for the job at hand.
A customer record-reading agent shouldn't automatically be allowed to alter customers' records. Database delete permissions should not be included in a reporting workflow. Read and write access must be distinguished from each other and irreversible actions should be subject to more rigorous controls, or to manual authorization.
It's also a good idea to have agents have their own identity and scoped credentials, rather than a developer's or one shared service account. This will minimize the impact that a faulty workflow could produce and will make actions more traceable.
Treat Retrieved and External Content as Untrusted
Malicious instructions can be contained within files, web pages, emails, support tickets, repository content and RAG documents.
A useful rule is:
The content is provided externally. It has no authority.
For instance, an agent can retrieve an invoice with data that is required, but the text within an invoice shouldn't be able to provide the agent with an ability to send an email, change a payment, or call another tool.
Isolate trusted instructions from retrieved content, identify the origin of content and limit the influence of external content.
Validate AI Inputs and Outputs Independently
Review information before it gets into the workflow and before it is produced and sent to another system.
Input controls can check the types, sizes, schemas, sources and permissions of files. In the case of output controls, check against the controlled values such as generated tool arguments, database queries, URLs, file paths and other structured values.
Instead of a model creating an arbitrary command, specify a given format:
{
"action": "get_customer",
"customer_id": "12345"
}Application code can then determine if it can get to the customer, if the ID is valid, and if the current user is authorized to access this customer.
The reliance on the model which gave rise to an action is not to be taken as the sole basis for deciding whether the action is safe.
Test AI Applications Against Adversarial Attacks
Normal tests are to verify whether the expected requests are working. Security tests must examine the system's reactiveness in the event that someone intentionally tries to abuse the system.
Conduct tests on the DPI, contaminated documents, unapproved data requests, corrupted tool arguments, efforts to get around permissions, dangerous files, and frequently denied actions. Agent testing should also include a test of accidental failures. When the user's request is clear, two tools can have the same description , and the model will select the wrong one.
Monitor AI Behavior and Tool Use in Production
The API log contains the normal log message indicating that a request has occurred but doesn't describe how the system got to that action.
If any workflows are important to track, record the entire workflow:
User request → Retrieved context → Model decision → Tool/arguments → Authorization → Result
This will allow you to identify which agent performed an action, what information the agent used and which tool the agent chose.
Checks are required before an operation for high-risk operations. It's better to block a suspicious payment, change to a database, or permission update before it actually occurs, than find out later through a dashboard.
Prepare for AI-Specific Security Incidents
Examples of real incidents make clear these controls are important. Replit admitted in July 2025 to mishandling data from a user's application database when it was deleted by its Agent. Production application changes may impact the data at development time, but the data could be restored if it is needed. The company then focused on further separating the different databases - development and production - and refining a chat-only mode, where planning would take place without touching the project or the database.
An even more dramatic example was during an evaluation of OpenAI's cyber-capabilities in July 2026. It has been reported that models with less restrictions on their access to the internet discovered and exploited a new security flaw in a package-registry proxy to access the internet, which was then used to breach the research environment and, finally, Hugging Face infrastructure, to try to get access to benchmark solutions. OpenAI emphasized that this occurred with "specialized evaluation conditions" and not during "regular public use. Hugging Face confirmed and acknowledged the intrusion.
These cases illustrate an important principle: The scope of an autonomous system should only be prescribed by instructions.
Teams will be able to halt the execution of tools, revoke credentials, isolate integrations within the tool, restore impacted data, roll back configurations, and keep execution traces for investigation. Backups and isolation of production environments are particularly significant when agents have the ability to change or remove production data.
Regulatory Frameworks Shaping AI Application Security
Security frameworks provide a shared approach to risk assessment, risk control selection and risk management documentation for engineering teams. The following four are of special importance, but they have different functions.
OWASP Top 10 for LLM/GenAI
The OWASP Top 10 for large language models and GenAI Applications can be helpful as a security checklist. It has an updated list of risks for 2025, such as prompt injection, disclosure of sensitive information, supply-chain vulnerabilities, data and model poisoning, wrong output handling, and excessive agency.
It can be utilized by teams for threat modeling, code reviews, security testing and architecture reviews to verify if common application level threats have been addressed. In December 2025, OWASP also published an Agentic Applications Top 10 for systems which can execute tools and actions.
NIST AI Risk Management Framework
The NIST AI Risk Management Framework (AI RMF) offers a more general perspective on risk management over the AI system's lifecycle. It has three main functions: Govern, Map, Measure and Manage, which can enable organizations to establish responsibility, determine risk, assess risk, and plan to manage risk.
When it comes to engineering teams, this is not just about vulnerability testing for security. Other areas of risk management include architecture, data, evaluation, deployment, monitoring, and governance. In addition, NIST offers a special generative artificial intelligence profile of generative intelligence related risks.
EU AI Act
The EU Artificial Intelligence Act outlined is not a checklist for security. It adopts a risk-based approach, which varies with the nature of the system and its use.
Requirements for systems under its high-risk rules are: Risk management, Data quality, Logging / Traceability, Documentation, Human oversight, Cyber security, Robustness, Accuracy.
The order of time is also important. The obligations for general purpose models will come into force earlier, on August 2, 2026, as will the enforcement powers of the Commission, while the requirements for transparency under Article 50 will begin on August 2, 2026. Teams must check the rules applicable to their product and deployment as some high-risk-system timelines have been modified or proposed for modification.
Google SAIF
Google's Secure AI Framework (SAIF) is especially helpful in converting identified risks to technical controls. It has a SAIF Map that links threats with controls that engineering teams can use throughout development and operation.
Such as training-data sanitisation for poisoning risks, access controls for sensitive data, and application-level protection for connected systems, etc. It explicitly relates to the permissions of tools (least privilege), user approval for action and visibility of tool use and agent activity for agents.
Where Engineering Teams Go Wrong
One of the most frequent errors is security being added at the end of the design process. Agents can have all the permissions, data can flow through unessential services, and high-risk actions could be unapproved by then. Threat modeling should be done during data flow, identities, tools and trust boundary definitions.
Teams over-depend on system prompts as security controls as well. Access control is not enforced by saying to a model "never expose confidential data". The credentials of an agent should not have delete permission if the agent is not supposed to delete the records. Authorizations should be in application code/infrastructure.
It's a problem that arises again if you share service accounts between agents. Having more than one agent with the same identity means that permissions are more difficult to control and actions are harder to track. Teams should have scoped credentials for each agent, so that they can manage and attribute the activity of each of them.
Standard API logging is also not enough for “tool using” systems. A log may report on an API call and not indicate which agent made the call, what arguments it included, or what might have been the context for the call. Execution traces should link the original request to the selection of tools and authorization of the action taken and the resulting action.
One of the errors is assuming that a secure model provider is a secure application. The provider ensures that it's own platform is not compromised, but not that your application will not be able to give too many permissions, get data for the wrong peer, expose secrets in the platform, or invoke unsafe tools calls. Such controls are still the responsibility of the engineering team.
Lastly, output to models should not always be taken for granted by the rest of the system. Any generated SQL, API parameters, shell command, URL or tool argument should be checked prior to execution. Assume model output is untrusted input, and use deterministic application control to determine what is allowed.
The engineering principle that runs through all six mistakes is that don't rely on model behavior for controls that can be codified, granted and reliably enforced by infrastructure.
How Coding Crafts Builds Secure AI Applications
Security is always a part of the Coding Crafts development process. The intention of the goal is to ensure that the application can successfully process real users, business data, associated tools, and unusual behavior.
Security Starts in Architecture
The first step in security is to know where data originates, where it's going, and what each part of the system has access to. Coding Crafts sets up trust boundaries, traces data flows and restricts permissions for each tool as necessary.
Tighter controls are placed on high-risk actions like record changes, running code or accessing sensitive systems. The "least-privilege access" is not an afterthought during development, but is part of the architecture.
Privacy and Governance by Design
Not all company data requires to be added to a prompt or a model. Coding Crafts detects and manages sensitive information early on and at its source.
This can be done by keeping unnecessary data in prompts and logs to a minimum, ensuring that the source of the data retrieved is preserved, splitting access when it's necessary, and ensuring that users and agents have access only to the data they need for their tasks.
Guardrails as Application Logic
Important security rules should not rely on a model to follow instructions correctly. Coding Crafts has built-in input and output validation, permission checks, tool restriction and approval steps in the application. Sensitive actions can be vetoed prior to running and irreversible or impactful actions may need to be approved by humans. This provides security controls even if the model responds in an unusual way
Production-Ready, Not Demo-Ready
A prototype does not require that it be 100% accurate. Failure needs to be addressed in the production system as well.
Coding Crafts creates applications that are secured with scoped credentials, tracked actions, runtime monitoring, secure integrations and incident response procedures. Teams can view what the system has done, determine which component or agent has taken an action, and be able to react rapidly if something goes wrong.
The outcome is an application that is not only high performing, but safe to use with actual users, data and production systems.
Security is built into the architecture, not an afterthought, as Coding Crafts creates AI systems with scoped credentials, guardrails, auditable actions and runtime monitoring.
Secure AI applications, end to end
Coding Crafts builds AI applications with least-privilege agents, validated inputs and outputs, execution tracing, and incident-ready operations.
More from the journal.
View all postsRelated reading from the Coding Crafts team.