AI Model Security: Protecting Models, Data, and Best Practices for Production

This is not some theory, as proved by recent red-team results. In 2026, NIST's CAISI (Center for AI Standards and Innovation) evaluated over 250,000 attack attempts by over 400 participants against 13 frontier models, resulting in at least one successful hijacking attack on each model. The UK AI Security Institute has also shown how effective AI systems can work around security weaknesses in the sandbox, such as typical configuration failures found in containers.

The attack area is wider for the engineering teams than the prompt. Poisoned datasets, unverified model artifacts, retrieved documents, exposed endpoints, unsafe tool combinations, and too many permissions are ways risk can enter into the system. If a system can access sensitive information and if it can take actions, then a weakness can result in exposure of data, changes to the data without authorization, or unexpected infrastructure costs.

Security of AI models is a problem at the system level. This guide details the sources of those risks that are ranging from training data through to model artifacts to inference, agents, and production operations, and the precautions teams should take before real data and real permissions come into the picture.

What Is AI Model Security?

Artificial intelligence model security refers to the protection of a system throughout the data flow process, from the source of data to the models' utilization during production. It means everything from the training data to the model files, infrastructure, APIs, credentials, connected tools, and sensitive data the application can access.

It's more than just blocking malicious calls. The data in the training set could be manipulated, files could be stolen or changed, endpoints could be used in an inappropriate way, and valuable information could be leaked in the responses. If agents are linked to business tools, the compromised interaction can also lead to actions that the user didn't intend for the agent to take.

The model can only be used for text generation, which is not enough. As you share the same system access to databases, email, customers' records, or internal systems and tools, it begins to look and act like a service account: what it can access and what it can do becomes a security decision.

That's for engineering teams to ensure that they have the entire trail around the model, not the model alone. These all need to work together as part of data integrity, access controls, protected credentials, scoped tool permissions, runtime guardrails, monitoring, and audit trails.

Why the Model Layer Became Its Own Attack Surface

The risk altered as models went from being used to process inputs and make predictions to being embedded within live applications. Today, the same system can access internal data, access external information, access API (via the internet), utilize saved credentials, and make decisions that impact other software.

That poses a new trust issue. The user doesn't have to enter a message in a chat box; it can be anything. Instructions may be received via email, web pages, uploaded files, retrieved documents, or tool replies. An attacker can direct untrusted content that influences such a system with privileged access to read, choose a specific tool, or trigger an action to be taken.

The stakes are increased by agents. A system that can access the database, with an API key, and with permission to update records is more of a service account and has more to do with reason about the permissions it has than a software library. Those permissions must, of course, be carefully considered, like any other privileged identity: least-privilege access, separate credentials, explicit approval for high-impact actions, and clear audit trails.

Additionally, the supply chain has grown. Pretrained models, datasets, adapters, dependencies, and artifacts can be from external repositories or vendors. If there's a compromised component, it can get into the application before it goes into production, thus making provenance and artifact verification part of the security boundary.

So now there's not just the user - application - database attack surface anymore. Untrusted content is now able to impact a decision-making layer between the user, company data, credentials, and operational tools. The more control that layer has, the more controls should be in place.

AI Model Attack Surface: Stage by Stage

Security threats can be found in various stages of the lifecycle, and thus must be addressed as a variety of “model security” issues. A different control is required for a poisoned dataset than for an over-permissioned agent or a stolen model artifact. Every risk having to do with the stage at which it enters the system makes it easier to see ownership and mitigation.

Training & Fine-Tuning Data

The training data is located within the security boundary. Poorly handled datasets can incorporate PII, customer information, or private company data into training, and poisoned samples can be leveraged to introduce hidden behaviors or hidden triggers.

Know where the datasets are from, who has modified them, and which version has been used for each training session. Limit write access, sanity check unusual distribution and/or suspicious samples prior to data entering the training pipeline, and perform privacy checks. In the event of a problem occurring after, the team should be able to identify the exact version and/or dataset that created the affected model.

The Model Supply Chain

Any external software used should be treated like a downloaded model. Unsafe serialization, compromised dependencies, typosquatted projects, or artifacts with unclear origins are examples of types of public repositories. When a model passes an accuracy test, it does not imply that the artifact is trustworthy.

Utilize approved registries, pin versions, scan artifacts and dependencies prior to deployment, and check hashes or signatures if available. Track provenance from download to production to ensure that the team knows which artifact is running and where from to ensure that they can trust the results.

Model Artifacts & Weights

Model weights can be a huge IP asset and can be specifically targeted. If any of these registry, storage bucket, backups, development environment, or leaked credentials is exposed, it can allow the attacker to copy the weights or replace an approved weight file with a modified one.

Encrypt weights in transit and at rest, limit access to the registry to authorized users, and separate production artifacts from other development environments. Download and manage the logs and confirm integrity of the artifacts when deploying. Confidentiality and integrity should be provided: No unprivileged person should be able to steal the model or make changes in it without anyone knowing.

Inference & Model Serving

There are a number of ways in which endpoints may be abused. Direct prompt injection is achieved by modifying the application via some form of user input; indirect injection occurs when the content is injected within content that the application processes, such as documents, web pages, emails, or other content. Extraction, inversion, and membership-inference attacks aim to gain insight into the model and/or information from the training data.

The bill could also be a target for attacks. Denial-of-wallet attacks can use resources by repeatedly requesting or adding unnecessary tool loops or input size changes, and/or by running compute-intensive tasks without taking the service offline.

Use authentication, rate limits and token limits, execution limits, input/output validation, and abuse monitoring to protect the endpoint. Don't put any malicious request on the model; apply important limits in application code and infrastructure.

The Agent and Tool Layer

Access to the tools allows a bad decision to be a possible action. An agent can access customer data, send messages, modify databases, run code, and communicate with external services with the stored credentials.

Many times the forgotten risk is the combination of permissions. It could be perfectly legal to read other agents' confidential records; it could be perfectly legal to send an external email, but if you give both agents both of those things, then there's a possible way to exfiltrate data.

Provide agents with distinct identities, just what they need for the task at hand. Provide separate read and write access, do not have access to tools that are too broad in scope, and have deterministic checks or approval from humans before taking high-impact actions. The same process should be applied to MCP servers and other external tool integrations: confirm the operators, what they may be able to access, and what capabilities they may provide before they're trusted.

Observability and Operations

These systems may start without crashing. A request could be successful from the application's point of view, while the wrong tool is called, sensitive information is returned, or an unauthorized action is attempted. Those failures might not be noticed if there is no runtime visibility.

Follow the execution through the steps of the request, the retrieved context, tool selection, arguments and results for the tools, permission decisions, approvals, and final actions. Ensure that those logs are treated as sensitive data and, therefore, have their own access, retention, and redaction.

Keep an eye out for out-of-the-ordinary use of tools, frequent violations of policies, unusual traffic patterns, unexpected bill increases, and behavior changes. Keep an eye out for anomalous tool activity, persistent policy infractions, unusual traffic, unforeseen expense hikes, and behavioral shifts.

AI Model Security Threats That Matter Most

The attack surface is wide, but the following are the vulnerabilities that are most likely to impact data, system integrity, or privileged actions in production. MITRE ATLAS also considers the following security concerns as separate: poisoning, prompt injection, supply-chain compromise, excessive agency and tool misuse.

Data Poisoning Attacks

Data poisoning is the deliberate addition of bad or biased examples to the training, fine-tuning, embedding, or retrieval data to affect future behavior. A poisoned source may circulate false information, surreptitiously change decision-making, or initiate hidden actions when a trigger occurs. Lower risk by using trusted data sources, keeping track of the provenance, limiting write access, validating before ingestion, and having regression tests to check for unexpected behavior.

Adversarial and Evasion Attacks

An adversarial attack is an attack that is designed to make a system misclassify or misinterpret something that it would normally classify correctly. That might enable fraud or something else malicious to circumvent a control intended to prevent it. Test inputs against expected and unexpected scenarios before release and do input validation, anomaly detection, and verification downstream.

Prompt Injection, Direct and Indirect

A direct injection is instructed by a user, and an indirect injection is concealed in content like webpages, emails, files, or documents that have been retrieved. When receiving systems also have tools, manipulated content can affect access to data or cause actions that were not requested by the original system user. Consider external content to be untrusted, treat it like a separate privileged instruction, restrict access to and the permissions of tools, and use code-level authorization/approval for sensitive actions.

Model Inversion and Membership Inference Attack

Model inversion attempts to get information about the training data back, and membership inference tries to find out if the data record was included in the training set. Probing can reveal valuable details about customers, employees, patients, or other people represented in the training set that can be helpful in a successful probe. Limit the volume of sensitive training data, limit access to endpoints, watch for repeated probing, and use the proper techniques for privacy preserving.

Model Extraction and Model Theft

Extraction attacks involve a series of queries to help determine a proprietary function, and direct theft involves stealing weights, checkpoints, or artifacts that have been stored. Either can reveal valuable intellectual property and facilitate the use or recreation of proprietary features by another party. Securely and carefully manage model artifacts with encryption, secure inference endpoints with authentication, rate limits, and abuse detection.

Model Supply Chain Attacks

Supply-chain attacks come through something trusted, such as models, datasets, adapters, dependencies, repositories, or loading mechanisms. Malicious code or executable code may be introduced into the artifact prior to when the application receives its first request. Before deploying, make use of trustworthy registries, pin versions, scan artifacts and dependencies, and confirm provenance, hashes, or signatures.

Sensitive Data Leakage

Sensitive information can leak out in the form of responses, retrieved context, tool results, or data that is only in the working context. The outcome could be the disclosure of customer information, credentials, internal documents or regulated information without any traditional database leak. Implement authorization rules prior to retrieval, reduce context, redact sensitive data, maintain secure logs and validate retrieved output before it leaves the application.

AI Agent and Tool-Use Risks

Tooled agents are dangerous when too many functions, permissions, or freedoms are allowed to be used to unify a manipulated decision into an actual decision. An overly privileged agent might be able to edit records, leak sensitive data, run code, or mash together seemingly harmless tools into a harmful workflow. Assign agents scoped identities, only show tools that he/she needs, and enforce deterministic authorization or human approval for high-impact actions.

Insecure Output Handling

Insecure output handling is when the output from a system is treated as trusted input to another system without validation. When this output is SQL, HTML, shell commands, API parameters, or tool arguments, vulnerabilities like XSS, SSRF, privilege escalation or remote code execution can result from manipulated content. Consider output generated as suspicious material and apply schemas and allowlists, sanitize it for its ultimate goal, and separately approve privileged operations.

Best Practices for AI Model Security

There is no single filter or more powerful system prompt that can guarantee good artificial intelligence model security. The solution is to manage what goes into the system, restrict what is able to get in, ensure that what's deployed is known, and detect suspicious activity when it hits production. NIST and Google SAIF guidance is similar and is based on a lifecycle perspective.

Establish Data Governance

To begin, one should know the data that is stored in the system and who is responsible for it. Keep records of training and fine-tuning datasets, knowledge sources retrieved, evaluation data and sensitive logs, and define the rules for accessing, keeping, and using them.

The most important thing is to retain provenance. A change to a dataset should be obvious and should be capable of being described as to what was changed, the author and origin of the change, and the versions of the models that relied on the dataset. This means privacy concerns, poisoning attacks, and faulty information can be contained rather than having to go on a wild goose chase to find out if the pipeline is compromised.

Apply Least-Privilege Access

Only grant application/agent as much as it needs to perform its task. Secure against using the same identities for read and write access, use scoped API permissions, and where possible, use short-lived credentials.

Consider combinations of permissions, rather than individual ones. Accessing customer records may be reasonable and allowing e-mail may be reasonable, but combining these two in the same agent creates a potential leg to be squeezed. In the case of high-impact actions, an authorization check or human approval will be needed outside of the model. In fact, Google SAIF specifically suggests that the permissions an agent gets when acting on a user's behalf should be limited and that user-initiated actions by an agent should be approved.

Protect Model Artifacts and Data

Weights, adapters, prompts, credentials, and evaluation sets should all be regarded as sensitive production assets and handled accordingly. Encrypt them on the move and while stored, limit access to registries and storage, and record downloads and administrative changes.

Protection also equates to not allowing silent replacement to occur. For approved versions, use pin or hashes/signatures to ensure the same version is deployed as the one that went through approval and tests.

Validate Training Data and Model Provenance

Avoid using a dataset or downloaded model because it has come from a familiar repository. Track its source and version, scan artifacts and dependencies, inspect custom loading code, and ensure integrity before it goes into production.

Apply the same method to the training set. Look for anything unusual, anything suspicious, unauthorized sources, sensitive information. Producing a production release should be traceable back to the model version, dataset version, dependencies, and configuration that created it.

Implement Input and Output Guardrails

Assume any user prompts, documents retrieved, webpages, emails, tool responses, and output as untrusted until validated. Input controls can filter out unwanted or malicious content, and output controls can apply schemas, eliminate sensitive content, and block the direct flow of unwanted content into downstream applications. SAIF expressly suggests that input and output validation should be done separately from the model.

Identify three things for each critical guardrail: what is being checked, what causes the guardrail to fail, and what happens if it fails. That action could be blocking the request, retries, removing a capability, or escalating to a person, depending on the risk.

Test and Red-Team Before Production

A functional test indicates whether the program does what it is supposed to do, while an adversarial test demonstrates what happens if someone tries to use the program outside the assumptions.

Try out the real production architecture, from prompt injection to malicious retrieved content, permission bypasses, sensitive data requests, misuse of tools, malformed inputs, to resource exhaustion. Celebrate wins to store in regression cases so that any vulnerability that was fixed one day is not regenerated the next time around due to the next prompt, model, tool, or configuration change. NIST’s Generative AI Profile considers testing and evaluation to be a continual process of risk management throughout the system life cycle.

Monitor AI Systems in Production

These applications may crash without giving any kind of error message. A request can be successful, but selecting the wrong tool, exposing information, or trying out something it should not have been able to do.

Maintain traces that are detailed enough to be reconstructed to understand how the system works, for example, what was input, what was retrieved, what tools were called, what was authorized, what was output, what failed, and what resources were used. Keep track of odd tool usage, suspicious logins, frequently denied logins, and abrupt charge spikes. Google SAIF names agent observability and auditable use of these tools as security controls specifically.

Feed production failures back to the test suite most importantly. Monitoring should get better in the next release, rather than simply a report on the failure in the current release.

Prepare for AI Security Incidents

Be able to contain a compromised system before an event. The response plan should outline who is responsible for removing credentials, disabling tools, isolating a model or dataset, rolling back a deployment, saving logs, and rolling back to a known safe version.

The answer should be similar to the error. There are different containment and recovery actions that need to be taken depending on the leaked credentials, poisoned data, compromised artifact, and repeated injection attacks. The specific guidance on incident response from OWASP further emphasizes the importance of considering these scenarios in existing security operations and not assuming that they are unusual application behavior.

AI Model Security Frameworks and Standards

No single solution can address all risks in production. An effective way is to apply NIST to the issue of risk management, OWASP to the issue of threat identification and testing, and Google SAIF to the issue of technical control.

NIST AI Risk Management Framework

The NIST AI RMF aids in the process of identifying, evaluating, assigning ownership, and controlling risks across the system lifecycle. The four functions – Govern, Map, Measure and Manage – can help to determine what can go wrong, how serious it is, how it will be measured, and who will deal with it.

NIST is particularly helpful at the organizational level: establishing acceptable risk, recording controls, establishing evaluation requirements, and establishing accountability prior to deployment. An important point for current planning is that AI RMF 1.0 is an evolving framework and not a checklist that needs to be complied with.

OWASP AI Security Guidance

Now, OWASP is getting nearer to what the engineers have to protect themselves from. Its guidance focuses on security issues in attack scenarios like prompt injection, too many permissions, exposure of sensitive data, unsafe handling of output, and compromise of the supply chain.

This makes it useful for threat modeling, architecture reviews, security testing, and red teaming. Teams can test specific queries, such as whether an untrusted document can affect a privileged tool call, rather than recording "agent misuse" as a general risk. Can the output generated be transferred to another system without validation? Is an agent allowed to do something that they were never intended to do?

Google Secure AI Framework (SAIF)

Google SAIF is one step further towards implementation by associating risks with controls. The following are some of the current areas covered by its catalog: training-data sanitization, artifact integrity, input/output validation, least-privilege agent permissions, approval for agent actions, observability, red-teaming, threat detection, and incident response.

When a team has determined that there is a risk, this is where SAIF comes in handy when they want to know what to do next: what should we do?

The three frameworks as such are thus complementary, rather than in competition. Organize risk with NIST, understand how it can be attacked with OWASP, and convert the risk to engineering controls with SAIF. While a framework can help inform the security program, it should not be used to replace architecture reviews, adversarial testing, monitoring, or production enforcement.

Where Teams Get This Wrong

It is important to note that most of the problems in security begin with an unusual attack. They begin with prosaic engineering tricks, which were tolerated during the prototype phase and silently made it into production.

Security is not a factor in the completion of the architecture. Teams first link data sources, tools, and APIs, and subsequently seek security's review of the resulting system. By that time, the permissions are too broad, and the trust relationships are too unsafe to be remedied, and it will cost more and take longer to define them in the design.

The security layer is formed by the system prompts. Words like “never disclose confidential information” or “use this tool only when called for” are assertions about behavior, which should be made, but aren't access control. Permission, approvals, input validation, and tool restrictions must be enforced outside the model, as it can be manipulated or it can just fail.

More than one service account is used by several agents. It makes it easy to integrate, but it does have the downside that a low-risk workflow might fall into the permissions of an admin workflow. Instead, provide each agent with its own identity and scoped access to the services, and shared service accounts are becoming an outdated concept in guidance.

Teams records the API request but not the execution route. If the request fails, they have access to the request and the response, but not to any of the documents that were retrieved, any of the tools that were called, or what actions were taken during the retrieval. Production traces should be made possible to audit the use of tools and actions of the agents, as a run may not fail, but do something wrong.

Application security is considered as provider security. A secure hosted model is not a factor in which database records you share, the strength of your credentials, whether or not retrieved content is trustworthy, or whether or not a tool call must be approved. These controls still fall to the team designing the application around the ones that have been created.

The common error is to allow a probabilistic element to take the place of authority. According to OWASP, excessive functionality, permissions, and autonomy are some of the primary drivers of damaging agent actions. The safer way is to have the model suggest what should be done, and have application code determine what will be allowed to happen.

How Coding Crafts Approaches AI Model Security

Architecture should be designed with security in mind prior to real data, credentials, and business operations being granted access to the system. At Coding Crafts, those controls are built in with the application as opposed to added on top of it after development.

Security Starts During AI Architecture

We start by outlining the areas that the application can access, trust, and change. Data sources, external integrations, tool permissions, boundaries of trust, and high-risk actions are all decided in advance, and the architecture doesn't create any needless access.

Privacy and Governance by Design

Not all available data will be required to be placed in a prompt, log, or model context. We outline the sensitivity of the information flow in the application, control access to the information at the source, and have transparent ownership and provenance of the datasets and models and third-party components.

Guardrails and Evaluation

Failure modes are associated with controls. The validation of input, permission checking, structured output, restriction of tools, and approvals by humans are checked through realistic misuse cases, not just as instructions in a system prompt.

The application is continually being tested for security as it evolves. Teams now have new integrations, timely updates, model changes, and production failures to evaluate as cases to help minimize regressions becoming incidents.

Production-Ready AI, Not Just Prototypes

To go live is not something that can be done once a security review has been passed. Coding Crafts develops applications that have scoped credentials, auditable actions, runtime monitoring, incident procedures, and clearly defined operational ownership to understand what happened and what to do when a thing has gone wrong.

The goal is simple: to create a system that you can safely use, maintain, and develop in production, rather than a system that only works in demo conditions.

Need to secure an AI application before production? Talk to Coding Crafts about building a secure, production-ready system with security designed in from the start.