Enterprise AI Security and POPIA Compliance in Practice

An AI system introduces a category of risk that existing enterprise security controls were not designed for. The perimeter, identity, and vulnerability management practices an organisation already runs remain necessary, but they do not address a model that can be induced to reveal its instructions, a retrieval layer that returns documents the requesting user was never entitled to see, or a training dataset assembled from personal information collected for an unrelated purpose.

Security review is where a great many promising AI initiatives are stopped, usually late and usually for reasons that were predictable at design time. The controls below are the ones we implement as a matter of course, because they address the questions a competent reviewer will ask.

Classify data before the model sees it

Every control that follows depends on knowing what data is in play. Classify at ingestion, not at consumption, and record the classification with the data rather than in a separate register that will drift.

For each class, decide three things explicitly: whether it may be used for training, whether it may be sent to a third-party inference endpoint, and where it may be processed geographically. Encode those decisions in the pipeline so that they are enforced by the platform. A policy document that says personal information must not leave the South African region is not a control; a pipeline that refuses to route classified records to an out-of-region endpoint is.

Treat POPIA as a design input

The Protection of Personal Information Act shapes AI architecture in several concrete ways, and the requirements are considerably easier to satisfy when designed in.

Purpose limitation is the constraint most frequently breached. Personal information collected to administer a policy has not thereby been made available for training a propensity model. Establishing the lawful basis for the new processing purpose, before the data is used, is the single most valuable conversation to have with your information officer.

Minimisation argues for training on the narrowest dataset that supports the objective, and for de-identification wherever the identifier is not itself predictive. In practice a surprising proportion of features that engineers include out of convenience contribute nothing to model performance and a great deal to regulatory exposure.

Automated decision-making that has legal consequences for a data subject attracts specific obligations. If a model's output can decline an application, terminate a service, or set a price, the design must preserve the ability to explain the decision and to route it to human review. Retaining the feature values that produced each decision, alongside the model version, is what makes this possible after the fact.

Retention and deletion must extend to derived artefacts. A deletion request that removes a record from the operational database but leaves it in a feature store, a vector index, and three months of training snapshots has not been fulfilled. Map every place data comes to rest, including indexes and caches, before the first deletion request arrives.

Controls specific to language models and retrieval

Systems built on foundation models require controls that have no analogue in conventional application security.

• Treat all model output as untrusted input. Anything a model produces that is then executed, rendered as markup, used in a query, or passed to a tool must be validated exactly as user input would be. This single principle prevents the majority of practical prompt injection exploits.
• Enforce authorisation in the retrieval layer, not in the prompt. A retrieval-augmented system must filter candidate documents by the requesting user's entitlements before they reach the model's context. Instructing a model not to disclose material it has already been given is not an access control.
• Give the system its own least-privileged identity. A model that can call tools should hold narrowly scoped credentials of its own, distinct from those of the calling user and the surrounding application, so that its blast radius is bounded and its actions are attributable.
• Constrain tool invocation. Where a model can trigger actions, define an explicit allow-list of operations with validated parameters, and require confirmation for anything irreversible or material.
• Log prompts, retrieved context identifiers, model version, and output for every inference, with a retention period agreed with your privacy function. Without this record, incident investigation is guesswork and the explainability obligations above cannot be met.
• Apply rate and cost limits per identity. Uncontrolled inference is both a denial-of-service vector and a budget risk, and the two failures tend to arrive together.

Secure the supply chain and the pipeline

Model artefacts, weights, and the packages used to produce them are software supply chain components and deserve the same treatment as any other dependency. Pin versions, verify checksums, load weights from formats that do not permit arbitrary code execution on deserialisation, and record the provenance of every artefact promoted to production.

The training pipeline itself is a target. An attacker who can influence training data can influence behaviour more durably and less visibly than one who attacks the running service. Version datasets, restrict write access to training inputs, and be able to reproduce any deployed model from a recorded dataset version and configuration. Reproducibility is a security property as much as an engineering convenience.

Evaluate adversarially, and monitor for drift as a signal

Before release, test the system the way an adversary would rather than only the way a user would. Attempt to extract system instructions, to retrieve records belonging to other users, to elicit outputs that breach policy, and to induce tool calls the requesting user is not entitled to make. Record these as regression cases and run them in the deployment pipeline, because a model upgrade can reopen a vulnerability that a previous version had closed.

In production, monitor input distributions and output patterns rather than only service health. A shift in the shape of incoming requests is frequently the first visible sign of either an upstream data fault or deliberate probing, and it will not appear in an error rate. Pair that monitoring with a documented response path: who is notified, what the safe fallback behaviour is, and who has the authority to disable the model and revert to the prior process.

An AI system that can produce this evidence on request — data classification, lawful basis, access enforcement in retrieval, artefact provenance, adversarial test results, decision logs, and a rehearsed rollback — is not merely more secure. It is defensible, and it will clear review in weeks rather than quarters.