Private AI describes an AI deployment whose data, model access and operations are governed within an organization-defined boundary. It does not necessarily mean an air-gapped server: a private cloud tenant, on-premises cluster or local workstation can each qualify for a particular workload, depending on who can access the data and how traffic and logs are controlled. “Private” is a design claim to test against an explicit threat model, not a guarantee conferred by hardware location.
Start with the boundary
List the people, documents, prompts, retrieved passages, model weights, telemetry and backups that the system touches. Mark which systems may process or retain each item. For a confidential document assistant, the boundary may permit authenticated employees and an approved inference endpoint but forbid sending retrieved passages to an unapproved external API. Verify the actual endpoint and logging settings rather than relying on a product label.
Employee identity → access policy → retrieval service → approved model endpoint
│ │ │
audit decision scoped documents response
└────────── monitored boundary ────┘An implementation sequence
- Classify workloads: distinguish public prompts from regulated or confidential data; identify permitted processing locations and retention limits.
- Choose an inference boundary: compare local, on-premises and private-cloud options for model fit, capacity, patching, physical security and disaster recovery. A local model can still leak data through logging or connected tools.
- Enforce identity: authenticate users, authorize access to each source document, and pass only authorized context to the model. A network firewall alone cannot enforce document-level permissions.
- Limit egress: allowlist necessary destinations, protect secrets in a server-side vault, and inspect model, embedding and telemetry routes for unintended transfers.
- Test and operate: measure answer quality on representative permitted questions, exercise revocation and incident response, and monitor capacity, updates and audit events.
Tradeoffs worth documenting
Self-hosting can reduce exposure to third-party inference providers, but transfers responsibility for patching, model lifecycle, GPU scheduling and resilience to the operator. Private-cloud isolation may simplify operations while still involving provider processing. Neither architecture automatically satisfies a regulation or prevents prompt injection. Record assumptions, responsible operators and validation evidence for each control.
For sizing and deployment considerations, see AI Systems. For layered controls, see AI Security; for retrieval permissions see RAG. To scope an architecture review, explore consulting.
