Most organizations are going to be self hosting on aws, gcp or azure... So as long as you use their inference services as your LLM then you can keep it all within the private network
Even self-hosting on AWS, GCP, or Azure isn't local enough for certain application, such as people doing export-controlled work where any sysadmin or person with physical access to the server/data is required to be a US Person (or equivalent in other countries). This is the niche that the govcloud solutions are aimed at serving. But some people just want to build big actually-private, actually self-hosted systems and do their own physical and network security.
Are all people with physical access to the servers or network access to the hosts guaranteed to be US persons? Are all physical and network accesses logged for audits? That's the kind of thing govcloud promises that export control auditors want to see.
I felt like "Confidential Compute" tech could solve this issue once and for all but I'm not so sure after seeing some of the attacks people can do with physical access.
Another option of course is to not use cloud at all and have your own rack in a locked room with a good security system and/or armed US person guards.
Exactly, enterprise customers almost always use private model endpoints on their cloud provider for any serious deployments. Data stays within the customer's VPC, data security and privacy is guaranteed by the cloud providers.
