Google designs persistent Private AI Compute memory around device-held keys

Google has published the architecture for persistent server-side memory in Private AI Compute, extending a system that had been built around ephemeral private inference. The new design gives a Google AI experience a per-user cloud memory that can survive individual sessions and be reused across devices, while the stored database remains encrypted outside an attested confidential-computing environment. Google describes the capability as infrastructure for future memory-dependent experiences and has not announced a broad product rollout, supported regions or account eligibility. The distinction matters because persistence changes the system's security problem. A stateless request can discard its sensitive context when inference ends; a memory service must retain useful state, route later requests back to the correct user's store and protect that data between sessions. Google's September technical brief therefore treats server-side memory as a separate threat model rather than as a small extension to its earlier private-inference design.

The memory subsystem runs a stateful Oak server inside an AMD SEV-SNP confidential virtual machine. Trail of Bits' review describes a key-encryption key derived on the user's device from a user secret and a per-user data-encryption key generated by the service and stored only in wrapped form. When memory is needed, the device opens an attested end-to-end encrypted Oak session and provides the key-encryption key; the enclave can then unwrap the data key and decrypt records in volatile memory. Google infrastructure outside that trusted boundary stores ciphertext rather than the plaintext memory.

Inside a request, an orchestration enclave retrieves the relevant records over mutually attested channels, combines them with the active prompt and passes the bundle to Google's hardened TPU inference environment. New memories can then be committed back to the Oak server, re-encrypted and written to persistent storage. Google's technical brief says volatile prompt context, tokens and intermediate activations are purged after the response, while the encrypted memory remains available for later sessions.

Google explicitly narrows one privacy claim for the stateful path. Stateless Private AI Compute can use anonymous routing so the inference service cannot associate a request with a specific user, but persistent memory must resolve a stable per-user identifier to find the right database. The technical brief therefore does not claim network-level non-targetability for memory requests. Its defence is instead that a targeted store contains ciphertext whose keys are released only inside an attested enclave.

Google has released the memory Oak Server within Project Oak and says the enclave application is written in Rust on the Oak Containers runtime. The company says reproducible builds connect inspectable source to production binary digests, which are recorded in an append-only ledger, and attestation prevents an unendorsed memory binary from receiving decryption keys. The wider Private AI Compute stack is only partly open, however, so public inspection of the memory component does not make the complete service independently reproducible.

Google also commissioned Trail of Bits to threat-model and review the secure-memory implementation. Four consultants worked from June to July with access to source code, configuration and internal documentation, followed by a fix review completed in August. The final report lists ten findings. Eight were marked resolved after the fix review; one residual memory-rollback issue was accepted as an unresolved risk, and one high-severity finding involving hashed model-response chunks leaving the trusted computing base remained unresolved while Google worked on an in-enclave Bloom-filter mitigation.

Trail of Bits also set limits on what its assessment proves. It did not re-review the hardened TPU platform, some Google-internal infrastructure or the complete Private AI Compute codebase, and it notes that substantial closed-source components remain. The assessment is a point-in-time review rather than a guarantee about every future deployment. That makes the audit meaningful evidence about the reviewed design and fixes, while leaving the service dependent on Google's attestation, update and transparency mechanisms after the review date.

Google's announcement establishes the architecture and the supporting security work, but it repeatedly uses future-facing language about what persistent memory will enable. Until Google names the products that use the memory layer and publishes rollout, retention and user-control details, the practical exposure is still unclear. The next test for the design will be whether those product implementations preserve the same key flow, attestation and transparency properties described in the technical brief.