Designing a Sovereign AI Stack for European Organisations

The temptation in AI sovereignty discussions is to focus exclusively on model providers. Which LLM do you use? Is it European? This framing is incomplete and potentially misleading.

AI systems interact with multiple infrastructure layers, each with its own sovereignty profile:

**Data ingestion**: Where does training and RAG data originate? How is it collected and stored? **Model inference**: Which provider processes prompts and generates outputs? Under whose jurisdiction? **Vector storage**: Where are embeddings stored and queried? Who can access them? **Application layer**: Where does the application that calls the AI model run? **Monitoring and logging**: Where are prompts, completions, and telemetry stored? Who can access operational data? **Access control and identity**: How are users authenticated? Which identity provider governs access?

A sovereignty gap at any layer compromises the sovereignty of the entire system. An organisation using Mistral (EU) for inference but Pinecone (US) for vector storage has created a data flow where embeddings of potentially sensitive documents are stored under US jurisdiction — potentially undermining the sovereignty gain from using an EU model provider.

**The stack must be assessed holistically.** This guide covers each layer.

The model layer is the most visible sovereignty decision, but not always the most impactful.

**EU-sovereign options**: - **Mistral AI** (France): The leading EU-native model provider. Open-weight models available for self-hosted deployment. Proprietary models accessible via API under EU jurisdiction. The default recommendation for sovereignty-conscious organisations. - **Aleph Alpha** (Germany): Enterprise-focused, German-headquartered. Strong in German-language tasks. More limited model range than Mistral. - **Open-source models on EU infrastructure**: LLaMA (Meta, US-originated but open-weight), Falcon (UAE-originated, open-weight), and community models can be deployed on EU infrastructure for full sovereignty. The model origin matters less than deployment jurisdiction for open-weight models.

**When US models may be acceptable**: - Non-sensitive workloads with documented risk acceptance - Specific capability requirements that EU models cannot meet (evaluate per use case) - Transitional deployments during migration to EU alternatives

**Key principle**: For open-weight models, sovereignty is determined by deployment location, not model origin. A Meta LLaMA model running on Hetzner in Germany is more sovereign than a proprietary EU model accessed via US cloud API.

Where AI models run is as important as which models you choose. Inference hosting determines the jurisdiction under which prompts and completions are processed.

**EU-sovereign hosting options**: - **OVHcloud** (France): Broad GPU infrastructure. Kubernetes-based ML workloads supported. EU-headquartered and EU-jurisdictioned. - **Scaleway** (France): Developer-friendly cloud with GPU instances. Strong Kubernetes support. EU-headquartered. - **Hetzner** (Germany): Excellent value GPU instances. More DIY, less managed services. EU-headquartered. - **IONOS** (Germany): Enterprise-positioned with GPU cloud. EU-headquartered. - **Self-hosted on-premises**: Maximum sovereignty. Requires GPU procurement and operational capacity.

**When US hyperscalers may be acceptable**: - Using EU regions of Google Cloud, Azure, or AWS for non-sensitive workloads with documented risk acceptance - Google sovereign cloud partnerships (T-Systems, Thales) that place operational control with EU entities - Edge cases where specific managed services (Kubernetes clusters, networking) are not available from EU providers

**Architecture recommendation**: Deploy inference on EU-headquartered cloud providers. Use containerised deployments (Docker, Kubernetes) for portability. Avoid proprietary inference APIs that create lock-in to any single provider.

Retrieval-Augmented Generation (RAG) systems require vector databases to store and query document embeddings. These embeddings are derived from organisational data and may contain semantically sensitive information even when the original documents are not stored alongside them.

**EU-sovereign options**: - **Qdrant** (Germany): Open-source vector database, self-hostable on any EU infrastructure. Managed cloud available with EU deployment. Strong performance and active development. - **Weaviate** (Netherlands): Open-source vector search engine. Self-hostable. Managed cloud with EU options. Good enterprise features. - **PostgreSQL + pgvector**: Extend existing EU-hosted PostgreSQL with vector search capability. No additional vendor dependency. Trade-off: less optimised for pure vector workloads at scale. - **Milvus** (open-source): Self-hostable vector database. Can be deployed on EU infrastructure. Note: Zilliz (commercial entity) is US-based.

**Not recommended for sovereignty-sensitive workloads**: - **Pinecone** (US): Fully managed, US-headquartered. No self-hosted option. US jurisdiction over all stored embeddings. - **Chroma** (US): US-headquartered. Open-source version is self-hostable (acceptable on EU infrastructure), but managed cloud is US-jurisdictioned.

**Architecture recommendation**: Use Qdrant or Weaviate self-hosted on EU infrastructure for maximum sovereignty. PostgreSQL + pgvector is acceptable for smaller-scale RAG deployments and avoids adding another vendor to the stack.

AI system monitoring captures prompts, completions, latency metrics, error rates, and usage patterns. This operational data can be as sensitive as the underlying business data — a log of all prompts sent to an AI model reveals what an organisation is working on, what questions it asks, and what data it processes.

**EU-sovereign options**: - **Self-hosted Grafana + Prometheus**: Open-source monitoring stack deployable on any EU infrastructure. Full control over all telemetry data. Trade-off: operational overhead. - **Elastic Stack (self-hosted)**: Elasticsearch + Kibana deployed on EU infrastructure for log aggregation and analysis. Elastic NV is Netherlands-headquartered but has US entity complexity — self-hosting resolves jurisdiction. - **VictoriaMetrics** (open-source): High-performance metrics backend. Self-hostable on EU infrastructure. Trade-off: less ecosystem tooling than Grafana/Prometheus.

**Not recommended for sovereignty-sensitive AI monitoring**: - **Datadog** (US): Comprehensive but US-headquartered. All telemetry sent to US-controlled infrastructure. - **New Relic** (US): US-headquartered. Same jurisdiction concerns as Datadog. - **Splunk** (US, Cisco-owned): US-headquartered. Enterprise monitoring under US jurisdiction.

**AI-specific monitoring**: Emerging tools like Langfuse (Germany, open-source) provide LLM-specific observability (prompt/completion logging, cost tracking, quality evaluation) and can be self-hosted on EU infrastructure. This is a critical and often overlooked sovereignty gap.

**Architecture recommendation**: Deploy Grafana + Prometheus for infrastructure monitoring and Langfuse (self-hosted) for AI-specific observability. Keep all telemetry on EU infrastructure.

The identity layer determines who can access AI systems and how authentication is managed. This layer often receives less sovereignty scrutiny than model selection, but it controls access to all other layers.

**EU-sovereign options**: - **Keycloak** (open-source, Red Hat): Self-hostable identity and access management. Full OIDC and SAML support. Deploy on EU infrastructure for sovereign identity. Trade-off: operational complexity. - **Authentik** (open-source): Modern identity provider, self-hostable. Good UX, growing community. EU-deployable. - **Organisation's existing EU-hosted identity**: Many organisations already run Active Directory or LDAP on-premises. Extending this to AI systems keeps identity sovereign.

**Acceptable with caveats**: - **Microsoft Entra ID (Azure AD)**: Widely used, US-jurisdictioned. Acceptable if the organisation has already accepted Microsoft sovereignty risk for identity and the AI system is not processing data more sensitive than other Entra-protected systems. - **Google Identity**: Similar considerations to Entra ID. US-jurisdictioned.

**Architecture recommendation**: For organisations building new AI infrastructure, Keycloak on EU infrastructure provides the most sovereignty-aligned identity layer. For organisations already committed to Entra ID, extending it to AI systems does not materially increase existing sovereignty exposure.

A fully EU-sovereign AI stack is achievable but involves trade-offs. For many organisations, a pragmatic approach accepts non-EU components at specific layers with documented risk assessment.

**Generally acceptable** (with documentation): - Open-source frameworks and libraries (PyTorch, TensorFlow, LangChain) — code is inspectable and runs on your infrastructure - Open-weight models from non-EU origins deployed on EU infrastructure - Development tools (IDEs, linting, formatting) that do not process production data - Package registries (npm, PyPI) for non-sensitive build dependencies

**Acceptable with formal risk assessment**: - US hyperscaler EU regions for non-sensitive compute (with understanding that jurisdiction is not resolved by region selection) - US-originated open-source databases self-hosted on EU infrastructure - CDN services from non-EU providers for public-facing, non-sensitive content

**Not acceptable for sovereignty-sensitive AI workloads**: - Proprietary US AI APIs for processing sensitive data without risk acceptance - US-hosted vector databases storing embeddings of sensitive documents - US-hosted monitoring capturing AI prompts and completions - Non-EU identity providers for AI systems processing regulated data

**The principle**: Sovereignty is about control over data, not purity of software origin. Open-source software deployed on EU infrastructure is sovereign regardless of where it was developed.

AI sovereignty is not a one-time architectural decision. The landscape evolves continuously — new providers emerge, regulations change, and existing providers alter their terms and jurisdictions.

**Quarterly review cadence**: - Reassess each stack layer for sovereignty posture changes - Evaluate new EU-native providers that may fill gaps - Review regulatory developments (EU AI Act enforcement, NIS2 implementation) - Update risk documentation for any accepted non-EU components

**Trigger-based review**: - Provider acquisition or ownership change (sovereignty status may change) - New regulation enforcement (new compliance requirements) - Data classification changes (previously non-sensitive data reclassified) - Security incidents involving stack components

**Documentation requirements**: - Maintain an AI sovereignty register listing each stack layer, provider, jurisdiction, and risk assessment - Update after each review cycle - Include in broader IT governance and procurement documentation

**Emerging considerations**: - EU AI Act enforcement timeline and obligations for deployers - European Chips Act impact on GPU availability for self-hosted inference - Evolving EU data sovereignty certification schemes (EUCS) - New EU-native AI providers entering the market

**The goal is not perfection but continuous improvement** — each review cycle should identify at least one sovereignty improvement opportunity and assess whether previously accepted risks remain acceptable.

Adopting a sovereign AI stack is a multi-phase undertaking. Most organisations should plan for 6–12 months for initial deployment, with continuous refinement thereafter.

**Phase 1 — Discovery & Risk Mapping** (3–4 weeks): Inventory all AI-related components across the five stack layers. Classify each by jurisdiction, data sensitivity, and business criticality. Identify the highest-risk components. What usually breaks: discovering undocumented AI usage and shadow IT AI tools. Risk reduced: visibility into AI sovereignty exposure.

**Phase 2 — Architectural Redesign** (3–4 weeks): Design the target sovereign stack using this guide as a reference architecture. Select providers for each layer. Define the migration sequence (typically: monitoring first, then vector storage, then inference, then models). What usually breaks: underestimating infrastructure requirements for self-hosted components. Risk introduced: new infrastructure management responsibility.

**Phase 3 — Data & Model Transition** (6–10 weeks): Deploy sovereign infrastructure. Migrate vector data to EU-hosted databases. Transition inference to EU providers. Validate model performance against production requirements. What usually breaks: data migration from proprietary vector formats, prompt engineering differences. Risk reduced: data jurisdiction exposure for migrated components.

**Phase 4 — Operational Hardening** (3–4 weeks): Implement sovereign monitoring (Grafana + Langfuse). Establish alerting, SLA baselines, and cost management. Configure disaster recovery within EU infrastructure. What usually breaks: monitoring coverage gaps in new infrastructure. Risk introduced: new operational baseline.

**Phase 5 — Ongoing Governance & Review** (continuous): Establish the quarterly review cadence. Document the sovereignty register. Train the team on governance processes. What usually breaks: governance enthusiasm fading after initial deployment. Risk reduced: continuous sovereignty posture improvement.