.jpg)
Hey folks,
Saugat here again 👋. After Linux hardening, let's talk about something that's quietly having a huge comeback in 2026: OpenStack. Yeah, the same OpenStack a lot of us wrote off a few years back as "too complex, too much ops overhead, just use the public cloud." Turns out the story isn't over — and AI workloads are a big part of why.
Today I want to walk through why OpenStack is becoming a serious contender for AI infrastructure, how it pairs with Kubernetes, and how to actually get a GPU-ready environment running. Let's get into it.
Why OpenStack for AI, Right Now
A few things are colliding at once, and together they explain why OpenStack is suddenly relevant again.
The VMware exodus is real
Since Broadcom's acquisition of VMware and the licensing changes that followed, a huge number of organizations have been actively looking for a way out. Industry surveys show the vast majority of OpenInfra Foundation members have fielded direct requests from customers wanting to migrate off VMware — and projections suggest VMware could lose a significant chunk of its workloads within a couple of years of those licensing changes. OpenStack, as a mature, vendor-neutral alternative, is one of the biggest beneficiaries of that migration wave.
Private cloud repatriation is accelerating
It's not just VMware refugees. A meaningful share of workloads that moved to public cloud over the last decade are now being pulled back on-prem, driven by cost control, data sovereignty regulations, and the simple fact that at scale, public cloud billing for GPU-heavy workloads gets brutal fast. OpenStack deployments are projected to grow dramatically over the next few years on the back of this trend, and interestingly, the fastest-growing segment isn't the hyperscalers — it's small and mid-sized businesses feeling the most pain from licensing costs and cloud bills.
AI workloads need what OpenStack is good at
Training large models is expensive, GPU-hungry, and data-gravity-sensitive. Public cloud pricing for sustained large-scale training can be prohibitive, while a private OpenStack environment gives you direct control over GPU resource pooling, high-throughput networking, and data locality — without paying a premium for every hour a GPU sits allocated to your workload. That combination of cost control and infrastructure control is exactly why enterprises are deploying AI toolkits directly onto open infrastructure instead of renting it all from a hyperscaler.
OpenStack + Kubernetes: How the Pairing Actually Works
This is the part people get confused about — OpenStack and Kubernetes aren't competitors here, they're two different layers of the same stack.
- OpenStack provides the infrastructure foundation — compute, storage, and networking, virtualized and pooled from your physical hardware.
- Kubernetes sits on top and orchestrates the actual AI workloads — training jobs, inference services, data pipelines — using that infrastructure.
Think of OpenStack as the data center operating system, and Kubernetes as the workload scheduler running on top of it. Together, they give you something close to a self-hosted, open-source alternative to a proprietary AI cloud platform — extensible, auditable, and not locked to a single vendor's roadmap.
GPU Scheduling & Resource Management
GPUs are expensive, and wasted GPU utilization is one of the fastest ways to blow your infrastructure budget. A few practices matter a lot here:
- Tighter Nova-to-Kubernetes GPU integration. Teams are increasingly wiring OpenStack's Nova scheduler more closely with Kubernetes GPU scheduling to avoid the classic problem of GPUs sitting idle while queued jobs wait elsewhere.
- PCI passthrough or vGPU, depending on your isolation needs. Full passthrough gives maximum performance for training; vGPU slicing lets you multiplex smaller inference workloads across a single card more efficiently.
- Utilization tracking as a first-class metric. Treat GPU idle time as seriously as you'd treat unused compute — it's the most expensive resource in the whole stack.
# Example: creating a GPU-enabled flavor in OpenStack
openstack flavor create gpu-training-large \
--ram 131072 --disk 500 --vcpus 16
openstack flavor set gpu-training-large \
--property "pci_passthrough:alias"="nvidia-a100:1"Training vs Inference: Two Very Different Infrastructure Needs
This is a mistake I see teams make constantly — treating training and inference as the same workload with the same infrastructure requirements. They're not.
| Aspect | Training | Inference |
|---|---|---|
| GPU usage pattern | Sustained, high utilization, long-running jobs | Bursty, latency-sensitive, short requests |
| Networking | High-throughput, low-latency east-west traffic between nodes | Optimized for external request/response latency |
| Scaling behavior | Scheduled, predictable batch jobs | Autoscaling based on real-time demand |
| Storage priority | High-throughput read of large datasets | Fast model-load and low-latency reads |
Best practice: build separate infrastructure paths for each. Use distinct OpenStack flavors, storage tiers, and network segments for training versus inference rather than trying to force one config to do both jobs well.
Storage for AI Workloads
AI infrastructure is storage-hungry in ways that catch a lot of traditional sysadmins off guard. You're dealing with:
- Datasets — often terabytes, needing high-throughput sequential reads
- Checkpoints — large, frequent writes during long training runs; losing a checkpoint from a multi-day job is genuinely painful
- Embeddings & model artifacts — smaller individually, but requiring fast, low-latency access at inference time
In practice, this usually means tiering storage: high-throughput object storage (Swift or Ceph-backed) for datasets and checkpoints, and faster block storage close to compute for active inference serving.
Networking Considerations
Distributed training jobs are chatty — nodes are constantly synchronizing gradients across the cluster. If your network can't keep up, your expensive GPUs sit idle waiting on data, which defeats the entire point of buying them.
- Use dedicated high-throughput network segments for east-west training traffic, separate from your general management network.
- Consider SR-IOV or DPDK-backed networking for latency-sensitive inference paths.
- Segment training, inference, and management traffic with distinct Neutron networks rather than flattening everything onto one.
Step-by-Step: Setting Up a Minimal GPU-Ready OpenStack Environment
This is a simplified walkthrough — enough to get a working GPU pool going in a lab or small production environment. Adjust for your actual hardware and scale.
1. Confirm GPU passthrough support on your hypervisor host:
lspci | grep -i nvidia
sudo dmesg | grep -i iommu2. Enable IOMMU in your kernel boot parameters (GRUB):
GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt"
sudo update-grub && sudo reboot3. Configure Nova to recognize the GPU as a PCI device:
# /etc/nova/nova.conf on the compute node
[pci]
passthrough_whitelist = { "vendor_id": "10de", "product_id": "20b0" }
alias = { "vendor_id":"10de", "product_id":"20b0", "device_type":"type-PCI", "name":"nvidia-a100" }4. Create a GPU-enabled flavor:
openstack flavor create gpu.medium --ram 65536 --disk 200 --vcpus 8
openstack flavor set gpu.medium --property "pci_passthrough:alias"="nvidia-a100:1"5. Launch an instance using that flavor:
openstack server create --flavor gpu.medium \
--image ubuntu-22.04-cuda \
--network training-net \
gpu-worker-016. Verify GPU visibility inside the instance:
nvidia-smiFrom here, you'd typically layer Kubernetes on top (via Magnum, Cluster API, or a separate bootstrap) with the NVIDIA device plugin installed, so your training and inference workloads can actually request GPU resources through standard Kubernetes scheduling.
Common Pitfalls & Best Practices
- Don't skip capacity planning. GPU procurement lead times are long — plan pool sizing months ahead, not when the training queue is already backed up.
- Don't treat storage as an afterthought. Undersized storage throughput is one of the most common causes of GPU idle time in real deployments.
- Isolate tenants properly. If multiple teams share the same OpenStack environment, use projects/quotas to prevent one team's training job from starving everyone else's GPU access.
- Automate upgrades. OpenStack's reputation for operational complexity is largely earned by teams doing manual, ad-hoc upgrades. Use a proper deployment tool (Kolla-Ansible, OpenStack-Ansible, or a managed distribution) instead of hand-rolling it.
- Consider simplified distributions if you're small. Projects like Canonical's Sunbeam exist specifically to cut down OpenStack's historically steep operational overhead — worth evaluating before you commit to a from-scratch deployment.
FAQ
Can OpenStack actually run AI workloads well?
Yes — when paired with Kubernetes for orchestration and configured with proper GPU passthrough/scheduling, OpenStack is increasingly used in production for both training and inference, including by organizations running well over a million CPU cores in production globally.
Is OpenStack better than public cloud for AI infrastructure?
It depends on scale and workload pattern. For sustained, large-scale training, OpenStack's private infrastructure model typically wins on cost and data control. For bursty, unpredictable workloads, public cloud's elasticity can still make more sense. Many organizations end up hybrid — private OpenStack for steady-state training, public cloud for burst capacity.
Do I need Kubernetes to run AI on OpenStack?
Not strictly, but it's strongly recommended. You can run AI workloads directly on OpenStack VMs, but you lose the workload orchestration, autoscaling, and portability that Kubernetes provides on top.
Is OpenStack still as hard to operate as its reputation suggests?
It's gotten meaningfully easier thanks to simplified distributions and better automation tooling, but it's still more operationally involved than a fully managed public cloud service. Budget for real ops investment, especially at the start.
Wrapping Up
OpenStack isn't the "dying project" a lot of us assumed it was a few years back. Between the VMware exodus, private cloud repatriation, and the sheer cost pressure of running AI workloads on public cloud at scale, it's finding a genuinely strong second wind — and this time, AI is the reason people are taking a second look.
If you're evaluating this for your org: start small, get GPU scheduling and storage tiering right before you scale up, and don't be afraid to lean on managed OpenStack services or simplified distributions if your team doesn't have deep OpenStack ops experience yet.
That's it for today, folks. If you're already running AI workloads on OpenStack, I'd love to hear what's working for you in the comments — and if you're just getting started, drop your questions, I'll do my best to help.
