TL;DR
We compress LLMs for customers, and we need to do it fast, on many kinds of GPUs, without losing track of anything. We built a small, cheap infrastructure setup on two tools: SkyPilot to find and manage GPUs across clouds, and MLflow to track experiments and store models. On-demand GPU clusters with a fresh install of our stack mean our newest research reaches customer projects right away. Here is how it works.
The problem: move fast without the chaos
A deeptech startup lives, well, off its tech. At Ora Computing, we built an LLM compression stack that combines pruning, quantization, and retraining — delivering small, fast, efficient models tailored to each customer's needs. Our research team keeps improving and extending this stack. They develop new compression algorithms, adapt them to a variety of model architectures, and optimize them for whatever hardware is in demand, both during compression and at deployment. If you want to read more about the compression tech itself, see other posts in our blog. This post is about something else: the infrastructure that lets us ship it fast.
The power of a startup is its agility. It moves fast, and changes get integrated quickly. But how do you get the hottest tech into customers' hands fast, and how do you make sure that agility doesn't turn into chaos? For that, you need a scalable and flexible ML infrastructure. Scalable means you can run many compression workflows for many customers with little overhead. Flexible means the workflows can adapt to the newest developments from the research team. Fast iterations between hands-on project work and research make sure customers get the best possible outcome in the fast-moving world of LLM compression, at all times. Here is how we do it.
Finding GPUs is a pain. SkyPilot does it for us.
At Ora Computing, we need access to a whole zoo of different GPUs. For testing, a cheap and small GPU will do. For large-scale training, we might need nodes with dozens of datacenter GPUs. To test deployment of exotic quantized models, we need newest-generation GPUs. And if a customer is driven by hardware limits or needs, we have to test our models under exactly the same restrictions.
On top of that, GPU scarcity is real. Depending on the model, you have to skim dozens of regional datacenters across different cloud providers, limited by both your own quota and the provider's availability. This is cumbersome and frustrating. So we hand this job to SkyPilot, a trusted and widely used GPU orchestrator. We run a central SkyPilot server that all our engineers and researchers can access. They simply ask for the type and amount of GPUs they want, SkyPilot lists the options with prices, and it keeps trying to fetch them until they are available. Once they are, our people have exactly the resources they need. A single SkyPilot launch configuration lets you spin up as many GPU clusters as you want (if you can afford them...). SkyPilot also manages the lifetime of GPUs, stopping them when you don't need them. The days are over when a training run that crashed at midnight quietly burns hundreds of bucks before you notice it in the morning.
Keeping our newest research one command away
But how do we make sure the newest developments from the research team reach customer projects right away? We keep our development stack lean and fast to build. When we provision on-demand GPUs, the Ora stack gets pulled in its freshest version, so the newest algorithms are available immediately. We do this with prebuilt wheels for compilation-heavy dependencies and uv as a fast, efficient Python dependency manager. For interactive development, we use the same setup as in production, which makes deployment fast and (in most cases) smooth. Running on-demand GPU clusters with fresh environment setups keeps our development code from drifting too far from production.
Keeping experiments organized, reproducible, and safe
Now that it's simple for our team to spin up the GPUs they need with the stack they need, they can run many experiments on many clusters. That means a ton of new models and plenty of insights — and it leads to the next problem. How do you keep the research organized, reproducible, and shareable? How do you back up models and insights?
Enter MLflow, one of the most widely used open-source ML platforms. Together with SkyPilot, it makes compressing models at scale easy and safe. SkyPilot hands each new GPU worker its authentication credentials and the MLflow server address. Our compression pipeline triggers automatic tracking in MLflow, which records all training and system metrics, plus internal metadata, so runs stay reproducible. Output models are automatically synced to an ephemeral bucket, where they live together with compression checkpoints for three months. Only the successful models get promoted and moved to a versioned bucket. With this, our researchers and engineers can focus on their work instead of worrying about where to track runs or store models. The central MLflow server also gives everyone fast, shareable insight into project progress.
The result
To wrap up: we run a cheap, minimal setup built on a SkyPilot server and an MLflow server. A lean stack and on-demand GPU clusters mean our newest developments are available immediately. GPU orchestration takes the hassle of hardware provisioning off our team's hands, and automated experiment tracking and model syncing let us apply our newest tech to many projects at once.
That is the part customers don't see, but feel: we don't just build strong compression tech, we can put it to work for them fast, and at scale.
You can see the result on our models page — the compressed models we ship, small enough to run on a single GPU. Many of them are free to download on Hugging Face. Go try them.