Bringing Datacenter-Grade AI to Edge Devices

In this post, we share the results of our Proof of Concept conducted with our partners in France, Hub Quantique, Maison du Quantique Alpes and CEA (the French Alternative Energies and Atomic Energy Commission), in which we compressed Qwen3-Omni-30B-A3B-Instruct, a state-of-the-art multimodal AI model, enough to run it affordably on compact edge devices from NVIDIA Jetson series.

Introduction

Speech assistants are becoming a standard way to interact hands-free with robots, vehicles, aerospace platforms, and industrial equipment: dispatching a robot, setting a route, querying a sensor, sending a message, or handling the kind of small task that would otherwise mean taking your eyes and hands off the job. Running the assistant on-device brings several advantages compared to cloud deployment: lower latency, data that never leaves the premises, and an assistant that keeps working even without a network connection, whether at a remote site, deep inside a factory or warehouse, in a tunnel, or simply an area with poor coverage.

The obstacle is model size. A state-of-the-art model cannot simply be dropped into a robot, vehicle, aircraft, or industrial machine: the hardware needed to run it is either too large or too expensive for embedded systems. Take Qwen3-Omni-30B-A3B-Instruct. On paper it is a strong candidate for such platforms, an omnimodal mixture-of-experts (MoE) model that handles speech, vision, and language in one system. But it requires over 70 GB of memory, which in practice means an onboard device costing almost $4,000, a significant share of the total cost of a robot, vehicle, or industrial machine. Without heavy compression, a model like this simply cannot be deployed economically.

Proof of Concept

This is the challenge Ora Computing addressed during a Proof of Concept with a major European industrial partner, within the exploration framework of Hub Quantique, coordinated by CEA, together with Maison du Quantique Alpes. We made it fit, without hampering it: we reduced the memory footprint of Qwen3-Omni by more than 70% while retaining the model's accuracy at 95.2% of the original model. In this technical post we walk through the Proof of Concept journey and explain how we made it happen.

Why on-device, and why it's hard

So how do you take a 70 GB model and fit it onto hardware built for robots, vehicles, aircraft, or industrial machines? The reference target in our Proof of Concept was the NVIDIA Jetson AGX Orin, a capable edge chip that ships with 32 or 64 GB of memory and costs from $1,250 to $2,300. This device is generous for an embedded environment, but a fraction of what the full model needs.

Shrinking a model comes down to two levers (see Figure 2). You can lower the precision of each weight, storing numbers in fewer bits: this is quantization. Or you can reduce the total number of parameters the model carries: this is pruning. Both shrink the footprint, and both, pushed too far, degrade the model.

It would be tempting to take an open-source quantization method and simply turn the dial. At modest compression rates, that works perfectly fine. But at the compression ratio needed to fit Qwen3-Omni onto a Jetson device, the standard algorithms break down.

Before showing how this happens, it helps to know the three benchmarks we tracked, each targeting a different ability that assistants embedded in robots, vehicles, aircraft, and industrial systems actually need. Two of them, BFCL and FDB, measure simple abilities:

• BFCL (Berkeley Function Calling Leaderboard) measures tool calling: whether the model picks the right function and fills in the right arguments, an essential ability for setting a route, sending a message, querying a sensor, or triggering a machine action;
• FDB (Full-Duplex-Bench) measures spoken conversation: staying relevant when a user pauses, interrupts, or changes their mind mid-sentence.

The third, OmniGAIA, measures long-horizon reasoning: the model's perception across vision, audio, and language and its ability to chain together several tool calls to reach an answer. OmniGAIA is the most difficult of the three, and particularly fragile under compression.

Figure 3 shows what happens when we compress the model with a standard, open-source quantizer (INT4, built on a state-of-the-art technique called GPTQ). The model's simple abilities barely change: both BFCL and FDB sit right at the baseline across the board. Long-horizon reasoning, however, crucial for real navigation and multi-step operational requests, tells a very different story. Without preceding pruning, quantization already drags OmniGAIA down to 73.6% of the baseline, and it gets even worse once pruning is applied before quantizing: reasoning drops to 48.1% of the baseline by the time we have removed 30% of the parameters.

Therefore, shrinking a model this far without hollowing it out is not one clever trick. It takes a stack of techniques designed to work together.

The Ora stack: OraPrune → OraTrain → OraQuant

Ora Computing has built novel algorithms for both levers. For pruning, OraPrune uses quantum-inspired methods to remove parameters with minimal accuracy loss. For quantization, OraQuant performs mixed-precision quantization: important weight matrices are kept at higher precision, while less critical ones are pushed to low bit-widths, preserving overall model quality.

As shown in Figure 4, compressing Qwen3-Omni runs in three stages. We first apply OraPrune to remove redundant parameters; here we consider 30% pruning, which corresponds to removing 35% of parameters from each of the MoE layers. We then run OraTrain, a light retraining pipeline that recovers the accuracy lost during pruning. Finally, we apply OraQuant to produce the compressed model checkpoints.

Calibration data is what ties the stages together. Both OraPrune and OraQuant decide what to remove based on how the model behaves on representative input, so the data mixture matters as much as the algorithms. Crucially, each stage's data is chosen in light of the previous one: OraTrain's mixture is tuned to recover exactly what pruning removed, and OraQuant's calibration follows the same principle.

Results

Delivered models

In the course of this Proof of Concept, we pruned, retrained, and quantized the Qwen3-Omni model. As a result, we delivered three models:

^*The FP8 model is included for a formal comparison only. This format cannot be deployed on a Jetson-type device.

An important difference is that for the INT4 model we used our in-house quantization technique, while for the FP8 model we used the static (data-aware) quantization available in llm-compressor, a standard library for compressing LLMs.

In Figure 5 we show where the Ora models land by plotting benchmark scores against disk size. The first result is reassuring: tool calling and conversation survive compression almost untouched. Across the pruned, FP8, and INT4 variants, BFCL stays within a few points of the baseline, and FDB even improves. This fits their nature as simple, short-horizon tasks: each one is resolved in a step or two, so a small error introduced by compression has no long chain of later steps to accumulate.

Reasoning is the sensitive axis. On OmniGAIA, pruning alone brings the model to 89.4% of the baseline's performance, a modest, expected cost for removing a third of the parameters. Quantization is where it gets interesting. The FP8 model, despite being the less aggressive of our two quantized variants, drops furthest, to 80.3% of the baseline. The reason is that FP8 quantizes both weights and activations, which particularly damages long reasoning chains where errors compound across dozens of steps.

Which sets up the most striking result. Standard INT4 quantization falls off a cliff on OmniGAIA: the 30%-pruned model, quantized with an open-source GPTQ, collapses to 48.1% of the baseline's score (see Figure 3). Our OraQwen3-Omni-INT4 model, starting from the same pruned weights and quantizing to the same bit-width, recovers the lost reasoning, landing at 90.9% of the baseline. That is better than the open-source llm-compressor FP8 variant, and it even edges out the pruned-and-retrained model before quantization. Same bit-width, same starting model: OraQuant is the entire difference between a model that reasons and one that collapses. This is the clearest evidence that a co-designed Ora stack beats open-source tools.

Where do the delivered models fit?

Now we assess the deployability of the produced models in terms of their memory requirements. In Figure 1 we compare the total memory footprints of the baseline and compressed models against the available device memory budgets. For each model we include the model's weights (colored bars) and the maximal KV-cache measured over OmniGAIA's tasks (grey segment on top), giving a rough estimate of how much memory the model needs in use. As reference deployment devices we take NVIDIA Jetson AGX Orin chips at 32 GB and 64 GB, plus the NVIDIA Jetson AGX Thor chip at 128 GB. Their budgets are drawn as dashed lines.

As the figure shows, the baseline model with its KV cache can only fit the expensive Thor device, while the models produced by Ora Computing fit the cheaper and smaller Orin-series devices. The pruned-and-retrained and FP8 variants fit on the NVIDIA Jetson AGX Orin 64 GB, and our INT4 model fits on an even smaller 32 GB device while almost fully retaining the model's abilities (see Figure 5). The 4.3 GB KV cache shown corresponds to roughly a 65k-token context length.

Conclusions

By reducing Qwen3-Omni's memory footprint from 70.5 GB to 20.9 GB, a 70.4% reduction, Ora Computing made the model deployable on an NVIDIA Jetson AGX Orin 32 GB, an edge chip the original model was far too large to run on. And we did it while retaining the drop of overall benchmark performance within 4.8% (for BFCL, FDB, OmniGAIA). Standard open-source solutions, by contrast, degrade the model so severely at this compression level that it becomes unusable.

The reason is that open-source tools address compression components in isolation. Reaching production-ready compression that retains the model's abilities requires synergistically combining pruning, quantization, and retraining. What makes the Ora Computing stack work is not that it bundles several techniques together, but that each stage uses novel algorithms that are co-designed and coupled through the calibration data that flows between them (see Figure 4). The information removed during pruning determines the data used for retraining, and the retrained model's benchmark performance in turn determines the data used for quantization. Each stage is conditioned on what the previous stage actually did. This synergy is what allows for the impressive retention of the model's abilities.