OmniDelta: Skill-Driven Budget Allocation for Token Compression in OmniLLMs

Omni-modal Large Language Models (e.g., Qwen2.5-Omni) can process text, audio, and video simultaneously, but long audio/video token sequences lead to high memory and inference latency. Existing compression methods primarily focus on selecting important tokens under a fixed budget (e.g., via attention scores), but ignore the more fundamental budget allocation problem: how many retained tokens should be allocated to audio versus video? Through experiments, the paper shows that direct query-to-audio/video similarity is unreliable for inter-modal allocation, and uniform intra-modal allocation can discard key evidence while retaining redundant content. To address these limitations, the authors propose OmniDelta, a training-free, skill-driven framework. First, audio and video skill pools are constructed by inferring required skills from the query (e.g., speech recognition, action detection), which shift the initial fixed budget across modalities. Then, within each modality, local complexity and temporal redundancy are used to reallocate budgets across audio segments and video frames. The resulting local budgets can be seamlessly integrated with existing pruning strategies (e.g., average pooling or top-k selection) while preserving the total retained-token ratio. Evaluated on four audio-video benchmarks (AudioSet, VideoQA, etc.) using Qwen2.5-Omni-7B and Qwen2.5-Omni-3B, OmniDelta outperforms uniform and attention-based allocation across pruning ratios, establishing a new Pareto frontier. At 25% token retention, GPU memory is reduced by 22% and inference speed increases by 1.64× with minimal accuracy loss. This work provides a promising direction for efficient deployment of OmniLLMs.