---
title: "Daedalus-150M：面向CPU推理的卷积-注意力混合小模型"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20210v1"
permalink: "https://feed.astralor.com/items/itm_2040d0cef604c3a2e9e56b8c"
published: "2026-08-20T16:09:43.000Z"
score: 77
channel: "ai"
tags: ["CPU推理","Daedalus-150M","卷积-注意力混合架构","小语言模型"]
---

# Daedalus-150M：面向CPU推理的卷积-注意力混合小模型

## AI 点评

> 这篇论文最值得看的地方不是刷榜，而是它把'部署约束'当成设计起点，而不是事后优化。对于做端侧AI、嵌入式或隐私计算的团队来说，'速度优势随上下文增长'这一点尤其重要——意味着对话越长，相对收益越明显，很适合长聊天和 agent 场景。如果你手上有 CPU 推理任务，不妨在真实硬件上复现一下它的带宽计算。

## 速览

Daedalus-150M是面向CPU推理的卷积注意力混合小模型。18个block中仅6个用完整注意力，其余12个用短卷积，缓存不再随上下文增长。在59.9B tokens上从头训练，五任务基准47.31，超过GPT-2 124M、Pythia-160M等更大模型，也超过MobileLLM-125M。与同尺寸全注意力模型相比，4-bit文件小6.3%，2048 tokens解码快1.76倍，且速度优势随序列长度增加。论文同时报告了失败的尝试。

## 关键点

- 架构上，18层中仅6层使用注意力，其余12层使用短卷积，缓存不随上下文增长，内存占用恒定。
- 在59.9B tokens上训练后，五任务基准得分47.31，超越多个数据量更大的同规模模型。
- 与同尺寸全注意力模型相比，4-bit模型文件小6.3%，2048 tokens上下文解码快1.76倍，速度优势随序列长度增加。
- 论文详细报告了失败设计：4-bit量化质量损失、约一半卷积通道失效、词汇表过大。

## 全文

Small language models are usually built like large ones and then squeezed onto a CPU afterwards. We did the opposite: we fixed the target first, one user, one token at a time, 4-bit weights, ordinary CPU, and chose the architecture to suit it. The result keeps full attention in only 6 of its 18 blocks. The other 12 use short convolutions whose memory is two timesteps wide no matter how long the conversation gets, so two thirds of the network never re-reads a growing cache. Trained from scratch on 59.9B tokens, the model scores 47.31 on a five-task benchmark against a bar of 42.20 that was fixed before training began. It beats GPT-2 124M, Pythia-160M, OPT-125M and GPT-neo-125M, all trained on three to six times more data, and exceeds MobileLLM-125M's published score despite that model seeing a trillion tokens. Validation bits-per-byte is 0.8685. To check the architecture rather than the training recipe, we trained a conventional all-attention model of the same size on the same data, and wrote down the winning condition before scoring either. The hybrid won the chosen quality metric by 0.81%, matched it on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76x faster at 2048 tokens of context, 2.08x against an external model of similar size. In every measurement the speed advantage is near zero at an empty context and grows with length, which is what the mechanism predicts and what a merely leaner model would not show. A simple bandwidth calculation predicts only 1.17x, so memory volume alone does not explain the gap. We also report what did not work: an unmitigated 4-bit quality cost, roughly half the convolution channels ending up inert and impossible to remove, and a vocabulary larger than this model size warrants.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_2040d0cef604c3a2e9e56b8c