---
title: "Learning to Modulate, Not to Cycle: Soft Actor-Critic Recovers Inverter-Style Heat-Pump Control"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.09453v1"
permalink: "https://feed.astralor.com/items/itm_d97c8f81e0d61b73b3827b23"
published: "2026-08-10T11:24:29.000Z"
score: 72
channel: "tech"
tags: ["BOPTEST","PPO","Soft Actor-Critic","Compressor Wear","Reinforcement Learning","Heat Pump Control"]
---

# Learning to Modulate, Not to Cycle: Soft Actor-Critic Recovers Inverter-Style Heat-Pump Control

## AI commentary

> This paper reminds us that reward engineering should include equipment-lifetime costs, not just macro KPIs. For building-energy RL and heat-pump control practitioners, the SAC-PPO gap is a pitfall worth knowing: not every policy optimization algorithm handles wear constraints in continuous action spaces well.

## At a glance

Reinforcement learning controllers for building HVAC systems typically optimize energy cost and thermal comfort while ignoring the wear caused by frequent compressor on-off cycling. This paper compares SAC and PPO on the BOPTEST hydronic heat pump benchmark with a compressor-wear penalty added to the reward. SAC learns a continuous modulation policy that keeps the compressor permanently engaged, achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles even more than the baseline. On the BOPTEST emulator, the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling. These results highlight that algorithm choice is critical when integrating equipment-lifetime costs into RL-based building control.

## Key points

- SAC learns a continuous modulation policy keeping the compressor permanently engaged, achieving zero start-ups per day on the same MDP.
- PPO collapses to bang-bang control, cycling more than the baseline heuristic policy.
- The SAC policy cuts thermal discomfort by up to 90.7% for only an 11.5% cost increase.
- The study uses the BOPTEST bestest hydronic heat pump case with a compressor-wear term added to the reward.

## Full text

On--off cycling is the main cause of compressor wear in residential heat pumps, yet reinforcement learning (RL) controllers for buildings typically optimise only energy cost and thermal comfort, ignoring how much the learned policy cycles. We add a levelised compressor-wear term to the control reward and study how the resulting behaviour depends on the RL algorithm. Training Soft Actor---Critic (SAC) and Proximal Policy Optimisation (PPO) on an identical Markov decision process for the BOPTEST bestest hydronic heat pump case, we find that SAC learns a continuous modulation policy that keeps the compressor permanently engaged---the operating principle of an inverter-driven heat pump---achieving zero start-ups per day, whereas PPO collapses to bang-bang control that cycles more than the baseline. On the BOPTEST emulator the SAC policy cuts thermal discomfort by up to 90.7% for an 11.5% cost increase, while eliminating all baseline cycling.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d97c8f81e0d61b73b3827b23