Skip to content

Commit b26f447

Browse files
fangvvfangvv
committed
update README
1 parent 6f3ebca commit b26f447

1 file changed

Lines changed: 153 additions & 19 deletions

File tree

‎README.md‎

Lines changed: 153 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,172 @@
1-
## HADRL
1+
# HADRL
22

33
This is the source code for our paper: **Joint Service Caching and Computation Offloading in Mobile Edge Networks: A Hierarchical DRL Approach with Active Inference**. A brief introduction of this work is as follows:
44

5-
> Mobile edge computing (MEC) is a promising paradigm that provides abundant computation and storage resources at the edge close to mobile devices (MDs). In MEC networks, MDs offload compute-heavy tasks to nearby edge servers (ESs) for delay-sensitive processing, where relevant services are stored to support task execution. However, the limited computation and storage capacities of ESs make joint optimization of service caching and computation offloading challenging due to coupled decisions, a large solution space, and dynamic environments. In this paper, we investigate the joint optimization of service caching and computation offloading in MEC networks, aiming to maximize the cache hit ratio and minimize the average service latency. To tackle this problem, the original formulation is decomposed into two hierarchical subproblems, namely high-level service caching and low-level computation offloading. We propose a novel hierarchical deep reinforcement learning (DRL) algorithm with active inference, termed HADRL. At the high-level, we adopt a deep deterministic policy gradient (DDPG) based DRL approach to maximize the cache hit ratio. At the low-level, we employ an active inference based DRL approach to minimize the average service latency. Unlike conventional DRL, the active inference based DRL approach selects policies by minimizing expected free energy instead of relying only on explicit rewards, making it well suited for highly dynamic low-level computation offloading. According to the simulation outcomes, the HADRL scheme surpasses the benchmark algorithms with respect to cache hit ratio as well as average service latency.
5+
Mobile edge computing (MEC) is a promising paradigm that provides abundant computation and storage resources at the edge close to mobile devices (MDs). In MEC networks, MDs offload compute-heavy tasks to nearby edge servers (ESs) for delay-sensitive processing, where relevant services are stored to support task execution. However, the limited computation and storage capacities of ESs make joint optimization of service caching and computation offloading challenging due to coupled decisions, a large solution space, and dynamic environments. In this paper, we investigate the joint optimization of service caching and computation offloading in MEC networks, aiming to maximize the cache hit ratio and minimize the average service latency. To tackle this problem, the original formulation is decomposed into two hierarchical subproblems, namely high-level service caching and low-level computation offloading. We propose a novel hierarchical deep reinforcement learning (DRL) algorithm with active inference, termed HADRL. At the high-level, we adopt a deep deterministic policy gradient (DDPG) based DRL approach to maximize the cache hit ratio. At the low-level, we employ an active inference based DRL approach to minimize the average service latency. Unlike conventional DRL, the active inference based DRL approach selects policies by minimizing expected free energy instead of relying only on explicit rewards, making it well suited for highly dynamic low-level computation offloading. According to the simulation outcomes, the HADRL scheme surpasses the benchmark algorithms with respect to cache hit ratio as well as average service latency.
66

7-
> 移动边缘计算（MEC）是一种前景广阔的计算范式，能够在靠近移动设备（MD）的网络边缘提供丰富的计算与存储资源。在MEC网络中，移动设备将计算密集型任务卸载到邻近的边缘服务器（ES）上进行时延敏感型处理，边缘服务器中需存储相关服务以支撑任务执行。然而，边缘服务器有限的计算与存储能力，使得服务缓存与计算卸载的联合优化面临决策相互耦合、解空间庞大以及环境动态变化等挑战。本文针对MEC网络中服务缓存与计算卸载的联合优化问题展开研究，旨在最大化缓存命中率并最小化平均服务时延。为解决该问题，我们将原始问题分解为两个分层子问题，即高层服务缓存与低层计算卸载。我们提出一种新颖的基于主动推理的分层深度强化学习（DRL）算法，称为HADRL。在高层，采用基于深度确定性策略梯度（DDPG）的DRL方法最大化缓存命中率；在低层，采用基于主动推理的DRL方法最小化平均服务时延。与传统DRL不同，基于主动推理的DRL方法通过最小化期望自由能来选择策略，而非仅依赖显式奖励信号，这使得其非常适合高度动态的低层计算卸载场景。仿真结果表明，HADRL方案在缓存命中率和平均服务时延方面均优于基准算法。
7+
移动边缘计算（MEC）是一种前景广阔的计算范式，能够在靠近移动设备（MD）的网络边缘提供丰富的计算与存储资源。在MEC网络中，移动设备将计算密集型任务卸载到邻近的边缘服务器（ES）上进行时延敏感型处理，边缘服务器中需存储相关服务以支撑任务执行。然而，边缘服务器有限的计算与存储能力，使得服务缓存与计算卸载的联合优化面临决策相互耦合、解空间庞大以及环境动态变化等挑战。本文针对MEC网络中服务缓存与计算卸载的联合优化问题展开研究，旨在最大化缓存命中率并最小化平均服务时延。为解决该问题，我们将原始问题分解为两个分层子问题，即高层服务缓存与低层计算卸载。我们提出一种新颖的基于主动推理的分层深度强化学习（DRL）算法，称为HADRL。在高层，采用基于深度确定性策略梯度（DDPG）的DRL方法最大化缓存命中率；在低层，采用基于主动推理的DRL方法最小化平均服务时延。与传统DRL不同，基于主动推理的DRL方法通过最小化期望自由能来选择策略，而非仅依赖显式奖励信号，这使得其非常适合高度动态的低层计算卸载场景。仿真结果表明，HADRL方案在缓存命中率和平均服务时延方面均优于基准算法。
88

9-
This work will be published by IEEE Internet of Things Journal. Click [here](https://doi.org/10.1109/JIOT.2026.3695862) for our paper.
9+
This work will be published by IEEE Internet of Things Journal. Click [here](https://doi.org/10.1109/JIOT.2026.3695862) for our paper online.
1010

1111
## Required software
1212

13-
PyTorch 1.10.0
13+
- PyTorch 1.10.0
1414

15-
代码文件命名中ESX表示有该场景中有X个边缘服务器，默认2个。
15+
- NumPy
16+
17+
- Matplotlib
18+
19+
> 代码文件命名中 ESX 表示该场景中有 X 个边缘服务器，默认 2 个。
20+
21+
## Project Structure
22+
23+
```
24+
HADRL/
25+
├── active_rl/ # Active inference modules (low-level policy)
26+
│ ├── buffer.py # Replay buffer for the ensemble / reward model
27+
│ ├── measures.py # Active-inference related measures
28+
│ ├── models.py # Ensemble dynamics model & reward model
29+
│ ├── normalizer.py # State / action normalizer
30+
│ ├── planner.py # Cross-entropy method (CEM) planner
31+
│ └── trainer.py # Trainer for ensemble + reward model
32+
├── model_ddpg.py # DDPG agent (high-level service caching policy)
33+
├── Replay_buffer.py # Replay buffer for DDPG
34+
├── arguments.py # Hyper-parameters & runtime arguments
35+
├── environment_ES2.py # MEC environment with 2 edge servers
36+
├── environment_ES3.py # MEC environment with 3 edge servers
37+
├── environment_ES4.py # MEC environment with 4 edge servers
38+
├── environment_ES5.py # MEC environment with 5 edge servers
39+
├── main_active_ES2.py # Main training script for the 2-ES scenario
40+
├── main_active_ES3.py # Main training script for the 3-ES scenario
41+
├── main_active_ES4.py # Main training script for the 4-ES scenario
42+
├── main_active_ES5.py # Main training script for the 5-ES scenario
43+
└── README.md
44+
```
45+
46+
## Core Modules
47+
48+
### MEC Environment (`environment_ESX.py`)
49+
50+
The environment class that models the MEC system with multiple edge servers. The `ESX` suffix denotes the number of edge servers in the scenario (default `X = 2`). Key responsibilities include:
51+
52+
- Modeling the mobile devices (MDs), edge servers (ESs), wireless channels and computation / caching resources.
53+
- Tracking the high-level service caching decisions made by the DDPG agent.
54+
- Tracking the low-level computation offloading, bandwidth and resource allocation decisions made by the active inference agent.
55+
- Computing the cache hit ratio and average service latency as the optimization objectives.
56+
- Returning the observation, reward and termination signal at each MDP step.
57+
58+
> The state / action dimensions, number of MDs / ESs and other scenario parameters are configured in `environment_ESX.py` and can be tuned for different experimental settings.
59+
60+
### High-level DDPG Agent (`model_ddpg.py`)
61+
62+
The DDPG agent handles the **high-level service caching** decisions, aiming to maximize the cache hit ratio.
63+
64+
**Network architecture:**
65+
66+
| Network | Layers | Output Activation |
67+
|---|---|---|
68+
| Actor | `state_dim` → `num_units_1` → `num_units_2` → `action_dim` | tanh |
69+
| Critic | `[state, action]` → `num_units_1` → `num_units_2` → `1` | linear (Q-value) |
70+
71+
**Key hyperparameters (default in `arguments.py`):**
72+
73+
| Parameter | Value | Description |
74+
|---|---|---|
75+
| `lr_a = 0.001` | Learning rate for Actor | |
76+
| `lr_c = 0.002` | Learning rate for Critic | |
77+
| `gamma = 0.95` | Discount factor | |
78+
| `tau = 0.005` | Soft target update rate | |
79+
| `batch_size = 64` | Training batch size | |
80+
| `memory_size = 500000` | Experience replay buffer size | |
81+
| `update_iteration = 10` | Number of gradient updates per step | |
82+
| `num_units_1 = 256` | Number of units in MLP layer 1 | |
83+
| `num_units_2 = 128` | Number of units in MLP layer 2 | |
84+
85+
**Key methods:**
86+
87+
- `select_action(state)` — Forward pass through the Actor network to produce a deterministic caching action.
88+
89+
- `update(args)` — Sample a batch from the replay buffer, compute the critic / actor losses, perform back-propagation and soft-update the target networks with rate `tau`.
90+
91+
- `save(args)` / `load(args)` — Persist / restore the Actor and Critic networks.
92+
93+
- `Replay_buffer.Replay_buffer` — First-in-first-out experience replay buffer with `push()` and `sample()` for DDPG training.
94+
95+
### Low-level Active Inference Agent (`active_rl/`)
96+
97+
The active inference agent handles the **low-level computation offloading** decisions, aiming to minimize the average service latency. Unlike conventional DRL, it selects policies by minimizing the expected free energy.
98+
99+
**Sub-modules:**
100+
101+
- `models.py` — Implements the `EnsembleModel` (an ensemble of probabilistic dynamics networks) and the `RewardModel`, which are used to predict the next state and reward.
102+
- `buffer.py` — Stores the transitions used to train the ensemble / reward models.
103+
- `normalizer.py` — Normalizes state and action values to stabilize the training of the ensemble networks.
104+
- `planner.py` — Cross-Entropy Method (CEM) planner that rolls out candidate action sequences through the learned ensemble, scores them using expected free energy, and returns the best action sequence.
105+
- `trainer.py` — Trains the ensemble dynamics model and the reward model on the replay buffer.
106+
- `measures.py` — Helper functions implementing active-inference related measures (e.g., epistemic value, information gain).
107+
108+
**Key hyperparameters (default in `arguments.py`):**
109+
110+
| Parameter | Value | Description |
111+
|---|---|---|
112+
| `ensemble_size = 20` | Number of models in the dynamics ensemble | |
113+
| `hidden_size = 400` | Hidden size of each ensemble network | |
114+
| `plan_horizon = 20` | Length of the planned action sequence (CEM horizon) | |
115+
| `optimisation_iters = 7` | Number of CEM iterations | |
116+
| `n_candidates = 700` | Number of candidate action sequences per CEM iteration | |
117+
| `n_train_epoch_large = 60` | Training epochs of the ensemble / reward model | |
118+
| `top_candidates = 70` | Number of elite candidates kept after each CEM iteration | |
119+
| `learning_rate = 1e-3` | Learning rate for the ensemble / reward optimizer | |
120+
| `grad_clip_norm = 1000` | Gradient clipping norm for stable training | |
121+
| `strategy = "information"` | Free-energy strategy used by the planner | |
122+
| `use_reward = True` | Whether reward is included in the free energy | |
123+
| `use_exploration = True` | Whether to add exploration noise to the planned action | |
124+
| `expl_scale = 0.1` | Scale of the exploration noise | |
125+
| `reward_scale = 1.0` | Scale of the reward term in the free energy | |
126+
127+
### Arguments (`arguments.py`)
128+
129+
Centralized configuration of all training, network and active-inference hyper-parameters. All scenarios (`ES2` – `ES5`) use the same set of arguments, with the environment file determined by the corresponding `main_active_ESX.py` entry script.
130+
131+
## Usage
132+
133+
```
134+
# Install dependencies (PyTorch 1.10.0 required)
135+
pip install torch==1.10.0 numpy matplotlib
136+
137+
# Run HADRL with 2 edge servers (default scenario)
138+
python main_active_ES2.py
139+
140+
# Run HADRL with 3 / 4 / 5 edge servers
141+
python main_active_ES3.py
142+
python main_active_ES4.py
143+
python main_active_ES5.py
144+
```
16145

17146
## Citation
18147

19-
@ARTICLE{11534197,
20-
author={Li, Haoyuan and Lv, Zhenjie and Wang, Yuhang and He, Ying and Fang, Weiwei and Yu, Fei Richard},
21-
journal={IEEE Internet of Things Journal},
22-
title={Joint Service Caching and Computation Offloading in Mobile Edge Networks: A Hierarchical DRL Approach with Active Inference},
23-
year={2026},
24-
volume={},
25-
number={},
26-
pages={1-1},
27-
keywords={Algorithms;Optimization;Modeling;Resource management;Joints;Internet of Things;Timing;Learning (artificial intelligence);Convergence;Educational institutions;Mobile edge computation;computation offloading;service caching;deep reinforcement learning;active inference},
28-
doi={10.1109/JIOT.2026.3695862}
29-
}
148+
If you find HADRL useful or relevant to your project and research, please kindly cite our paper:
149+
150+
```
151+
@ARTICLE{11534197,
152+
author={Li, Haoyuan and Lv, Zhenjie and Wang, Yuhang and He, Ying and Fang, Weiwei and Yu, Fei Richard},
153+
journal={IEEE Internet of Things Journal},
154+
title={Joint Service Caching and Computation Offloading in Mobile Edge Networks: A Hierarchical DRL Approach with Active Inference},
155+
year={2026},
156+
volume={},
157+
number={},
158+
pages={1-1},
159+
keywords={Algorithms;Optimization;Modeling;Resource management;Joints;Internet of Things;Timing;Learning (artificial intelligence);Convergence;Educational institutions;Mobile edge computation;computation offloading;service caching;deep reinforcement learning;active inference},
160+
doi={10.1109/JIOT.2026.3695862}
161+
}
162+
```
30163

31164
## Acknowledgement
165+
32166
To implement this repo, we refer to the code from [Active Inference](https://github.com/r-gould/active-inference). Thanks to [Rhys Gould](https://r-gould.github.io/2023/08/15/free-energy.html) for his great work.
33-
167+
34168
## Contact
35169

36-
Haoyuan Li (24110127@bjtu.edu.cn)
170+
Haoyuan Li ([24110127@bjtu.edu.cn](mailto:24110127@bjtu.edu.cn))
37171

38172
> Please note that the open source code in this repository was mainly completed by the graduate student author during his master's degree study. Since the author did not continue to engage in scientific research work after graduation, it is difficult to continue to maintain and update these codes. We sincerely apologize that these codes are for reference only.

0 commit comments

Comments
 (0)