Noise-driven enhancement: deep reinforcement learning for autonomous UAV navigation

Autonomous navigation is a technology that enables Unmanned Aerial Vehicles (UAVs) to perceive its surroundings using onboard sensors and navigate safely and efficiently from a starting point to a destination in complex, obstacle-rich environments. It is a critical component of UAV intelligence. Therefore, achieving safe and accurate arrival at designated areas, enhancing obstacle avoidance and path selection capabilities, and optimizing flight maneuverability and performance indicators are of great importance. The Twin Delay Deep Deterministic Policy Gradient (TD3) algorithm often suffers from issues that impede training efficiency and decision-making capabilities. These include insufficient exploration leading to entrapment in local optima, high computational costs from excessive exploration, and the neglect of deterministic experience.

Professor Ai Jianliang’s team from Fudan University proposed a "Noise-Driven Enhancement" strategy that effectively improves the model's convergence speed and achieves higher episode average reward. The core of this strategy consists of a dual model for exploration noise control and dual experience replay buffers. In test environments of varying scales, this innovative method enables superior decision-making in complex, dynamic, and uncertain scenarios, thereby balancing the trade-offs between danger, smoothness, and efficiency in navigation trajectories.

By analyzing the relationship between four frequently encountered UAV situations and the demand of exploration, the paper established a local noise control model. Considering the relationship between exploration intensity and training progress, a global noise control model was also developed. This dual-model architecture not only regulates exploration noise on a global level but also considers the specific situations of the UAV during navigation tasks on a local level. This allows for the design of a differentiated exploration noise control scheme that reduces computational costs from excessive exploration and mitigates the problem of navigation getting stuck in local optima.

Furthermore, to further enhance the distinct impacts of deterministic experiences versus experiences with exploration noise on convergence of model, the method utilizes dual experience replay buffers to store these two types of experiences separately. A sampling ratio for these buffers is designed based on the learning progress. The dual-buffer approach facilitates rapid convergence in the early stages of learning and ensures network stability in the later stages.

The authors established the Noise-Driven Enhancement-PMTD3 (NDE-PMTD3) method and framework, validating its decision-making and generalization performance in diverse scenarios. The results show that the NDE-PMTD3 training process is more stable, achieves a higher episode average reward after convergence, and demonstrates a convergence stability improvement of up to approximately 82%. The path length and flight maneuvering were effectively optimized in both static and dynamic obstacle scenarios. Generalization experiments confirmed that the proposed method successfully balances task execution efficiency with risk level associated with the task.

The authors believe this research contributes to the application of TD3 algorithms in fields such as UAV control. Targeted regulation of exploration noise optimizes training efficiency and allows human experience to help guide the training direction through exploration tuning. In the future, Professor Ai's team will conduct further research on the impact of various types of exploration noise on multi-UAV autonomous navigation, continuing to investigate the relationship between exploration and exploitation in reinforcement learning for autonomous navigation.

Original Source

H. ZHANG, Y. LI, L. CHENG, J. AI. Noise-driven enhancement for exploration: Deep reinforcement learning for UAV autonomous navigation in complex environments[J]. Chinese Journal of Aeronautics, 2025, https://doi.org/10.1016/j.cja.2025.103769.

About Chinese Journal of Aeronautics 

Chinese Journal of Aeronautics (CJA) is an open access, peer-reviewed international journal covering all aspects of aerospace engineering, monthly published by Elsevier. The Journal reports the scientific and technological achievements and frontiers in aeronautic engineering and astronautic engineering, in both theory and practice. CJA is indexed in SCI (IF = 5.7, Q1), EI, IAA, AJ, CSA, Scopus.