Improvements in learning to control perched landings

Fletcher, L.; Clarke, R.; Richardson, T.; Hansen, M.

Abstract

Reinforcement learning has previously been applied to the problem of controlling a perched landing manoeuvre for a custom sweep-wing aircraft. Previous work showed that the use of domain randomisation to train with atmospheric disturbances improved the real-world performance of the controllers, leading to increased reward. This paper builds on the previous project, investigating enhancements and modifications to the learning process to further improve performance, and reduce final state error. These changes include modifying the observation by adding information about the airspeed to the standard aircraft state vector, employing further domain randomisation of the simulator, optimising the underlying RL algorithm and network structure, and changing to a continuous action space. Simulated investigations identified hyperparameter optimisation as achieving the most significant increase in reward performance. Several test cases were explored to identify the best combination of enhancements. Flight testing was performed, comparing a baseline model against some of the best performing test cases from simulation. Generally, test cases that performed better than the baseline in simulation also performed better in the real world. However, flight tests also identified limitations with the current numerical model. For some models, the chosen policy performs well in simulation yet stalls prematurely in reality, a problem known as the reality gap.

• Journal Article Type: Article
• Acceptance Date: Apr 11, 2022
• Online Publication Date: May 4, 2022
• Publication Date: Jul 4, 2022
• Deposit Date: May 5, 2022
• Publicly Available Date: May 6, 2022
• Journal: Aeronautical Journal
• Print ISSN: 0001-9240
• Electronic ISSN: 2059-6464
• Publisher: Cambridge University Press (CUP)
• Peer Reviewed: Peer Reviewed
• Volume: 126
• Issue: 1301
• Pages: 1101-1123
• DOI: https://doi.org/10.1017/aer.2022.48
• Keywords: Aerospace Engineering; UAV; Machine learning; Reinforcement learning; Perching; Agile manoeuvres
• Public URL: https://uwe-repository.worktribe.com/output/9457781
• Additional Information: Copyright: © The Author(s), 2022. Published by Cambridge University Press on behalf of Royal Aeronautical Society; License: This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.; Free to read: This content has been made available to all.