Solar sail transfers under uncertainties: A deep reinforcement learning approach

Relevant technological advancements in recent years have established solar sailing as an attractive propulsion system for terrestrial and interplanetary space missions, particularly missions that require low-thrust propulsion for long periods of time. The initial stages in designing solar sail missions involve a thorough analysis of possible transfer trajectories. Traditional approaches primarily focus on deterministic scenarios where the spacecraft dynamics is precisely defined and fully characterized. Whereas, space trajectories are inevitably subject to various sources of uncertainty, including modeling errors in the dynamics, inaccurate measurements, and unpredictable environmental effects, such as fluctuations in solar irradiance. These uncertainties necessitate the development of robust control strategies that can effectively handle stochastic effects. In recent years, reinforcement learning (RL) offers a promising framework for addressing optimization problems characterized by significant uncertainty, which can be challenging to handle using traditional methods. Meanwhile, deep reinforcement learning (DRL) utilizes neural networks (NNs) to approximate complex policies, i.e. control laws. In a research article recently published in Space: Science & Technology, researchers from University of Pisa, Italy, developed a deep reinforcement learning approach to analyze the optimal 3-dimensional interplanetary transfers of a solar sail, accounting for various sources of uncertainty. This RL-based approach offers a promising framework for designing robust and adaptive control strategies for solar sail missions.

First, the dynamical model and the problem description are provided. The 3-dimensional motion of the solar sail is described by a set of modified equinoctial orbital elements (MEOEs) {p, f, g, h, k, L}. The spacecraft equations of motion are written as

(2)

where x = [p, f, g, h, k, L]^T is the sail state vector, a is the propulsive acceleration due to the solar radiation pressure, �� ∊ ℝ^6×3, and b ∊ ℝ^6×1. The propulsive acceleration is written as

(5)

where  is the radial unit vector (from the Sun to the sail),  is the unit vector normal to the sail plane,  ≥ 0, and force coefficients {b₁, b₂, b₃} depend on the sail optical properties (i.e. ρ, s, ��_F, ��_B, B_F B_B). With reference to Fig. 1, the propulsive acceleration is decomposed into a_R, a_T, a_N. The first source of uncertainty considered is the inaccurate knowledge of the optical coefficients of the sail, which can cause variations in the characteristic acceleration and influence the direction of thrust. The optical coefficients are assumed here to be Gaussian-distributed, and the mean values and standard deviations of each parameter are summarized in Table 2. The second source of is imperfections such as wrinkles and billowing, which can lead to deviations from the ideal, perfectly flat surface, causing thrust to deviate from the nominal case. The unit vector  in Eq. (5) is replaced by an equivalent unit vector , the orientation of which is known in terms of a pair of angles {��, ��} as shown in Fig. 2. For this study, a simplified version of probability distribution functions (PDFs) for these two angles, derived from their experimental data, is employed.

Then, RL-based transfer strategy is elaborated. The interplanetary transfer under consideration is formulated as a Markov decision process. Specifically, the total time of flight is divided into N intervals, corresponding to N trajectory arcs. At the beginning of each arc, the agent selects an action a_κ = {α_κ, δ_κ} according to a policy, π_Θ, which is a function of the current observation vector, o_κ = [κ, x_κ^T]^T, a_κ = π_Θ(o_κ). The sail dynamics is then propagated by integrating Eq. (2), maintaining a constant sail orientation for each trajectory arc. At the end of each arc, the spacecraft transitions to a new state, x_��+1, and receives a reward, R_κ. Since the total time of flight is not initially known, it is estimated by solving a minimum-time problem using an indirect method. The primary objective of the training process is to derive the optimal control policy, π_Θ^*, that maximizes the expected discounted return, J, accumulated by the agent over an episode. This paper considers 3 termination conditions: (C1) �� = N, i.e., the flight time reaches the maximum value t_f^max, (C2) the final position and velocity errors fall below a specified threshold, and (C3) the spacecraft exits the allowed region of operation. An episode terminates as soon as any of these conditions are met. A PPO algorithm is employed to train the agent. PPO is a policy-gradient method that directly learns the policy, π_Θ, which maximizes the defined expected cumulative reward J. This policy is approximated by a deep NN, where Θ represents the set of network parameters (weights and biases). The input layer of the policy network consists of 7 neurons, corresponding to the size of the observation vector. The output layer provides the 2 control angles, {α_κ, δ_κ}. The NN architecture employs 3 hidden layers with the following sizes: h₁ = 10h_in, h₂ = , and h₃ = 10h_out, where h_in = 7 is the input layer size and h_out = 2 is the output layer size. Hyperbolic tangent (tanh) activation functions are used for all hidden layers, while a linear activation function is applied to the output layer (see Fig. 3).

Finally, numerical simulations are presented. The above RL algorithm was applied to a 3-dimensional transfer from Earth to Venus. A minimum-time reference trajectory was obtained using an indirect optimization method, providing the optimal initial true anomaly along Earth’s orbit. This initial true anomaly, which is constant, is employed in all simulations, while the final true anomaly along Venus’ orbit was left as a free variable. RL training for all scenarios was conducted on 16 parallel workers utilizing a server equipped with an Intel Xeon Gold 6140M processor @2.30 GHz. In the scenario where the uncertainty in the optical coefficients is considered, each episode was simulated with unique sail optical properties. The agent was trained for approximately 169 million steps. To assess the robustness of the trained policy, a Monte Carlo simulation with 10,000 randomly chosen environmental realizations was conducted. Fig. 4 illustrates the piecewise constant time history of the attitude control angles (α and δ) generated by the trained policy for 5 representative trajectories. The continuous control law obtained from a classical indirect optimization method in a deterministic scenario is also plotted in red. While the clock angle behavior closely matches the optimal solution, the cone angle exhibits significant discrepancies, resulting in a longer flight time. Fig. 5 presents a sample of 20 trajectories from the Monte Carlo simulations, visually demonstrating that all trajectories converge to the target orbit with a longer flight time compared to the optimal deterministic solution. The trained policy successfully guided the sail to the target with the required accuracy in 99.22% of the transfers in the Monte Carlo simulation. In the scenario where considers presence of wrinkles, the orientation of the equivalent unit vector m̂ relative to the sail is randomly determined at the beginning of each episode, both during training and subsequent Monte Carlo simulations. The agent was trained for approximately 169 million steps, and a Monte Carlo simulation with 10,000 randomly chosen environmental realizations was conducted. The Monte Carlo trajectories exhibit behavior similar to the previous scenario. The trained policy successfully guided the sail to the target with the required accuracy in 99.66% of the transfers. In summary, this RL-based approach offers a promising framework for designing robust and adaptive control strategies for solar sail missions. Its versatility allows for the inclusion of various other sources of uncertainty and perturbations within the dynamical model, enabling the development of stochastic control laws that can be implemented onboard the spacecraft for real-time, autonomous navigation.