Adaptive lunar jumping strategies: Understanding Temporal associations for behavior consequences

With the growing demand for in-situ exploration of complex extreme terrains such as polar craters in lunar exploration missions, quadruped robots have demonstrated unique advantages owing to their efficient jumping and locomotion capabilities in weak gravitational fields. However, lunar gravity is only one-sixth of Earth's, which substantially prolongs the flight phase of underactuated jumping. Incorrect takeoff actions on rugged terrain generate rotational torques that cause the robot to continuously rotate in mid-air, with the angular momentum accumulating over time and leading to catastrophic consequences, thus imposing significant uncertainty on landing control. Conventional optimal control methods rely on offline preplanned trajectories and are unable to cope with the variability of unknown rugged terrains; existing deep reinforcement learning models lack effective utilization of temporal causal relationships, relying solely on current states for decision-making and struggling to predict the long-term consequences of actions. Therefore, how to leverage historical sequence information to model temporal dependencies between actions, enabling the robot to possess forward-looking prediction and adaptive adjustment capabilities during the jumping process, has become a key technical challenge for jumping control on complex lunar terrains.

In a recent study published in Space: Science & Technology, the team led by Wang Shuquan from the Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences, proposed an adaptive curriculum reinforcement learning algorithm based on a causal sequence Transformer. The study designs a custom feature extractor that maps sequences of states, actions, and rewards within a continuous time window into a unified feature vector. This vector is then fed into a Transformer encoder–decoder architecture, where temporal dependencies and causal relationships are captured through self-attention mechanisms, ultimately outputting the parameters of the action distribution. Simultaneously, a time-window-based experience replay mechanism is designed, which balances the temporal continuity of sequences with sampling randomness. A progressive curriculum transfer learning framework is constructed, transitioning from flat ground to randomly undulating terrain and then to lunar craters, enabling the agent to gradually adapt to increasingly complex environments based on previously acquired foundational skills. Simulation and ablation experiments on three representative lunar jumping tasks—jumping on randomly undulating terrain, jumping into a crater, and jumping out of a crater—demonstrate that the complete algorithm successfully accomplishes the objectives in all tasks. In the undulating terrain task, it achieves a jump height of 1.71 m and a jump distance of 2.71 m, with the pitch angular velocity maintained near zero during the flight phase. In contrast, the ablation group, lacking temporal modeling capability, fails to predict the takeoff angular momentum, resulting in backward somersaults or collisions with crater walls. This research provides a reinforcement learning solution for adaptive jumping of quadruped robots in complex lunar surface environments that integrates both temporal causal understanding and efficient exploration capability.

First, this paper focuses on the temporal modeling challenge in the jumping locomotion of quadruped robots over complex lunar terrains, and proposes the overall architecture of the causal sequence Transformer-based adaptive curriculum reinforcement learning algorithm (C-STAR). Complex terrains such as lunar polar craters are key targets for scientific exploration; however, conventional rovers are incapable of surmounting the rugged lunar surface obstacles. Quadruped robots, leveraging their jumping capability, can fully exploit the weak gravitational field of the lunar surface to achieve efficient obstacle traversal, offering unique advantages. Nevertheless, lunar gravity is only one-sixth of Earth's, which substantially prolongs the flight phase of underactuated jumping, while other inertial forces remain unchanged. As a result, rotational torques generated by incorrect takeoff actions on rugged terrain continuously accumulate and amplify in mid-air, posing severe challenges to landing control. Existing deep reinforcement learning models make decisions solely based on current observations, neglecting the understanding of long-term temporal causal relationships between actions—similar takeoff states at the current moment may lead to drastically different landing outcomes due to different historical evolution processes. To address this, the study proposes a policy network that integrates the Transformer self-attention mechanism with reinforcement learning. Through a custom feature extractor, the sequences of states, actions, and rewards within a continuous time window are mapped into a unified feature vector, and the encoder–decoder architecture is employed to capture temporal dependencies and latent causal relationships within the high-dimensional sequences, enabling decision-making that associates historical action sequences rather than relying solely on current states. As illustrated in Fig. 1, the overall architecture of the C-STAR algorithm comprises two core modules: a Transformer-based policy network and a progressive curriculum transfer learning framework. Fig. 2 depicts the design of three progressive lunar jumping tasks: from jumping on randomly undulating terrain to jumping into a crater and then jumping out of a crater, with each new task gradually increasing the training difficulty based on knowledge transfer from the preceding task.

Second, this paper elaborates on the design principles of the Transformer-based policy network and the time-window-based experience replay mechanism. As shown in Fig. 3, the feature extractor employs three independent multilayer perceptrons (MLPs) to process the raw inputs of states, actions, and rewards, respectively, mapping them into a unified 128-dimensional feature space. Positional encoding is then added before feeding into the Transformer encoder–decoder architecture. The encoder extracts temporal dependencies and causal relationships within the sequence through multi-layer self-attention mechanisms; the decoder, combined with masking operations, ensures causality, and finally, the output at the last time step is mapped through a fully connected layer to the mean and standard deviation of the action distribution. Conventional random sampling in reinforcement learning disrupts the temporal continuity of sequences, conflicting with the design intent of the Transformer. To address this, the study designs a time-window-based experience replay mechanism: a random set of N starting indices is selected from the replay buffer, and for each index, a consecutive sequence of T time steps is truncated to form a complete sequence, ensuring that within a batch, the starting points of different sequences are random while maintaining temporal continuity within each sequence. The sequence window length is set to 200 steps, matching the single-jump cycle period of the robot under lunar gravity. This design preserves long-term temporal dependencies within each sequence while reducing inter-sequence sample correlation through random starting points, preventing the policy from overfitting to specific trajectories and becoming trapped in local optima.

Finally, this paper validates the effectiveness of the C-STAR algorithm through simulation experiments on three lunar jumping tasks. Table 1 summarizes the success rate comparisons between the complete algorithm and three ablation groups across the three tasks; the complete algorithm successfully accomplishes the objectives in all tasks, whereas each ablation group fails in some or all of the tasks. Fig. 4 presents snapshots comparing the complete algorithm and the ablation group in the random undulating terrain jumping task: C-STAR maintains body stability through a four-legged relative in-phase takeoff strategy, achieving a jump height of 1.71 m and a jump distance of 2.71 m; Ablation 1, lacking Transformer-based temporal modeling, fails to predict and compensate for the accumulated angular momentum during the flight phase, ultimately resulting in a backward somersault; although Ablation 2 can complete the jump, its performance is conservative, with sustained oscillations in pitch angular velocity during the flight phase, indicating that the absence of curriculum learning leaves the policy trapped in a local optimum. Fig. 5 compares the pitch angular velocity data: C-STAR maintains near-zero values during the flight phase and rapidly recovers balance after landing, whereas the ablation groups exhibit continuous accumulation of angular velocity errors during the flight phase, ultimately losing longitudinal stability. In the crater entry and crater exit tasks, the complete algorithm establishes predictive causal chains linking current actions to future states, anticipating the rotational dynamics during the flight phase based on historical observations at the takeoff stage and adjusting the action plan in advance; the ablation groups, lacking temporal modeling capability, fail to predict the takeoff angular momentum, resulting in collisions with crater walls or backward somersaults. This research provides a reinforcement learning solution for adaptive jumping of quadruped robots in complex lunar surface environments that integrates both temporal causal understanding and efficient exploration capability.