What I was trying to model
A PM01 competition robot must make decisions that differ in both frequency and meaning. Joint-level control operates continuously. Attack and defense choices are made over short tactical windows. A round-level policy reacts to an opponent and the match state. Between sparse events, a BO3 scheduler decides whether to continue, pause, reset, or request urgent repair.
I used a hierarchical design because treating all of these choices as one flat action space would make exploration expensive and difficult to interpret. Lower layers ask whether a motion is physically executable and safe; middle layers choose skills or tactics; the upper layer decides how to use limited resources across a series.
The original framework follows information upward from physical state and safe execution to skill selection, single-round tactics, and BO3 resource decisions. Strategic commands travel downward through explicit interfaces rather than directly controlling joint torque.
Separating physics from competition rules
The bottom of the stack contains two modules with different responsibilities. A parameterized digital twin represents robot dynamics, contact, balance, actuator behavior, friction, delay, and disturbance. A rule engine converts physical events into match semantics such as valid scores, falls, boundary events, resets, and maintenance opportunities.
This separation matters because a competition rule change should not require relearning the robot’s physical model, and a physics calibration update should not change the meaning of a score. The report treats uncertain quantities as parameters and uses domain randomization over friction, sensor noise, control latency, mild disturbances, and opponent posture. That makes robustness a property to be measured rather than an assumption that the simulator is exact.
Building an action-prior library
Instead of asking reinforcement learning to discover every motion from raw torque, the framework starts from an interpretable library of attack, defense, recovery, and meta-skills. A target-conditioned tracker converts a selected skill into a motion target, while a safety executor applies torque limits, balance constraints, edge-risk checks, and controlled-fall logic.
This library serves three purposes. It compresses the action space, provides human-readable tactical units, and blocks obviously unsafe exploration before it reaches the actuator layer. The hierarchy can still learn when and how to use a skill, but it does not need to rediscover the basic vocabulary of competition behavior.
Offline attack and defense evaluation
Attack selection is first treated as an offline robustness problem. Candidate actions are evaluated across sampled scenarios using Monte Carlo simulation. The score combines expected competitive value with risk terms such as instability, edge exposure, energy use, and tail loss. A CVaR-style correction penalizes actions whose average outcome looks attractive but whose worst cases are unacceptable.
Defense is not evaluated in isolation. The project constructs an attack-defense-scenario payoff tensor and records whether a response preserves standing, limits damage, controls edge risk, and creates a useful counterattack transition. This produces a defense map that can be consumed by the tactical policy rather than a single universal defensive ranking.
The outputs of these two offline stages form a filtered skill space for reinforcement learning. Their role is not to dictate every later action. They provide calibrated priors so that self-play spends more time exploring plausible tactical branches.
Learning a single-round tactical policy
The single-round layer is formulated as a partially observable competitive game. Its observation includes the robot’s recent state, an estimate of opponent behavior, match context, and the available skill set. A hierarchical self-play process trains against a league containing rule-based opponents, historical policies, held-out policies, and unseen behavior profiles.
The framework uses PPO at the tactical layer and distinguishes tactical modes such as probing at range, cutting off the edge, combining at middle distance, counterattacking, and recovering from instability. After training, behavior can be distilled into a tactical automaton. The distilled representation is less expressive than the neural policy, but it makes mode switches and safety fallbacks easier to inspect.
Cross-play is more informative here than performance against one fixed opponent. Evaluation includes win-related outcomes, score margin, falls, pressure success, and the reasonableness of tactical switching. The goal is to test whether the policy transfers across opponent styles rather than memorizing a single matchup.
Scheduling resources across a BO3 series
Pause, reset, repair, and battery decisions occur only when relevant events happen. The resource layer is therefore modeled as a semi-Markov decision process instead of a high-frequency controller. Its state summarizes the series score, robot condition, remaining resources, recent tactical behavior, and the estimated value of intervention.
An event-driven Dueling Double DQN estimates the value of resource actions. The resulting policy can be summarized with thresholds: intervene when the expected improvement in series win probability justifies the cost and lost flexibility. Keeping this layer separate prevents sparse series-level rewards from interfering with dense single-round control learning.
What I learned from the simulation results
The original result group combines learning curves, baseline comparisons, ablations, and sensitivity behavior. It summarizes how the layered design was evaluated inside the parameterized simulation setting.
The report evaluates each layer with metrics appropriate to its decision. Attack ranking is judged by risk-adjusted score and ranking stability; defense by closed-loop success, standing retention, and counterattack value; tactics by cross-play score margin and fall frequency; BO3 scheduling by resource efficiency and a final-win-probability proxy.
Within the reported simulation, the full framework improved the attack robustness score by 0.272 over its comparison, defense closed-loop success by 0.055, and the BO3 win-probability proxy by 0.124. The tactical comparison increased average score margin by 30.558 and reduced average falls from 84.458 to 55.250. I treat these numbers as results from the configured digital twin rather than evidence of physical-robot performance.
The ablation logic is more important than any single number. Removing the candidate skill pool, attack-defense tensor, league self-play, or event-driven resource layer tests whether the hierarchy contributes beyond the base policy. Sensitivity checks vary tail-risk penalties, counterattack weighting, fall penalties, safety thresholds, and the resource layer’s emphasis on win probability.
Limits and next steps
The framework’s strength and weakness both come from its interfaces. A well-calibrated hierarchy reduces search complexity and makes decisions interpretable. An inaccurate upstream model can also pass a misleading prior to every layer above it.
The main limitations are therefore practical:
- digital-twin accuracy directly affects action rankings and policy transfer;
- payoff tensors and event transitions require broad simulation coverage;
- parameterized scoring rules trade specialization for robustness when official details are incomplete;
- major rule or hardware changes require recalibration;
- stable PPO or DQN training in this experiment is not a universal convergence guarantee.
The project helped me understand why decomposition can be as important as the choice of reinforcement-learning algorithm. If I continued it, the next priority would be testing how sensitive the learned policy is to simulation error and whether the interfaces remain useful on a physical platform. I present it as a completed modeling project, not a deployed robot-control system.