Skip to content

Double pole balancing

The problem

Two poles stand side by side on one cart, a long one and a short one, and a controller pushes the cart to keep both upright and the cart on the track. Wieland (1991) introduced it as a harder successor of the cart-pole: the poles respond differently to the same push, and a single force has to catch both. It became the standard benchmark of neuroevolution: Stanley and Miikkulainen (2002) introduced NEAT on it, and Gomez, Schmidhuber and Miikkulainen (2008) compare many methods on it.

The system is Wieland's, with the friction corrected as Florian (2007) derives it for the cart-pole, and the settings of Gomez et al. (2008):

Setting Value
Cart 1 kg, on a track from −2.4 m to 2.4 m
Poles 0.1 kg and 1 m long; 0.01 kg and 0.1 m long
Friction 0.0005 of the cart on the track, 0.000002 N m s at each hinge
Force from −10 N to 10 N, held for each step of 0.02 s
Start the long pole at 4° from vertical, the short one upright, the cart at rest in the middle
Failure either pole beyond 36°, or the cart beyond the track's ends
Success 100,000 steps without failing, over 33 minutes of simulated time

The equations are integrated by fourth-order Runge-Kutta in two steps of 0.01 s per step, with genoxide's portable math::sin_cos, so a run is the same bits on every platform (genoxide::problems::control::DoublePole).

The Python version runs the task, the network and the fitness in Rust (gx.problems.control.DoublePole, gx.nn.Mlp, Balance), and NEAT with gx.Neat, its networks' policies run in Rust too: it prints the same output and writes the same trace.

What makes it hard

The short pole is a tenth of the long one's length and falls about three times as fast (the time a pole takes to fall grows with the square root of its length), and both stand on the same cart: a push that saves one can topple the other. A controller has to hold both near vertical and bring the cart back to the middle, a six-variable unstable system, for 100,000 steps. Random weight guessing, which solves the cart-pole in a few hundred attempts, needs about 474,000 here (Gomez et al. 2008).

Representation

A multilayer perceptron (nn::Mlp) of 6 inputs, 6 hidden tanh units and a tanh output, without biases: the network with which Igel (2003, Neuroevolution for reinforcement learning using evolution strategies, CEC 2003: 2588-2595) got his best results on this task, 42 weights, a Real genome in [−1, 1] each. The inputs are the cart's position and velocity and each pole's angle and angular velocity, scaled to about [−1, 1]; the output, in (−1, 1), is the force in units of 10 N. Without biases the network is an odd function of the state, as the task is symmetric; Igel found that with biases CMA-ES needs about three times the evaluations, and here it's 10 times as many.

The fitness is the number of steps the network balances both poles, up to 100,000, maximized.

Algorithm

CMA-ES (Hansen and Ostermeier, 2001, Evolutionary Computation 9(2): 159-195) with its defaults: a population of 4 + ⌊3 ln 42⌋ = 15 and a step size of 0.3 of each gene's range, and IPOP restarts in case a run converges without a solution. The run stops at 100,000 steps, or after 100,000 evaluations.

Then NEAT (neat::Neat, Stanley and Miikkulainen 2002) solves the same task, evolving the network's structure with its weights, with the paper's settings: 150 feed-forward networks that start from the 6 inputs and a bias connected to the output, and grow hidden nodes and connections. Its output, the paper's steepened sigmoid in (0, 1), is the force as 2 × output − 1. The XOR by NEAT page describes the method.

Output

The first line gives the steps the best network balanced, the evaluations and the generations it took. The second gives how far the cart and the two poles went from the middle and the vertical over the 100,000 steps, and the third the network's weights. The last two lines are NEAT's: the steps its best network balanced, after how many evaluations and generations, and the network's hidden nodes and enabled connections.

The project page plays this run back.

Good results

The goal is the task's success criterion, 100,000 steps. The run of output.txt reaches it after 855 evaluations, in 56 generations; its network keeps the cart within 0.99 m of the middle, the long pole within 15.1° and the short one within 18.6°.

Over seeds 1 to 100, all 100 runs solved the task, after 637 evaluations on average (a median of 645, at most 1,425). The counts are of whole generations of 15: the engine evaluates a generation before it checks the stop. With biases (49 weights), 96 of 100 runs solved it, after 6,242 evaluations on average.

Gomez et al. (2008, table 3) list the average evaluations to solve this task: 895 for CMA-ES (from Igel 2003, with this network), 954 for CoSyNE, 3,600 for NEAT, 3,800 for ESP, 12,600 for SANE, 22,100 for CNE, 307,200 for evolutionary programming and 474,329 for random weight guessing; of the value-function methods only Q-MLP solved it, in 10,582. Igel's runs started the long pole at 1°, Stanley and Miikkulainen's (NEAT) too; Gomez et al. start it at 4°, as here.

NEAT's run of output.txt solves it after 3,685 evaluations, in 25 generations, with 3 hidden nodes and 12 enabled connections. Over seeds 1 to 20, all 20 runs solved it, after 2,921 evaluations on average (a median of 2,893), fewer than the paper's 3,600: the paper started the long pole at 1°, and its networks could be recurrent.

Reference: Wieland, A. P. (1991). Evolving neural network controllers for unstable systems. IJCNN 1991, vol. 2: 667-673. Settings of Gomez, F., Schmidhuber, J. and Miikkulainen, R. (2008). Accelerated neural evolution through cooperatively coevolved synapses. JMLR 9: 937-965.

Known optimum: Balanced for 100,000 steps of 0.02 s (the success criterion of Gomez et al. 2008)

Source: examples/double_pole

Interactive run: tachsin.gr/projects/genoxide/examples/double-pole

cargo run --release --example double_pole

//! Double pole balancing: evolve the weights of a neural network that balances two poles on a
//! cart for 100,000 steps, by CMA-ES.
//!
//! Wieland's (1991) task, with Florian's (2007) corrected equations and Gomez, Schmidhuber and
//! Miikkulainen's (2008) settings: a 1 kg cart on a 4.8 m track, with two poles side by side, of
//! 1 m and 0.1 kg and of 0.1 m and 0.01 kg, the long one starting at 4° from vertical, and a force
//! of up to 10 N every 0.02 s. The network sees the cart's position and velocity and each pole's
//! angle and angular velocity, and outputs the force: 6 inputs, 6 hidden tanh units and a tanh
//! output, without biases, 42 weights (Igel's 2003 network). The fitness is the number of steps
//! before a pole passes 36° or the cart leaves the track, maximized by CMA-ES until a network
//! balances them for 100,000 steps, over 33 minutes of simulated time.
//!
//! With `GENOXIDE_TRACE=<file>`, it also writes a trace of its run for the plot on the example's
//! page, with `trace.rs`.
//!
//! ```text
//! cargo run --release --example double_pole
//! ```

mod trace;

use genoxide::neat::{Neat, Network};
use genoxide::nn::{Activation, Mlp};
use genoxide::prelude::*;
use genoxide::problems::control::{DoublePole, SUCCESS_STEPS};

// 6 inputs, 6 hidden units, 1 output, without biases: Igel's (2003) best network for this task
pub fn network() -> Result<Mlp> {
    let mlp = Mlp::new([6, 6, 1], Activation::Tanh)?;
    Ok(mlp.output_activation(Activation::Tanh).bias(false))
}

// the largest |x|, |θ₁| and |θ₂| (in degrees) over an episode of `steps` steps
fn extent(task: &DoublePole, network: &Mlp, weights: &[f64], steps: u32) -> Result<[f64; 3]> {
    let mut policy = network.with(weights)?;
    let mut task = *task;
    let (mut observation, mut action) = ([0.0; 6], [0.0]);
    let mut largest = [0.0_f64; 3];
    for _ in 0..steps {
        task.observe(&mut observation);
        policy.forward(&observation, &mut action);
        task.step(action[0]);
        let [x, _, theta_1, _, theta_2, _] = task.state();
        let values = [
            x.abs(),
            theta_1.abs().to_degrees(),
            theta_2.abs().to_degrees(),
        ];
        for (largest, value) in largest.iter_mut().zip(values) {
            *largest = largest.max(value);
        }
    }
    Ok(largest)
}

fn main() -> Result<()> {
    let mlp = network()?;
    let task = DoublePole::new();
    // the steps balanced, up to 100,000
    let steps = |weights: &Reals| -> Option<f64> {
        let mut policy = mlp.with(weights).ok()?;
        Some(f64::from(task.run(&mut policy, SUCCESS_STEPS)))
    };
    let cmaes = Cmaes::builder(mlp.representation(-1.0..=1.0)?)
        .restarts(cmaes::Restarts::Ipop)
        .maximize()
        .seed(1)
        .build()?;
    // with GENOXIDE_TRACE=<file>, a trace of the run for the plot on the example's page
    let mut trace = trace::Trace::from_env(&mlp, &task);
    let outcome = Engine::new(cmaes, steps)
        .stop_when(Stop::target(f64::from(SUCCESS_STEPS)).or(Stop::evaluations(100_000)))
        .on_generation(|snapshot| trace.record(snapshot))
        .run()?;

    println!(
        "balanced for {} steps (the goal: {SUCCESS_STEPS}) after {} evaluations in {} generations",
        outcome.best_fitness(),
        outcome.evaluations(),
        outcome.generations()
    );
    let weights = outcome.best_genome();
    let [position, long, short] = extent(&task, &mlp, weights, SUCCESS_STEPS)?;
    println!(
        "over the 100000 steps: the cart within {position:.4} m, the poles within {long:.4}° and {short:.4}°"
    );
    let rounded: Vec<String> = weights.iter().map(|w| format!("{w:.3}")).collect();
    println!("weights: [{}]", rounded.join(", "));
    trace.write(weights);
    neat(&task)?;
    Ok(())
}

// the same task by NEAT, with the paper's settings: networks that grow from the inputs and a bias
// connected to the output, whose output, in (0, 1), is the force as 2 × output − 1
fn neat(task: &DoublePole) -> Result<()> {
    let steps = |network: &Network| -> Option<f64> {
        let mut evaluator = network.feed_forward().ok()?;
        let mut policy = |observation: &[f64], action: &mut [f64]| {
            let mut output = [0.0];
            evaluator.activate(observation, &mut output);
            action[0] = 2.0 * output[0] - 1.0;
        };
        Some(f64::from(task.run(&mut policy, SUCCESS_STEPS)))
    };
    let neat = Neat::builder(6, 1).seed(1).build()?;
    let outcome = Engine::new(neat, steps)
        .stop_when(Stop::target(f64::from(SUCCESS_STEPS)).or(Stop::evaluations(100_000)))
        .run()?;
    let network = outcome.best_genome();
    println!(
        "\nNEAT: balanced for {} steps after {} evaluations in {} generations",
        outcome.best_fitness(),
        outcome.evaluations(),
        outcome.generations()
    );
    println!(
        "the network: {} hidden nodes, {} enabled connections",
        network.hidden(),
        network.enabled()
    );
    Ok(())
}

python examples/double_pole/main.py

"""Double pole balancing: evolve the weights of a neural network that balances two poles on a cart
for 100,000 steps, by CMA-ES.

Wieland's (1991) task, with Florian's (2007) corrected equations and Gomez, Schmidhuber and
Miikkulainen's (2008) settings: a 1 kg cart on a 4.8 m track, with two poles side by side, of 1 m
and 0.1 kg and of 0.1 m and 0.01 kg, the long one starting at 4° from vertical, and a force of up
to 10 N every 0.02 s. The network sees the cart's position and velocity and each pole's angle and
angular velocity, and outputs the force: 6 inputs, 6 hidden tanh units and a tanh output, without
biases, 42 weights (Igel's 2003 network). The fitness is the number of steps before a pole passes
36° or the cart leaves the track, maximized by CMA-ES until a network balances them for 100,000
steps, over 33 minutes of simulated time. Then the same task by NEAT.

The task, the networks and the fitness run in Rust (``gx.problems.control``, ``gx.nn``,
``gx.neat``), so the runs are the Rust example's, to the bit, on every platform.

With ``GENOXIDE_TRACE=<file>``, it also writes a trace of its run for the plot on the example's
page, with trace.py.

    python examples/double_pole/main.py
"""

import numpy as np

import genoxide as gx
from genoxide.problems.control import SUCCESS_STEPS, Balance, DoublePole

from trace import Trace, degrees

# 6 inputs, 6 hidden units, 1 output, without biases: Igel's (2003) best network for this task
mlp = gx.nn.Mlp([6, 6, 1], "tanh", output_activation="tanh", bias=False)
task = DoublePole()


def extent(weights, steps):
    """The largest |x|, |θ₁| and |θ₂| (in degrees) over an episode of ``steps`` steps."""
    states = np.abs(task.episode(mlp.policy(weights), steps))
    return (
        float(np.max(states[:, 0])),
        float(np.max(degrees(states[:, 2]))),
        float(np.max(degrees(states[:, 4]))),
    )


# the steps balanced, up to 100,000, evaluated in Rust
cmaes = gx.Cmaes(mlp.representation((-1.0, 1.0)), restarts="ipop", objective="maximize", seed=1)
# with GENOXIDE_TRACE=<file>, a trace of the run for the plot on the example's page
trace = Trace(mlp, task)
result = cmaes.run(
    Balance(task, mlp), target=SUCCESS_STEPS, evaluations=100_000, on_generation=trace.record
)

print(
    f"balanced for {result.best_fitness:.0f} steps (the goal: {SUCCESS_STEPS}) after "
    f"{result.evaluations} evaluations in {result.generations} generations"
)
weights = result.best_genome
position, long, short = extent(weights, SUCCESS_STEPS)
print(
    f"over the 100000 steps: the cart within {position:.4f} m, the poles within {long:.4f}° and "
    f"{short:.4f}°"
)
print("weights: [" + ", ".join(f"{w:.3f}" for w in weights) + "]")
trace.write(weights)


# the same task by NEAT, with the paper's settings: networks that grow from the inputs and a bias
# connected to the output, whose output, in (0, 1), is the force as 2 × output − 1
def steps(network):
    policy = network.feed_forward().policy(scale=2.0, offset=-1.0)
    return task.run(policy, SUCCESS_STEPS)


neat = gx.Neat(6, 1, seed=1)
result = neat.run(steps, target=SUCCESS_STEPS, evaluations=100_000)
network = result.best_genome
print(
    f"\nNEAT: balanced for {result.best_fitness:.0f} steps after {result.evaluations} evaluations "
    f"in {result.generations} generations"
)
print(f"the network: {network.hidden()} hidden nodes, {network.enabled()} enabled connections")

What it prints, from a seeded run:

balanced for 100000 steps (the goal: 100000) after 855 evaluations in 56 generations
over the 100000 steps: the cart within 0.9891 m, the poles within 15.1266° and 18.6145°
weights: [0.343, 0.266, 0.640, 0.274, -0.605, -0.302, -0.166, 0.159, -0.865, 0.546, 0.590, -0.280, -0.020, 0.612, -0.647, 0.009, 0.408, 0.166, 0.641, -0.922, 0.172, -0.083, -0.323, -0.588, 0.143, -0.154, 0.703, 0.899, -0.639, -0.677, 0.286, 0.293, 0.307, 0.769, -0.591, 0.540, -0.764, -0.035, 0.151, 0.132, -0.897, -0.759]

NEAT: balanced for 100000 steps after 3685 evaluations in 25 generations
the network: 3 hidden nodes, 12 enabled connections