Implemented Simulators

stepthrough(problem, policy, [spec])
stepthrough(problem, policy, [spec], [rng=rng], [max_steps=max_steps])
stepthrough(mdp::MDP, policy::Policy, [init_state], [spec]; [kwargs...])
stepthrough(pomdp::POMDP, policy::Policy, [up::Updater, [initial_belief, [initial_state]]], [spec]; [kwargs...])

Create a simulation iterator. This is intended to be used with for loop syntax to output the results of each step as the simulation is being run.

Example:

pomdp = BabyPOMDP()
policy = RandomPolicy(pomdp)

for (s, a, o, r) in stepthrough(pomdp, policy, "s,a,o,r", max_steps=10)
    println("in state $s")
    println("took action $a")
    println("received observation $o and reward $r")
end

The optional spec argument can be a string, tuple of symbols, or single symbol and follows the same pattern as eachstep called on a SimHistory object.

Under the hood, this function creates a StepSimulator with spec and returns a [PO]MDPSimIterator by calling simulate with all of the arguments except spec. All keyword arguments are passed to the StepSimulator constructor.

RolloutSimulator(rng, max_steps)
RolloutSimulator(; <keyword arguments>)

A fast simulator that just returns the reward

The simulation will be terminated when either

1. a terminal state is reached (as determined by isterminal() or
2. the discount factor is as small as eps or
3. max_steps have been executed

Keyword arguments:

• rng::AbstractRNG (default: Random.default_rng()) - A random number generator to use.
• eps::Float64 (default: 0.0) - A small number; if γᵗ where γ is the discount factor and t is the time step becomes smaller than this, the simulation will be terminated.
• max_steps::Int (default: typemax(Int)) - The maximum number of steps to simulate.

Usage (optional arguments in brackets):

ro = RolloutSimulator()
history = simulate(ro, pomdp, policy, [updater [, init_belief [, init_state]]])

See also: HistoryRecorder, run_parallel

A simulator that records the history for later examination

The simulation will be terminated when either

1. a terminal state is reached (as determined by isterminal() or
2. the discount factor is as small as eps or
3. max_steps have been executed

Keyword Arguments: - rng: The random number generator for the simulation - capture_exception::Bool: whether to capture an exception and store it in the history, or let it go uncaught, potentially killing the script - show_progress::Bool: show a progress bar for the simulation - eps - max_steps

Usage (optional arguments in brackets):

hr = HistoryRecorder()
history = simulate(hr, pomdp, policy, [updater [, init_belief [, init_state]]])

sim(polfunc::Function, mdp::MDP; [<keyword arguments>])
sim(polfunc::Function, pomdp::POMDP; [<keyword arguments>])

Alternative way of running a simulation with a function specifying how to calculate the action at each timestep.

Usage

sim(mdp) do s
    # code that calculates action `a` based on `s` - this is the policy
    # you can also do other things like display something
    return a
end

for an MDP or

sim(pomdp) do o
    # code that calculates 'a' based on observation `o`
    # optionally you could save 'o' in a global variable or do a belief update
    return a
end

or with a POMDP

sim(pomdp, updater) do b
    # code that calculates 'a' based on belief `b`
    # `b` is calculated by `updater`
    return a
end

for a POMDP and a belief updater.

Keyword Arguments

All Versions

• initialstate: the initial state for the simulation
• simulator: keyword argument to specify any simulator to run the simulation. If nothing is specified for the simulator, a HistoryRecorder will be used as the simulator, with all keyword arguments forwarded to it, e.g.

  sim(mdp, max_steps=100, show_progress=true) do s
      # ...
  end

  will limit the simulation to 100 steps.

POMDP version

• initialobs: this will control the initial observation given to the policy function. If this is not defined, rand(initialobs(m, s)) will be used if it is available. If it is not, missing will be used.

POMDP and updater version

• initialbelief: initialize_belief(updater, initialbelief) is the first belief that will be given to the policy function.

SimHistory

An (PO)MDP simulation history returned by simulate(::HistoryRecorder, ::Union{MDP,POMDP},...).

This is an AbstractVector of NamedTuples containing the states, actions, etc.

Examples

hist[1][:s] # returns the first state in the history

hist[:a] # returns all of the actions in the history

for t in eachstep(hist, [spec])
    ...
end

Iterate through the steps in SimHistory hist. spec is a tuple of symbols or string that controls what is returned for each step.

For example,

for (s, a, r, sp) in eachstep(h, "(s, a, r, sp)")    
    println("reward $r received when state $sp was reached after action $a was taken in state $s")
end

returns the start state, action, reward and destination state for each step of the simulation.

Alternatively, instead of expanding the steps implicitly, the elements of the step can be accessed as fields (since each step is a NamedTuple):

for step in eachstep(h, "(s, a, r, sp)")    
    println("reward $(step.r) received when state $(step.sp) was reached after action $(step.a) was taken in state $(step.s)")
end

The possible valid elements in the iteration specification are

• Any node in the (PO)MDP Dynamic Decision network (by default :s, :a, :sp, :o, :r)
• b - the initial belief in the step (for POMDPs only)
• bp - the belief after being updated based on o (for POMDPs only)
• action_info - info from the policy decision (from action_info)
• update_info - info from the belief update (from update_info)
• t - the timestep index

Sim(m::MDP, p::Policy[, initialstate]; kwargs...)
Sim(m::POMDP, p::Policy[, updater[, initial_belief[, initialstate]]]; kwargs...)

Create a Sim object that contains everything needed to run and record a single simulation, including model, initial conditions, and metadata.

A vector of Sim objects can be executed with run or run_parallel.

Keyword Arguments

• rng::AbstractRNG=Random.default_rng()
• max_steps::Int=typemax(Int)
• simulator::Simulator=HistoryRecorder(rng=rng, max_steps=max_steps)
• metadata::NamedTuple a named tuple (or dictionary) of metadata for the sim that will be recorded, e.g.(solver_iterations=500,)`.

run_parallel(queue::Vector{Sim})
run_parallel(f::Function, queue::Vector{Sim})

Run Sim objects in queue in parallel and return results as a DataFrame.

By default, the DataFrame will contain the reward for each simulation and the metadata provided to the sim.

Arguments

• queue: List of Sim objects to be executed
• f: Function to process the results of each simulation

This function should take two arguments, (1) the Sim that was executed and (2) the result of the simulation, by default a SimHistory. It should return a named tuple that will appear in the dataframe. See Examples below.

Keyword Arguments

• show_progress::Bool: whether or not to show a progress meter
• progress::ProgressMeter.Progress: determines how the progress meter is displayed

Examples

run_parallel(queue) do sim, hist
    return (n_steps=n_steps(hist), reward=discounted_reward(hist))
end

will return a dataframe with with the number of steps and the reward in it.

run(queue::Vector{Sim})
run(f::Function, queue::Vector{Sim})

Run the Sim objects in queue on a single process and return the results as a dataframe.

See run_parallel for more information.

DisplaySimulator(;kwargs...)

Create a simulator that displays each step of a simulation.

Given a POMDP or MDP model m, this simulator roughly works like

for step in stepthrough(m, ...)
    display(render(m, step))
end

Keyword Arguments

• display::AbstractDisplay: the display to use for the first argument to the display function. If this is nothing, display(...) will be called without an AbstractDisplay argument.
• render_kwargs::NamedTuple: keyword arguments for POMDPTools.render(...)
• max_fps::Number=10: maximum number of frames to be displayed per second - sleep will be used to skip extra time, so this is not designed for high precision
• predisplay::Function: function to call before every call to display(...). The only argument to this function will be the display (if it is specified) or nothing
• extra_initial::Bool=false: if true, display an extra step at the beginning with only elements t, sp, and bp for POMDPs (this can be useful to see the initial state if render displays only sp and not s).
• extra_final::Bool=true: iftrue, display an extra step at the end with only elementst,done,s, andbfor POMDPs (this can be useful to see the final state ifrenderdisplays onlysand notsp`).
• max_steps::Integer: maximum number of steps to run for
• spec::NTuple{Symbol}: specification of what step elements to display (see eachstep)
• rng::AbstractRNG: random number generator

See the POMDPSimulators documentation for more tips about using specific displays.