Edit on GitHub

Parallel

POMDPSimulators contains a utility for running many Monte Carlo simulations in parallel to evaluate performance. The basic workflow involves the following steps:

  1. Create a vector of Sim objects, each specifying how a single simulation should be run.
  2. Use the run_parallel or run function to run the simulations.
  3. Analyze the results of the simulations contained in the DataFrame returned by run_parallel.

Example

An example can be found in the POMDPExamples Package.

Sim objects

Each simulation should be specified by a Sim object which contains all the information needed to run a simulation, including the Simulator, POMDP or MDP, Policy, Updater, and any other ingredients.

POMDPSimulators.Sim — Type
Sim(m::MDP, p::Policy[, initialstate]; kwargs...)
Sim(m::POMDP, p::Policy[, updater[, initial_belief[, initialstate]]]; kwargs...)

Create a Sim object that contains everything needed to run and record a single simulation, including model, initial conditions, and metadata.

A vector of Sim objects can be executed with run or run_parallel.

Keyword Arguments

  • rng::AbstractRNG=Random.GLOBAL_RNG
  • max_steps::Int=typemax(Int)
  • simulator::Simulator=HistoryRecorder(rng=rng, max_steps=max_steps)
  • metadata::NamedTuple a named tuple (or dictionary) of metadata for the sim that will be recorded, e.g.(solver_iterations=500,)`.
source

Running simulations

The simulations are actually carried out by the run and run_parallel functions.

POMDPSimulators.run_parallel — Function
run_parallel(queue::Vector{Sim})
run_parallel(f::Function, queue::Vector{Sim})

Run Sim objects in queue in parallel and return results as a DataFrame.

By default, the DataFrame will contain the reward for each simulation and the metadata provided to the sim.

Arguments

  • queue: List of Sim objects to be executed
  • f: Function to process the results of each simulation

This function should take two arguments, (1) the Sim that was executed and (2) the result of the simulation, by default a SimHistory. It should return a named tuple that will appear in the dataframe. See Examples below.

Keyword Arguments

  • show_progress::Bool: whether or not to show a progress meter
  • progress::ProgressMeter.Progress: determines how the progress meter is displayed

Examples

run_parallel(queue) do sim, hist
    return (n_steps=n_steps(hist), reward=discounted_reward(hist))
end

will return a dataframe with with the number of steps and the reward in it.

source

The run function is also provided to run simulations in serial (this is often useful for debugging). Note that the documentation below also contains a section for the builtin julia run function, even though it is not relevant here.

Base.run — Function
run(queue::Vector{Sim})
run(f::Function, queue::Vector{Sim})

Run the Sim objects in queue on a single process and return the results as a dataframe.

See run_parallel for more information.

source

Specifying information to be recorded

By default, only the discounted rewards from each simulation are recorded, but arbitrary information can be recorded.

The run_parallel and run functions accept a function (normally specified via the do syntax) that takes the Sim object and history of the simulation and extracts relevant statistics as a named tuple. For example, if the desired characteristics are the number of steps in the simulation and the reward, run_parallel would be invoked as follows:

df = run_parallel(queue) do sim::Sim, hist::SimHistory
    return (n_steps=n_steps(hist), reward=discounted_reward(hist))
end

These statistics are combined into a DataFrame, with each line representing a single simulation, allowing for statistical analysis. For example,

mean(df[:reward]./df[:n_steps])

would compute the average reward per step with each simulation weighted equally regardless of length.