Model Tools

weighted_iterator(d)

Return an iterator through pairs of the values and probabilities in distribution d.

This is designed to speed up value iteration. Distributions are encouraged to provide a custom optimized implementation if possible.

Example

julia> d = BoolDistribution(0.7)
BoolDistribution(0.7)

julia> collect(weighted_iterator(d))
2-element Array{Pair{Bool,Float64},1}:
  true => 0.7
 false => 0.3

obs_weight(pomdp, s, a, sp, o)

Return a weight proportional to the likelihood of receiving observation o from state sp (and a and s if they are present).

This is a useful shortcut for particle filtering so that the observation distribution does not have to be represented.

ordered_actions(mdp)

Return an AbstractVector of actions ordered according to actionindex(mdp, a).

ordered_actions(mdp) will always return an AbstractVector{A} v containing all of the actions in actions(mdp) in the order such that actionindex(mdp, v[i]) == i. You may wish to override this for your problem for efficiency.

ordered_states(mdp)

Return an AbstractVector of states ordered according to stateindex(mdp, a).

ordered_states(mdp) will always return a AbstractVector{A} v containing all of the states in states(mdp) in the order such that stateindex(mdp, v[i]) == i. You may wish to override this for your problem for efficiency.

ordered_observations(pomdp)

Return an AbstractVector of observations ordered according to obsindex(pomdp, a).

ordered_observations(mdp) will always return a AbstractVector{A} v containing all of the observations in observations(pomdp) in the order such that obsindex(pomdp, v[i]) == i. You may wish to override this for your problem for efficiency.

a, ai = action_info(policy, x)

Return a tuple containing the action determined by policy 'p' at state or belief 'x' and information (usually a NamedTuple, Dict or nothing) from the calculation of that action.

By default, returns nothing as info.

policy, si = solve_info(solver, problem)

Return a tuple containing the policy determined by a solver and information (usually a NamedTuple, Dict or nothing) from the calculation of that policy.

By default, returns nothing as info.

bp, i = update_info(updater, b, a, o)

Return a tuple containing the new belief and information (usually a NamedTuple, Dict or nothing) from the belief update.

By default, returns nothing as info.

transition_matrices(p::SparseTabularProblem)

Accessor function for the transition model of a sparse tabular problem. It returns a list of sparse matrices for each action of the problem.

transition_matrices(m::Union{MDP,POMDP})
transition_matrices(m; sparse=true)

Construct transition matrices for (PO)MDP m.

The returned object is an associative object (usually a Dict), where the keys are actions. Each value in this object is an AbstractMatrix where the row corresponds to the state index of s and the column corresponds to the state index of s'. The entry in the matrix is the probability of transitioning from state s to state s'.

reward_vectors(m::Union{MDP, POMDP})

Construct reward vectors for (PO)MDP m.

The returned object is an associative object (usually a Dict), where the keys are actions. Each value in this object is an AbstractVector where the index corresponds to the state index of s and the entry is the reward for that state.

SparseTabularMDP

An MDP object where states and actions are integers and the transition is represented by a list of sparse matrices. This data structure can be useful to exploit in vectorized algorithm (e.g. see SparseValueIterationSolver). The recommended way to access the transition and reward matrices is through the provided accessor functions: transition_matrix and reward_vector.

Fields

• T::Vector{SparseMatrixCSC{Float64, Int64}} The transition model is represented as a vector of sparse matrices (one for each action). T[a][s, sp] the probability of transition from s to sp taking action a.
• R::Array{Float64, 2} The reward is represented as a matrix where the rows are states and the columns actions: R[s, a] is the reward of taking action a in sate s.
• initial_probs::SparseVector{Float64, Int64} Specifies the initial state distribution
• terminal_states::Set{Int64} Stores the terminal states
• discount::Float64 The discount factor

Constructors

• SparseTabularMDP(mdp::MDP) : One can provide the matrices to the default constructor or one can construct a SparseTabularMDP from any discrete state MDP defined using the explicit interface.

Note that constructing the transition and reward matrices requires to iterate over all the states and can take a while. To learn more information about how to define an MDP with the explicit interface please visit https://juliapomdp.github.io/POMDPs.jl/latest/explicit/ .

• SparseTabularMDP(smdp::SparseTabularMDP; transition, reward, discount) : This constructor returns a new sparse MDP that is a copy of the original smdp except for the field specified by the keyword arguments.

SparseTabularPOMDP

A POMDP object where states and actions are integers and the transition and observation distributions are represented by lists of sparse matrices. This data structure can be useful to exploit in vectorized algorithms to gain performance (e.g. see SparseValueIterationSolver). The recommended way to access the transition, reward, and observation matrices is through the provided accessor functions: transition_matrix, reward_vector, observation_matrix.

Fields

• T::Vector{SparseMatrixCSC{Float64, Int64}} The transition model is represented as a vector of sparse matrices (one for each action). T[a][s, sp] the probability of transition from s to sp taking action a.
• R::Array{Float64, 2} The reward is represented as a matrix where the rows are states and the columns actions: R[s, a] is the reward of taking action a in sate s.
• O::Vector{SparseMatrixCSC{Float64, Int64}} The observation model is represented as a vector of sparse matrices (one for each action). O[a][sp, o] is the probability of observing o from state sp after having taken action a.
• initial_probs::SparseVector{Float64, Int64} Specifies the initial state distribution
• terminal_states::Set{Int64} Stores the terminal states
• discount::Float64 The discount factor

Constructors

• SparseTabularPOMDP(pomdp::POMDP) : One can provide the matrices to the default constructor or one can construct a SparseTabularPOMDP from any discrete state MDP defined using the explicit interface.

Note that constructing the transition and reward matrices requires to iterate over all the states and can take a while. To learn more information about how to define an MDP with the explicit interface please visit https://juliapomdp.github.io/POMDPs.jl/latest/explicit/ .

• SparseTabularPOMDP(spomdp::SparseTabularMDP; transition, reward, observation, discount) : This constructor returns a new sparse POMDP that is a copy of the original smdp except for the field specified by the keyword arguments.

transition_matrix(p::SparseTabularProblem, a)

Accessor function for the transition model of a sparse tabular problem. It returns a sparse matrix containing the transition probabilities when taking action a: T[s, sp] = Pr(sp | s, a).

reward_vector(p::SparseTabularProblem, a)

Accessor function for the reward function of a sparse tabular problem. It returns a vector containing the reward for all the states when taking action a: R(s, a). The length of the return vector is equal to the number of states.

observation_matrix(p::SparseTabularPOMDP, a::Int64)

Accessor function for the observation model of a sparse tabular POMDP. It returns a sparse matrix containing the observation probabilities when having taken action a: O[sp, o] = Pr(o | sp, a).

reward_matrix(p::SparseTabularProblem)

Accessor function for the reward matrix R[s, a] of a sparse tabular problem.

observation_matrices(p::SparseTabularPOMDP)

Accessor function for the observation model of a sparse tabular POMDP. It returns a list of sparse matrices for each action of the problem.

FullyObservablePOMDP(mdp)

Turn MDP mdp into a POMDP where the observations are the states of the MDP.

GenerativeBeliefMDP(pomdp, updater)
GenerativeBeliefMDP(pomdp, updater; terminal_behavior=TerminalStateTerminalBehavior())

Create a generative model of the belief MDP corresponding to POMDP pomdp with belief updates performed by updater. Each step is performed by sampling a state from the current belief, generating an observation from that state and action, and then using updater to update the belief.

A belief is considered terminal when all POMDP states in the support with nonzero probability are terminal.

The default behavior when a terminal POMDP state is sampled from the belief is to transition to terminalstate. This can be controlled by the terminal_behavior keyword argument. Using terminal_behavior=ContinueTerminalBehavior(pomdp, updater) will cause the MDP to keep attempting a belief update even when the sampled state is terminal. This can be further customized by providing terminal_behavior with a Function or callable object that takes arguments b, s, a, rng and returns a new belief (see the implementation of ContinueTerminalBehavior for an example). You can customize behavior additionally using determine_gbmdp_state_type.

UnderlyingMDP(m::POMDP)

Transform POMDP m into an MDP where the states are fully observed.

UnderlyingMDP(m::MDP)

Return m

StateActionReward(m::Union{MDP,POMDP})

Robustly create a reward function that depends only on the state and action.

If reward(m, s, a) is implemented, that will be used, otherwise the mean of reward(m, s, a, sp) for MDPs or reward(m, s, a, sp, o) for POMDPs will be used.

Example

using POMDPs
using POMDPModels
using POMDPTools

m = BabyPOMDP()

rm = StateActionReward(m)

rm(true, true)

# output

-15.0

TerminalState

A type with no fields whose singleton instance terminalstate is used to represent a terminal state with no additional information.

This type has the appropriate promotion logic implemented to function like Missing when added to arrays, etc.

Note that terminal states NEED NOT be of type TerminalState. You can define any state to be terminal by implementing the appropriate isterminal method. Solvers and simulators SHOULD NOT check for this type, but should instead check using isterminal.

terminalstate

The singleton instance of type TerminalState representing a terminal state.