POMDPs.jl

A Julia interface for defining, solving and simulating partially observable Markov decision processes and their fully observable counterparts.

Package and Ecosystem Features

• General interface that can handle problems with discrete and continuous state/action/observation spaces
• A number of popular state-of-the-art solvers implemented for use out-of-the-box
• Tools that make it easy to define problems and simulate solutions
• Simple integration of custom solvers into the existing interface

Available Packages

The POMDPs.jl package contains only the interface used for expressing and solving Markov decision processes (MDPs) and partially observable Markov decision processes (POMDPs). The POMDPTools package acts as a "standard library" for the POMDPs.jl interface, providing implementations of commonly-used components such as policies, belief updaters, distributions, and simulators. The list of solver and support packages maintained by the JuliaPOMDP community is available at the POMDPs.jl Readme.

Documentation Outline

Documentation comes in three forms:

1. An explanatory guide is available in the sections outlined below.
2. How-to examples are available throughout this documentation with specicic examples in Examples and Gallery of POMDPs.jl Problems.
3. Reference docstrings for the entire POMDPs.jl interface are available in the API Documentation section.

Basics

• Installation
• Getting Started
• Concepts and Architecture
• • POMDPs and MDPs
  • Beliefs and Updaters
  • Solvers and Policies
  • Simulators

Defining POMDP Models

• Defining POMDPs and MDPs
• • A Basic Example: The Tiger POMDP
  • Guide to Defining POMDPs
  • • A more complex example: A partially-observable mountain car
    • State, action, and observation spaces
    • Transition and observation distributions
    • Reward functions
    • Other Components
  • Other ways to define a (PO)MDP
  • • Object-oriented
    • Using a single generative function instead of separate $T$, $Z$, and $R$
    • Tabular
• Spaces and Distributions
• • Spaces
  • Distributions

Writing Solvers and Updaters

• Solvers
• • Online and Offline Solvers
  • Examples
• Example: Defining an offline solver
• Example: Defining an online solver
• Defining a Belief Updater
• • Defining a Belief Type
  • Defining an Updater

Analyzing Results

• Simulation Standard
• • POMDP Simulation
  • MDP Simulation
• Running Simulations
• Interacting with Policies

Examples and Gallery

• Examples
• • Outline
• Defining a POMDP
• • CryingBaby Problem Definition
  • QuickPOMDP Interface
  • Explicit Interface
  • Generative Interface
  • Probability Tables
• Using Different Solvers
• • Checking Requirements
  • Offline (SARSOP)
  • Online (POMCP)
  • Heuristic Policy
• Simulations Examples
• • Stepthrough
  • Rollout Simulations
  • Recording Histories
  • Parallel Simulations
• GridWorld MDP Tutorial
• • Dependencies
  • Problem Overview
  • Defining the Grid World MDP Type
  • Grid World State Space
  • Grid World Action Space
  • Grid World Transition Function
  • Grid World Reward Function
  • Grid World Remaining Functions
  • Solving the Grid World MDP (Value Iteration)
  • Solving the Grid World MDP (MCTS)
  • Visualizing the Value Iteration Policy
  • Seeing a Policy In Action
• Gallery of POMDPs.jl Problems
• • EscapeRoomba
  • DroneSurveillance
  • QuickMountainCar
  • RockSample
  • TagPOMDPProblem
  • Adding New Gallery Examples

POMDPTools - the standard library for POMDPs.jl

• POMDPTools: the standard library for POMDPs.jl
• Implemented Distributions
• • Sparse Categorical (SparseCat)
  • Implicit
  • Bool Distribution
  • Deterministic
  • Uniform
  • Pretty Printing
• Model Tools
• • Interface Extensions
  • Model Transformations
  • Utility Types
• Visualization
• Implemented Belief Updaters
• • Discrete (Bayesian Filter)
  • K Previous Observations
  • Previous Observation
  • Nothing Updater
• Implemented Policies
• • Function
  • Alpha Vector Policy
  • Random Policy
  • Stochastic Policies
  • Vector Policies
  • Value Dict Policy
  • Exploration Policies
  • Playback Policy
  • Utility Wrapper
  • Pretty Printing Policies
• Policy Evaluation
• Implemented Simulators
• • Which Simulator Should I Use?
  • Stepping through
  • Rollouts
  • History Recorder
  • sim()
  • Histories
  • Parallel
  • Display
• CommonRLInterface Integration
• • Environment Wrapper Types
  • Constructors
• Testing
• • Testing (PO)MDP Models
  • Testing Solvers

Reference

• Frequently Asked Questions (FAQ)
• • What is the difference between transition, gen, and @gen?
  • How do I save my policies?
  • Why is my solver producing a suboptimal policy?
  • What if I don't use the rng argument?
  • Why are all the solvers in separate modules?
  • How can I implement terminal actions?
  • Why are there two versions of reward?
  • How do I implement reward(m, s, a) if the reward depends on the next state?
  • Why do I need to put type assertions pomdp::POMDP into the function signature?
• API Documentation
• • Contents
  • Index
  • Types
  • Model Functions
  • Belief Functions
  • Policy and Solver Functions
  • Simulator