The two types of simulators used in building perception systems are those that work on real data and those that don't work on real data. By defining the state vector that separates the best team to build a perception system from the best team to build a planner, they can replay their perception data and test their system with different objects in the scene over time.