To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Hierarchical Planning with Foundation Models (HiP), a framework that leverages different modalities of knowledge to capture information supporting the different levels of decision-making. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in two different long-horizon table-top manipulation tasks.

Paint Block Results

---

Successful execution trajectories of HiP on novel long-horizon tasks in paint-block environment.

Object Arrange Results

---

Successful execution trajectories of HiP on novel long-horizon tasks in object-arrange environment.