Hierarchical Decision Making by Generating and

Following Natural Language Instructions


NeurIPS 2019

Hengyuan Hu* and Denis Yarats* and Qucheng Gong and Yuandong Tian and Mike Lewis

Facebook AI Research and New York University

We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Experiments show that models using natural language as a latent variable significantly outperform models that directly imitate human actions. The compositional structure of language proves crucial to its effectiveness for action representation. We also release our code, models and data.

*Equal Contribution

If you use our paper or code in your research, please consider citing the paper as follows:

@article{DBLP:journals/corr/abs-1906-00744,
  author    = {Hengyuan Hu and
               Denis Yarats and
               Qucheng Gong and
               Yuandong Tian and
               Mike Lewis},
  title     = {Hierarchical Decision Making by Generating and Following Natural Language
               Instructions},
  journal   = {CoRR},
  volume    = {abs/1906.00744},
  year      = {2019},
  url       = {http://arxiv.org/abs/1906.00744},
  archivePrefix = {arXiv},
  eprint    = {1906.00744},
  timestamp = {Thu, 13 Jun 2019 13:36:00 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1906-00744},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

Sample replays

p3.m4v

Sample Replay 1


p2.m4v

Sample Replay 2


p1.m4v

Sample Replay 3


p0.m4v

Sample Replay 4