Language Grounding to Vision and Control
Fall 2017, CMU 10-808


Instructor: Katerina Fragkiadaki
Lectures: T, 9:00-12pm, 5222 Gates and Hillman Centers (GHC)
Office Hours: Tuesday 3:00-4:00pm, 8015 GHC

Class goals

This is a seminar course that will visit recent progress on the problem of language acquisition through pairing of multiple modalities (vision, haptics, audio etc), as well as active interaction with the world. The central questions/topics we will visit are:
  • How can language help accelerate learning of an autonomous agent (if at all)
  • How humans acquire language and why?
  • Inductive biases for strong generalization
  • Architectures for agent capable of compositional grounding of language
  • State representation of video visual scenes and imaginations from story reading
  • Language for high level planning and control.
  • Neural-symbolic architectures for hierarchical symbolic grounding

Schedule

The following schedule is tentative, it will continuously change based on time constraints and interest of the people in the class. Lecture notes will be added as lectures progress.

Date Topic Readings Presenters (Slides)
8/29 The Grounding Problem, Learning from Data vs. Programming with Language, Explanation-Based Learning, Course Overview [1-6] Katerina (Intro.pdf)
9/5 Grounding language on programs(I): Executable Semantic Parsing [16-19] Katerina, Tejas, Sarah (18.pdf, 16.pdf, 17.pdf)
9/12 Compositionally of Meaning and Recursive networks, Pointer Networks [20-24, 53, 69] Katerina, Ricson (RNNsPNs.pdf, 20.pdf)
9/19 Grounding Language on Visual Concepts (I) [43-47, 70-72] Nikhil, Rishub, Ben, Katerina (43.pdf, 44.pdf, 70.pdf, 71.pdf)
9/26 Grounding Language on Visual Programs (II) [35-37, 74-75] Sumedha, Siliang, Sarah, Arjun (37.pdf, 75.pdf)
10/3 Grounding Language through Multi-Agent Collaboration [48-51] Manasi, Arjun, Ricson, Varun, Siliang, Deepika (49.pdf, 44.pdf)
10/10 Language and Memory State Representations: Architectures that Keep Track of State [26-29, 73] Mohit, Tanmay, Ricson, Varun, Siliang (27.pdf, 28.pdf, 29.pdf)
10/17 Neural-Symbolic Architecture, Rule-based NN [7-8, 76-81] Rishub, Varun, Ricson, Sumedha, Ben, Sarah (7.pdf, 78.pdf, 81.pdf)
10/24 Learning Theorem Proving, Neural-Symbolic Architectures, Rule-based NN [11-14] Ricson, Varun, Rishub, Nikhil (13.pdf)
10/31 Grounding Language on Programs: Program Induction [30-34] Mohit, Ben, Varun, Ricson, Sumedha (31.pdf, 32.pdf)
11/7 Conversational Agents [88-91] Ricson, Varun, Sarah, Rhea, Arjun, Deepika, Siliang (89.pdf, 90.pdf)
11/14 Grounding Language to Robotic Programs (I) [57-59, 67-68] Siliang, Deepika, Rishub, Ricson (57.pdf, 58.pdf, 59.pdf)
11/21 Grounding Language to Robotic Programs (II) [60-65] Ricson, Deepika, Siliang, Rhea, Sarah (60.pdf, 63.pdf, 64.pdf)
11/28 Weakly Supervised Semantic Parsing [82-87] Mohit, Sumedha, Arjun, Rhea, Ben, Deepika (85.pdf, 87.pdf)
12/5 Future of Language Grounding Katerina

Resources

Readings

  1. The Development of Embodied Cognition: Six Lessons from Babies
  2. How language programs the mind
  3. Explanation-Based Generalization: A Unifying View
  4. The symbol grounding problem
  5. Symbol Grounding and Meaning- A Comparison of High-Dimensional and Embodied Theories of Meaning
  6. From Machine Learning to Machine Reasoning
  7. Harnessing Deep Neural Networks with Logic Rules
  8. Deep Neural Networks with Massive Learned Knowledge
  9. The Genesis Story Understanding and Story Telling System: A 21st Century Step toward Artificial Intelligence
  10. Model-based Story Summary
  11. End-to-end Differentiable Proving
  12. Learning Knowledge Base Inference with Neural Theorem Provers
  13. Deep Network Guided Proof Search
  14. DeepMath - Deep Sequence Models for Premise Selection
  15. Multimodal Distributional Semantics
  16. Language to Logical Form with Neural Attention
  17. Learning Executable Semantic Parsers for Natural Language Understanding
  18. Neural Symbolic Machines: Learning Semantic Parsers on Freebase withWeak Supervision
  19. From Language to Programs- Bridging Reinforcement Learning and Maximum Marginal Likelihood
  20. Recursive Neural Networks Can Learn Logical Semantics
  21. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
  22. Deep Recursive Neural Networks for Compositionality in Language
  23. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks
  24. Learning Continuous semantic representations of symbolic expressions
  25. A simple neural network module for relational reasoning
  26. Learning graphical state transitions
  27. Gated Graph Sequence Neural Networks
  28. Tracking the World State with Recurrent Entity Networks
  29. End-To-End Memory Networks
  30. Unsupervised Learning by Program Synthesis
  31. Differentiable Programs with Neural Libraries
  32. Programming with a differentiable Forth interpreter
  33. DeepCoder- Learning to Write Programs
  34. Program Synthesis using Natural Language
  35. Learning to Compose neural networks for Question Answering
  36. Learning to Reason- End-to-End Module Networks for Visual Question Answering
  37. Inferring and Executing Programs for Visual Reasoning
  38. Word learning as Bayesian inference
  39. Visual Dialog
  40. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
  41. Coherent Dialogue with Attention-based Language Models
  42. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
  43. Generating Visual Explanations
  44. Situation Recognition: Visual Semantic Role Labeling for Image Understanding
  45. Imagined Visual Representations as Multimodal Embeddings
  46. Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
  47. Learning Abstract Concept Embeddings from Multi-Modal Data- Since You Probably Can't See What I Mean
  48. Translating neuralese
  49. Emergence of Grounded Compositional Language in Multi-Agent Populations
  50. Multi-Agent Cooperation and the Emergence of (Natural) Language
  51. Guiding Interaction Behaviors for Multi-modal Grounded Language Learning with Backpropagation
  52. Pointer networks
  53. Parsing with Compositional Vector Grammars
  54. The Need for Biases in Learning Generalizations
  55. Theory-based Bayesian models of inductive learning and reasoning
  56. Intuitive Theories
  57. Mapping Instructions and Visual Observations to Actions with Reinforcement Learning
  58. Reinforcement Learning for Mapping Instructions to Actions
  59. Environment-Driven Lexicon Induction for High-Level Instructions
  60. Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
  61. Tell Me Dave- Context-Sensitive Grounding of Natural Language to Manipulation Instructions
  62. Interpreting and Executing Recipes with a Cooking Robot
  63. Learning to Follow Navigational Directions
  64. Learning to Interpret Natural Language Commands through Human-Robot Dialog
  65. Learning to Interpret Natural Language Navigation Instructions from Observations
  66. Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation
  67. A Natural Language Planner Interface for Mobile Manipulators
  68. Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation
  69. Attention Is All You Need
  70. Deep visual-semantic alignments for generating image descriptions
  71. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
  72. A simple neural network module for relational reasoning
  73. Tracking by Natural Language Specification
  74. Segmentation from Natural Language Expressions
  75. Generation and Comprehension of Unambiguous Object Descriptions
  76. Leveraging Knowledge bases in LSTMs for improving machine reading
  77. A Neural Knowledge Language Model
  78. Traversing Knowledge Graphs in Vector Space
  79. EMBEDDING ENTITIES AND RELATIONS FOR LEARNING AND INFERENCE IN KNOWLEDGE BASES
  80. Representing Text for Joint Embedding of Text and Knowledge Bases
  81. Differentiable Learning of Logical Rules for Knowledge Base Reasoning
  82. Symbolic Priors for RNN-based Semantic Parsing
  83. Sequence based structured prediction for Semantic Parsing
  84. Building a Semantic Parser Overnight
  85. Human-in-the-Loop Parsing
  86. The Value of Semantic Parse Labeling for Knowledge Base Question Answering
  87. Weakly Supervised Training of Semantic Parsers
  88. Deal or No-Deal? End-to-End Learning For Negotiation Dialogues
  89. Best of Both Worlds: Transferring Knowledge from Discriminative Learning to a Generative Visual Dialog Model
  90. Evaluating Visual Conversational Agents via Cooperative Human-AI Games
  91. Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning

Grading

The grade is determined by a paper presentation you need to do, your participation in class (asking good questions, making connections between topics etc.) as well as a final project. The final project can be a small innovation on top of methods and algorithms presented in the course, or your own project idea on topics covered in the course. The course grade is a weighted average of your participation in class (30%), your paper presentation (30%), and your final project (40%).

Prerequisites

This course assumes familiarity with Computer Vision, basic NLP concepts, machine learning, deep learning.

Web design: Anton Badev