SHAO-HUA SUNNoh, HyeonwooHyeonwooNohSomasundaram, SriramSriramSomasundaramLim, Joseph J.Joseph J.Lim2025-12-082025-12-082018https://www.scopus.com/pages/publications/105020572333https://scholars.lib.ntu.edu.tw/handle/123456789/734340Interpreting decision making logic in demonstration videos is key to collaborating with and mimicking humans. To empower machines with this ability, we propose a neural program synthesizer that is able to explicitly synthesize underlying programs from behaviorally diverse and visually complicated demonstration videos. We introduce a summarizer module as part of our model to improve the network’s ability to integrate multiple demonstrations varying in behavior. We also employ a multi-task objective to encourage the model to learn meaningful intermediate representations for end-to-end training. We show that our model is able to reliably synthesize underlying programs as well as capture diverse behaviors exhibited in demonstrations. The code is available at https://shaohua0116.github.io/demo2program.falseconference paper2-s2.0-105020572333