This paper addresses the problem of building a speech recognition system for unmanned aerial vehicles. The task of creating voice interfaces for them is largely unaddressed. We find that recurrent neural networks are a solution to both tasks.
This paper addresses the problem of building a speech recognition system
attuned to the control of unmanned aerial vehicles (UAVs). Even though UAVs are
becoming widespread, the task of creating voice interfaces for them is largely
unaddressed. To this end, we introduce a multi-modal evaluation dataset for UAV
control, consisting of spoken commands and associated images, which represent
the visual context of what the UAV "sees" when the pilot utters the command. We
provide baseline results and address two research directions: (i) how robust
the language models are, given an incomplete list of commands at train time;
(ii) how to incorporate visual information in the language model. We find that
recurrent neural networks (RNNs) are a solution to both tasks: they can be
successfully adapted using a small number of commands and they can be extended
to use visual cues. Our results show that the image-based RNN outperforms its
text-only counterpart even if the command-image training associations are
automatically generated and inherently imperfect. The dataset and our code are
available at http://kite.speed.pub.ro.