Advances in artificial intelligence and machine learning - in particular neural networks - have given rise to a new generation of virtual assistants and chatbots. Within this work, we describe the motivation and architecture of NADiA - Neurally Animated Dialog Agent - which leverages both the user’s verbal input and facial expressions for multi-modal conversation. NADiA combines a neural language model that generates conversational responses, a convolutional neural network for facial expression analysis, and virtual human technology that is deployed on a mobile phone.