The study of the expression of affects and their expression by Embodied Conversational Agents is complex. This is because affects are expressed by non-verbal behaviors on a variety of different modalities, and that these behaviors are influenced by the context of the interaction and other interactants’ behaviors. To overcome these challenges, we present a multi-layer framework and apply it to the study interpersonal stance dynamics. For this purpose, we built a corpus of non-verbal behavior annotations and interpersonal stance traces for job interviews.