Guided Attention for Interpretable Motion Captioning
Abstract
While much effort has been invested in generating human motion from text, relatively few studies
have been dedicated to the reverse direction, that is, generating text from motion. Much of the
research focuses on maximizing generation quality without any regard for the interpretability of the
architectures, particularly regarding the influence of particular body parts in the generation and the
temporal synchronization of words with specific movements and actions. This study explores the
combination of movement encoders with spatio-temporal attention models and proposes strategies
to guide the attention during training to highlight perceptually pertinent areas of the skeleton in
time. We show that adding guided attention with adaptive gate leads to interpretable captioning
while improving performance compared to higher parameter-count non-interpretable SOTA systems.
On the KIT MLD dataset, we obtain a BLEU@4 of 24.4% (SOTA+6%), a ROUGE-L of 58.30%
(SOTA +14.1%), a CIDEr of 112.10 (SOTA +32.6) and a Bertscore of 41.20% (SOTA +18.20%).
On HumanML3D, we obtain a BLEU@4 of 25.00 (SOTA +2.7%), a ROUGE-L score of 55.4%
(SOTA +6.1%), a CIDEr of 61.6 (SOTA -10.9%), a Bertscore of 40.3% (SOTA +2.5%). Our code implementation and reproduction details will be soon available at https://github.com/rd20karim/M2T-Interpretable/tree/main.
Domains
Computer Science [cs]Origin | Files produced by the author(s) |
---|