Influence of rhythm features on beat/movement synchronization using a low-cost vision system
Résumé
The process of synchronizing movements to musical rhythms involves a complex interplay between
auditory perception and motor coordination. This study explores the impact of tempo, beat division,
movement type, and musical expertise on sensorimotor synchronization.
Participants, from France, were tasked with synchronizing two types of movements — haptic (finger
tapping) and no haptic (arm swinging) — to an external audio rhythm. The audio stimuli were taken
from the GTZAN rhythm corpus, featuring a variety of tempo levels (slow, medium, fast) and beat
divisions (binary, swing, ternary).
We developed a software that extracts kinematic data using a deep learning-based 2D pose estimator
from a live webcam at 30 fps, that is sufficient for capturing detailed information up to 120 bpm. This
software captures skeletal movements, enabling us to track interphalangeal joints of the hand and the
position of arm joints. It provides clear and interpretable data on the timing and spatial coordination of
motion dynamics. We use the Euclidean distance between fingers to identify motion beats in the tapping
task. We analyze changes in arm joint positions to determine the motion beats in the arm swing task.
We computed synchronization scores based on music and movement features from each audio-video
performance, where the data are represented as discrete time series. In that scope, we considered both
global (e.g., synchronization continuity) and local approaches (e.g., Gaussian function) to calculate the
score from these time series of music and movements. Each metric evaluation had its contribution and
significance, providing a multi-faceted view of synchronization quality.
Our results demonstrate clear trends regarding the influence of factors such as tempo, beat division,
musical skills, and type of movement on synchronization performance. For instance, synchronization
scores decreased as tempo increased. Additionally, the task of finger tapping enabled better synchronization
scores than arm motion, both among participants with and without musical skills. This could be
explained by the more accurate sensory feedback (haptic and auditory) provided by finger tapping, which
is not present in the arm swing motion.