Influence of rhythm features on beat/movement synchronization using a low-cost vision system - IMT Mines Alès
Communication Dans Un Congrès Année : 2024

Influence of rhythm features on beat/movement synchronization using a low-cost vision system

Hamza Bayd
  • Fonction : Auteur
  • PersonId : 1136238
  • IdHAL : hamza-bayd
Patrice Guyot
Benoit G. Bardy
Pierre Slangen

Résumé

The process of synchronizing movements to musical rhythms involves a complex interplay between auditory perception and motor coordination. This study explores the impact of tempo, beat division, movement type, and musical expertise on sensorimotor synchronization. Participants, from France, were tasked with synchronizing two types of movements — haptic (finger tapping) and no haptic (arm swinging) — to an external audio rhythm. The audio stimuli were taken from the GTZAN rhythm corpus, featuring a variety of tempo levels (slow, medium, fast) and beat divisions (binary, swing, ternary). We developed a software that extracts kinematic data using a deep learning-based 2D pose estimator from a live webcam at 30 fps, that is sufficient for capturing detailed information up to 120 bpm. This software captures skeletal movements, enabling us to track interphalangeal joints of the hand and the position of arm joints. It provides clear and interpretable data on the timing and spatial coordination of motion dynamics. We use the Euclidean distance between fingers to identify motion beats in the tapping task. We analyze changes in arm joint positions to determine the motion beats in the arm swing task. We computed synchronization scores based on music and movement features from each audio-video performance, where the data are represented as discrete time series. In that scope, we considered both global (e.g., synchronization continuity) and local approaches (e.g., Gaussian function) to calculate the score from these time series of music and movements. Each metric evaluation had its contribution and significance, providing a multi-faceted view of synchronization quality. Our results demonstrate clear trends regarding the influence of factors such as tempo, beat division, musical skills, and type of movement on synchronization performance. For instance, synchronization scores decreased as tempo increased. Additionally, the task of finger tapping enabled better synchronization scores than arm motion, both among participants with and without musical skills. This could be explained by the more accurate sensory feedback (haptic and auditory) provided by finger tapping, which is not present in the arm swing motion.
Fichier non déposé

Dates et versions

hal-04728239 , version 1 (09-10-2024)

Identifiants

  • HAL Id : hal-04728239 , version 1

Citer

Hamza Bayd, Patrice Guyot, Benoit G. Bardy, Pierre Slangen. Influence of rhythm features on beat/movement synchronization using a low-cost vision system. SOMOS 2024 - Third Symposium of the ICTMD Study Group on Sound, Movement, and the Sciences, Sep 2024, Oslo, Norway. ⟨hal-04728239⟩
13 Consultations
0 Téléchargements

Partager

More