22.4.2.7 Music Related Gestures, Systems, Music Video Analysis

Morita, H.[Hideyuki], Hashimoto, S.[Shuji], Ohteru, S.[Sadamu],
A Computer Music System the Follows a Human Conductor,
Computer(24), No. 7, July 1991, pp. 44-53. Dataglove and camera. BibRef 9107

Adams, B.[Brett], Dorai, C.[Chitra], Venkatesh, S.[Svetha],
Finding The Beat: An Analysis Of The Rhythmic Elements Of Motion Pictures,
IJIG(2), No. 2, April 2002, pp. 215-245. 0204
BibRef
Earlier:
Towards Automatic Extraction of Expressive Elements from Motion Pictures: Tempo,
ICME00(TA2). 0007
BibRef

Adams, B.,
Novel Approach to Determining Tempo and Dramatic Story Sections in Motion Pictures,
ICIP00(Vol II: 283-286).
IEEE DOI 0008
BibRef

Jean, F.[Frederic], Albu, A.B.[Alexandra Branzan],
The visual keyboard: Real-time feet tracking for the control of musical meta-instruments,
SP:IC(23), No. 7, August 2008, pp. 505-515.
Elsevier DOI 0804
Motion tracking; Real-time event detection BibRef

Gillet, O., Essid, S., Richard, G.,
On the Correlation of Automatic Audio and Visual Segmentations of Music Videos,
CirSysVideo(17), No. 3, March 2007, pp. 347-355.
IEEE DOI 0703
BibRef

Licsar, A.[Attila], Sziranyi, T.[Tamàs], Kovacs, L.[Làszlò], Pataki, B.[Balàzs],
A Folk Song Retrieval System with a Gesture-Based Interface,
MultMedMag(16), No. 3, July-September 2009, pp. 48-59.
IEEE DOI 0909
BibRef

Licsár, A.[Attila], Szirányi, T.[Tamás],
Hand Gesture Recognition in Camera-Projector System*,
CVHCI04(83-93).
Springer DOI 0505
BibRef

Tindale, A., Kapur, A., Tzanetakis, G.,
Training Surrogate Sensors in Musical Gesture Acquisition Systems,
MultMed(13), No. 1, 2011, pp. 50-59.
IEEE DOI 1102
BibRef

Amelynck, D.[Denis], Grachten, M.[Maarten], van Noorden, L.[Leon], Leman, M.[Marc],
Toward E-Motion-Based Music Retrieval a Study of Affective Gesture Recognition,
AffCom(3), No. 2, 2012, pp. 250-259.
IEEE DOI 1208
BibRef

Rebelo, A.[Ana], Fujinaga, I.[Ichiro],
Optical music recognition: state-of-the-art and open issues,
MultInfoRetr(1), No. 3, October 2012, pp. 173-190.
WWW Link. 1211
BibRef

Schramm, R., Jung, C.R.[C. Rosito], Miranda, E.R.[E. Reck],
Dynamic Time Warping for Music Conducting Gestures Evaluation,
MultMed(17), No. 2, February 2015, pp. 243-255.
IEEE DOI 1502
gesture recognition BibRef

Duan, Z., Essid, S., Liem, C.C.S., Richard, G., Sharma, G.,
Audiovisual Analysis of Music Performances: Overview of an Emerging Field,
SPMag(36), No. 1, January 2019, pp. 63-73.
IEEE DOI 1901
Music, Visualization, Task analysis, Instruments, Multiple signal classification, Signal processing, Cameras, Acoustics BibRef

Turchet, L., Barthet, M.,
Co-Design of Musical Haptic Wearables for Electronic Music Performer's Communication,
HMS(49), No. 2, April 2019, pp. 183-193.
IEEE DOI 1903
Haptic interfaces, Portable computers, Performance evaluation, Electronic music, Prototypes, wearables BibRef

Liu, J., Yang, Y., Jeng, S.,
Weakly-Supervised Visual Instrument-Playing Action Detection in Videos,
MultMed(21), No. 4, April 2019, pp. 887-901.
IEEE DOI 1903
Videos, Instruments, Visualization, Training, Task analysis, Support vector machines, Object detection, Action detection, object localization BibRef

Daly, I., Williams, D., Malik, A., Weaver, J., Kirke, A., Hwang, F., Miranda, E., Nasuto, S.J.,
Personalised, Multi-Modal, Affective State Detection for Hybrid Brain-Computer Music Interfacing,
AffCom(11), No. 1, January 2020, pp. 111-124.
IEEE DOI 2003
Brain-Computer Interface. Electroencephalography, Physiology, Music, Calibration, Training, Medical treatment, Psychology, EEG, GSR, affective state detection, personalised affective state detection BibRef

Hsu, Y., Wang, J., Chiang, W., Hung, C.,
Automatic ECG-Based Emotion Recognition in Music Listening,
AffCom(11), No. 1, January 2020, pp. 85-99.
IEEE DOI 2003
Electrocardiography, Music, Emotion recognition, Physiology, Feature extraction, Multiple signal classification, machine learning BibRef

Wang, C.Y.[Chien-Yao], Chang, P.C.[Pao-Chi], Ding, J.J.[Jian-Jiun], Tai, T.C.A.[Tzu-Chi-Ang], Santoso, A.[Andri], Liu, Y.T.[Yu-Ting], Wang, J.C.[Jia-Ching],
Spectral-Temporal Receptive Field-Based Descriptors and Hierarchical Cascade Deep Belief Network for Guitar Playing Technique Classification,
Cyber(52), No. 5, May 2022, pp. 3684-3695.
IEEE DOI 2206
Auditory system, Music, Instruments, Feature extraction, Machine learning, Hidden Markov models, Brain modeling, spectral-temporal receptive fields (STRFs) BibRef

Varni, G.[Giovanna], Mancini, M.[Maurizio], Fadiga, L.[Luciano], Camurri, A.[Antonio], Volpe, G.[Gualtiero],
The Change Matters! Measuring the Effect of Changing the Leader in Joint Music Performances,
AffCom(13), No. 2, April 2022, pp. 700-712.
IEEE DOI 2206
Music, Conductors, Performance evaluation, Kinematics, Psychology, Synchronization, Joint action, group, change, entrainment, musical performance BibRef

Wu, E.[Erwin], Nishioka, H.[Hayato], Furuya, S.[Shinichi], Koike, H.[Hideki],
Marker-removal Networks to Collect Precise 3D Hand Data for RGB-based Estimation and its Application in Piano,
WACV23(2976-2985)
IEEE DOI 2302
Training, Measurement, Annotations, Pose estimation, Manuals, Algorithms: Image recognition and understanding (object detection, Arts/games/social media BibRef

Bayd, H.[Hamza], Guyot, P.[Patrice], Bardy, B.[Benoit], Slangen, P.R.L.[Pierre R. L.],
Automatic Scoring of Synchronization from Fingers Motion Capture and Music Beats,
ISHAPE22(235-245).
Springer DOI 2208
BibRef

Narasimhan, M.[Medhini], Ginosar, S.[Shiry], Owens, A.[Andrew], Efros, A.[Alexei], Darrell, T.J.[Trevor J.],
Strumming to the Beat: Audio-Conditioned Contrastive Video Textures,
WACV22(507-516)
IEEE DOI 2202
Measurement, Limiting, Computational modeling, Semantics, Fitting, Synchronization, Multimedia Applications BibRef

Gan, C.[Chuang], Huang, D.[Deng], Chen, P.H.[Pei-Hao], Tenenbaum, J.B.[Joshua B.], Torralba, A.B.[Antonio B.],
Foley Music: Learning to Generate Music from Videos,
ECCV20(XI:758-775).
Springer DOI 2011
BibRef

Gan, C., Huang, D., Zhao, H., Tenenbaum, J.B., Torralba, A.,
Music Gesture for Visual Sound Separation,
CVPR20(10475-10484)
IEEE DOI 2008
Videos, Visualization, Semantics, Feature extraction, Music, Task analysis, Training BibRef

van Heek, J.[Justin], Woo, G.[Gideon], Park, J.[Jack], Tsang, H.H.[Herbert H.],
A Comparison of Computational Intelligence Techniques for Real-time Discrete Multivariate Time Series Classification of Conducting Gestures,
CVS19(573-585).
Springer DOI 1912
BibRef

Xie, Y., Wang, H., Hao, Y., Xu, Z.,
Visual Rhythm Prediction with Feature-Aligning Network,
MVA19(1-6)
DOI Link 1806
data visualisation, feature extraction, image sequences, learning (artificial intelligence), music, neural nets, Training BibRef

Seo, K., Cho, H., Choi, D., Lee, S., Lee, J., Ko, J.,
TwohandsMusic: Multitask Learning-Based Egocentric Piano-Playing Gesture Recognition System for Two Hands,
ICIP19(4614-4618)
IEEE DOI 1910
piano-playing gesture, hand pose estimation, gesture classification, multitask learning BibRef

Lee, J., Doosti, B., Gu, Y., Cartledge, D., Crandall, D., Raphael, C.,
Observing Pianist Accuracy and Form with Computer Vision,
WACV19(1505-1513)
IEEE DOI 1904
convolutional neural nets, learning (artificial intelligence), musical instruments, Image resolution BibRef

Shlizerman, E., Dery, L., Schoen, H., Kemelmacher-Shlizerman, I.,
Audio to Body Dynamics,
CVPR18(7574-7583)
IEEE DOI 1812
Videos, Avatars, Correlation, Music, Training, Face, Principal component analysis BibRef

Zhou, Y., Wang, Z., Fang, C., Bui, T., Berg, T.L.[Tamara L.],
Visual to Sound: Generating Natural Sound for Videos in the Wild,
CVPR18(3550-3558)
IEEE DOI 1812
Videos, Visualization, Task analysis, Generators, Solid modeling, Music, Gold BibRef

Alexanderson, S., Kjellström, H.,
Computer Analysis of Sentiment Interpretation in Musical Conducting,
FG17(400-405)
IEEE DOI 1707
Computational modeling, Computers, Conductors, Data models, Hidden Markov models, Music, Production BibRef

Salgian, A.[Andrea], Hsu, L.[Leighanne], Milkosky, N.[Nathaniel], Vickerman, D.[David],
Conductor Tutoring Using the Microsoft Kinect,
ISVC15(II: 24-34).
Springer DOI 1601
BibRef

Serafin, S.[Stefania], de Gotzen, A.[Amalia], Dimitrov, S.[Smilen], Gelineck, S.[Steven], Erkut, C.[Cumhur], Nilsson, N.C.[Niels Christian], Grani, F.[Francesco], Nordahl, R.[Rolf], Trento, S.[Stefano],
The digital Intonarumori,
3DUI15(207-208)
IEEE DOI 1511
BibRef

Zielasko, D.[Daniel], Rausch, D.[Dominik], Law, Y.C.[Yuen C.], Knott, T.C.[Thomas C.], Pick, S.[Sebastian], Porsche, S.[Sven], Herber, J.[Joachim], Hummel, J.[Johannes], Kuhlen, T.W.[Torsten W.],
Cirque des bouteilles: The art of blowing on bottles,
3DUI15(209-210)
IEEE DOI 1511
BibRef

Garcia, J., Bresson, J., Carpentier, T.,
Towards interactive authoring tools for composing spatialization,
3DUI15(151-152)
IEEE DOI 1511
authoring systems BibRef

Cabral, M.[Marcio], Montes, A.[Andre], Roque, G.[Gabriel], Belloc, O.[Olavo], Nagamura, M.[Mario], Faria, R.R.A.[Regis R A], Teubl, F.[Fernando], Kurashima, C.[Celso], Lopes, R.[Roseli], Zuffo, M.[Marcelo],
Crosscale: A 3D virtual musical instrument interface,
3DUI15(199-200)
IEEE DOI 1511
BibRef

Fillwalk, J.[John],
ChromaChord: A virtual musical instrument,
3DUI15(201-202)
IEEE DOI 1511
Human Computer Interface BibRef

Lages, W.[Wallace], Nabiyouni, M.[Mahdi], Tibau, J.[Javier], Bowman, D.A.[Doug A.],
Interval Player: Designing a virtual musical instrument using in-air gestures,
3DUI15(203-204)
IEEE DOI 1511
3D input;Virtual musical instrument;in-air gesture BibRef

Moore, A.G.[Alec G.], Howell, M.J.[Michael J.], Stiles, A.W.[Addison W.], Herrera, N.S.[Nicolas S.], McMahan, R.P.[Ryan P.],
Wedge: A musical interface for building and playing composition-appropriate immersive environments,
3DUI15(205-206)
IEEE DOI 1511
Virtual musical instrument;free-hand gesture BibRef

Waranusast, R., Bang-Ngoen, A., Thipakorn, J.,
Interactive tangible user interface for music learning,
IVCNZ13(400-405)
IEEE DOI 1402
augmented reality BibRef

Goodwin, A., Green, R.,
Key detection for a virtual piano teacher,
IVCNZ13(282-287)
IEEE DOI 1402
Hough transforms BibRef

Panagiotakis, C.[Costas], Holzapfel, A.[Andre], Michel, D.[Damien], Argyros, A.A.[Antonis A.],
Beat Synchronous Dance Animation Based on Visual Analysis of Human Motion and Audio Analysis of Music Tempo,
ISVC13(II:118-127).
Springer DOI 1311
BibRef

Oka, A.[Akiya], Hashimoto, M.[Manabu],
Marker-less piano fingering recognition using sequential depth images,
FCV13(1-4).
IEEE DOI 1304
BibRef

Cui, Y.[Yue], Jin, J.S.[Jesse S.], Zhang, S.L.[Shi-Liang], Luo, S.H.[Su-Huai], Tian, Q.[Qi],
Music video affective understanding using feature importance analysis,
CIVR10(213-219).
DOI Link 1007
BibRef

Scarr, J., Green, R.,
Retrieval of guitarist fingering information using computer vision,
IVCNZ10(1-7).
IEEE DOI 1203
BibRef

Lin, Y.T.[Yin-Tzu], Guan, S.H.[Shuen-Huei], Yao, Y.C.[Yuan-Chang], Cheng, W.H.[Wen-Huang], Wu, J.L.[Ja-Ling],
U-Drumwave: An Interactive Performance System for Drumming,
MMMod12(609-620).
Springer DOI 1201
BibRef

Visentini, I.[Ingrid], Rodà, A.[Antonio], Canazza, S.[Sergio], Snidaro, L.[Lauro],
Audio-Video Analysis of Musical Expressive Intentions,
CIAP11(II: 219-228).
Springer DOI 1109
BibRef

Bianco, T.[Tommaso], Freour, V.[Vincent], Rasamimanana, N.[Nicolas], Bevilaqua, F.[Frederic], Caussé, R.[René],
On Gestural Variation and Coarticulation Effects in Sound Control,
GW09(134-145).
Springer DOI 0902
BibRef

Grond, F.[Florian], Hermann, T.[Thomas], Verfaille, V.[Vincent], Wanderley, M.M.[Marcelo M.],
Methods for Effective Sonification of Clarinetists' Ancillary Gestures,
GW09(171-181).
Springer DOI 0902
BibRef

Tuuri, K.[Kai],
Gestural Attributions as Semantics in User Interface Sound Design,
GW09(257-268).
Springer DOI 0902
BibRef

Theune, M.[Mariët], Brandhorst, C.J.[Chris J.],
To Beat or Not to Beat: Beat Gestures in Direction Giving,
GW09(195-206).
Springer DOI 0902
BibRef

Varni, G.[Giovanna], Camurri, A.[Antonio], Coletta, P.[Paolo], Volpe, G.[Gualtiero],
Emotional entrainment in music performance,
FG08(1-5).
IEEE DOI 0809
BibRef

Kozak, M.[Mariusz], Nymoen, K.[Kristian], Godøy, R.I.[Rolf Inge],
Effects of Spectral Features of Sound on Gesture Type and Timing,
GW11(69-80).
Springer DOI 1211
BibRef

Godøy, R.I.[Rolf Inge],
Geometry and Effort in Gestural Renderings of Musical Sound,
GW07(205-215).
Springer DOI 0705
BibRef

Rasamimanana, N.H.[Nicolas H.], Bernardin, D.[Delphine], Wanderley, M.[Marcelo], Bevilacqua, F.[Frédéric],
String Bowing Gestures at Varying Bow Stroke Frequencies: A Case Study,
GW07(216-226).
Springer DOI 0705
BibRef

Rasamimanana, N.H.[Nicolas H.], Fléty, E.[Emmanuel], Bevilacqua, F.[Frédéric],
Gesture Analysis of Violin Bow Strokes,
GW05(145-155).
Springer DOI 0505
BibRef

Marshall, M.T.[Mark T.], Malloch, J.[Joseph], Wanderley, M.M.[Marcelo M.],
Gesture Control of Sound Spatialization for Live Musical Performance,
GW07(227-238).
Springer DOI 0705
BibRef

Sim, T.[Terence], Ng, D.[Dennis], Janakiraman, R.[Rajkumar],
VIM: Vision for Interactive Music,
WACV07(32-32).
IEEE DOI 0702
BibRef

Gorodnichy, D.O.[Dmitry O.], Yogeswaran, A.,
Detection and tracking of pianist hands and fingers,
CRV06(63-63).
IEEE DOI 0607
BibRef

Ilmonen, T.[Tommi], Takala, T.[Tapio],
Detecting Emotional Content from the Motion of an Orchestra Conductor,
GW05(292-295).
Springer DOI 0505
BibRef

Godøy, R.I.[Rolf Inge], Haga, E.[Egil], Jensenius, A.R.[Alexander Refsum],
Playing 'Air Instruments': Mimicry of Sound-Producing Gestures by Novices and Experts,
GW05(256-267).
Springer DOI 0505
BibRef

Camurri, A.[Antonio], Castellano, G.[Ginevra], Ricchetti, M.[Matteo], Volpe, G.[Gualtiero],
Subject Interfaces: Measuring Bodily Activation During an Emotional Experience of Music,
GW05(268-279).
Springer DOI 0505
BibRef

Carreira, J., Peixoto, P.,
A Vision Based Interface for Local Collaborative Music Synthesis,
FGR06(591-596).
IEEE DOI 0604
BibRef

Ke, Y.[Yan], Hoiem, D.[Derek], Sukthankar, R.[Rahul],
Computer Vision for Music Identification,
CVPR05(I: 597-604).
IEEE DOI 0506
BibRef
And:
Computer Vision for Music Identification: Video Demonstration,
CVPR05(II: 1184).
IEEE DOI 0507
BibRef

Salgian, A.[Andrea], Pfirrmann, M.[Micheal], Nakra, T.M.[Teresa M.],
Follow the Beat? Understanding Conducting Gestures from Video,
ISVC07(I: 414-423).
Springer DOI 0711
BibRef

Vines, B.W.[Bradley W.], Wanderley, M.M.[Marcelo M.], Krumhansl, C.L.[Carol L.], Nuzzo, R.L.[Regina L.], and Levitin, D.J.[Daniel J.],
Performance Gestures of Musicians: What Structural and Emotional Information Do They Convey?,
GW03(468-478).
Springer DOI 0405
BibRef

Wanderley, M.M.[Marcelo M.],
Non-obvious Performer Gestures in Instrumental Music,
GW99(37).
Springer DOI 9903
BibRef

Dahl, S.[Sofia], Friberg, A.[Anders],
Expressiveness of Musician's Body Movements in Performances on Marimba,
GW03(479-486).
Springer DOI 0405
BibRef

Florens, J.L.[Jean-Loup],
Expressive Bowing on a Virtual String Instrument,
GW03(487-496).
Springer DOI 0405
BibRef

Cirotteau, D.[Damien], de Poli, G.[Giovanni], Mion, L.[Luca], Vidolin, A.[Alvise], Zanon, P.[Patrick],
Recognition of Musical Gestures in Known Pieces and in Improvisations,
GW03(497-508).
Springer DOI 0405
BibRef

Leman, M.[Marc], Vermeulen, V.[Valery], de Voogdt, L.[Liesbeth], Taelman, J.[Johannes], Moelants, D.[Dirk], Lesaffre, M.[Micheline],
Correlation of Gestural Musical Audio Cues and Perceived Expressive Qualities,
GW03(40-54).
Springer DOI 0405
BibRef

Godøy, R.I.[Rolf Inge],
Gestural Imagery in the Service of Musical Imagery,
GW03(55-62).
Springer DOI 0405
BibRef

Camurri, A.[Antonio], Canepa, C.[Corrado], Ghisio, S.[Simone], Volpe, G.[Gualtiero],
Automatic Classification of Expressive Hand Gestures on Tangible Acoustic Interfaces According to Laban's Theory of Effort,
GW07(151-162).
Springer DOI 0705
BibRef

Glowinski, D.[Donald], Camurri, A.[Antonio], Chiorri, C.[Carlo], Mazzarino, B.[Barbara], Volpe, G.[Gualtiero],
Validation of an Algorithm for Segmentation of Full-Body Movement Sequences by Perception: A Pilot Experiment,
GW07(239-244).
Springer DOI 0705
BibRef

Mazzarino, B.[Barbara], Peinado, M.[Manuel], Boulic, R.[Ronan], Volpe, G.[Gualtiero], Wanderley, M.M.[Marcelo M.],
Improving the Believability of Virtual Characters Using Qualitative Gesture Analysis,
GW07(48-56).
Springer DOI 0705
BibRef

Camurri, A.[Antonio], Mazzarino, B.[Barbara], Ricchetti, M.[Matteo], Timmers, R.[Renee], Volpe, G.[Gualtiero],
Multimodal Analysis of Expressive Gesture in Music and Dance Performances,
GW03(20-39).
Springer DOI 0405
BibRef

Camurri, A.[Antonio], Mazzarino, B.[Barbara], Volpe, G.[Gualtiero],
Analysis of Expressive Gesture: The EyesWeb Expressive Gesture Processing Library,
GW03(460-467).
Springer DOI 0405
BibRef

Shao, X.[Xi], Xu, C.S.[Chang-Sheng], Kankanhalli, M.S.[Mohan S.],
A new approach to automatic music video summarization,
ICIP04(I: 625-628).
IEEE DOI 0505
BibRef
Earlier:
Automatically generating summaries for musical video,
ICIP03(II: 547-550).
IEEE DOI 0312
BibRef

Murphy, D.[Declan], Andersen, T.H.[Tue Haste], Jensen, K.[Kristoffer],
Conducting Audio Files via Computer Vision,
GW03(529-540).
Springer DOI 0405
BibRef

Modler, P.[Paul], Myatt, T.[Tony],
A Video System for Recognizing Gestures by Artificial Neural Networks for Expressive Musical Control,
GW03(541-548).
Springer DOI 0405
BibRef

Arfib, D.[Daniel], Couturier, J.M.[Jean-Michel], Kessous, L.[Loïc],
Design and Use of Some New Digital Musical Instruments,
GW03(509-518).
Springer DOI 0405
BibRef

Segen, J.[Jakub], Gluckman, J.M.[Joshua M.], Kumar, S.[Senthil],
Visual Interface for Conducting Virtual Orchestra,
ICPR00(Vol I: 276-279).
IEEE DOI 0009
BibRef

Segen, J., Kumar, S.,
Human-computer interaction using gesture recognition and 3D hand tracking,
ICIP98(III: 188-192).
IEEE DOI 9810
BibRef

Fels, S., Mase, K.,
Iamascope: A Musical Application for Image Processing,
AFGR98(610-615).
IEEE DOI BibRef 9800

Chapter on Face Recognition, Human Pose, Detection, Tracking, Gesture Recognition, Fingerprints, Biometrics continues in
American Sign Language, ASL Recognition .