A Multi-Modal Conducting Simulator

Abstract

Soft-computing applications for music are increasing in recent years. They are regarded as suitable solutions for musical information processing. In this paper, an application for a multi-modal conducting simulator is mentioned. It recognizes human conducting and behaves like an orchestra in real-time. Even for the same conducting intention, both the gesture characteristics and the time-domain structure vary depending on the conductor, and on the connection between previous or latter parts. Soft-computing is applied for recognition to cope with both these variance and to detect conductor's intention. The beat recognition process is built on the model of actual musicians' recognition processes. The right-hand conducting gesture is captured by accelerometers and recognized with HMM(Hidden Markov Models). Based on interviews with conductors and orchestra players, musical score information and performance information are also applied to the recognition on the fuzzy model. The watching point and breath of a user is detected with a eye tracker and a breath sensor, respectively. To shape the musical performance in real-time, the system recognizes important and universal conducting elements conforming to the grammar of conducting : i.e., beginning and ending of a piece, Einsatz(cue for a player or a part, an important direction by a conductor)with eye, beat timing including Agogik(rubato, changes in tempo), the beat number in a measure, fermata(to lengthen notes or rests arbitrarily, to pause), Dynamik(dynamics), some aspects of articulation(i.e.espressivo-staccato), and breathing. The system simulates orchestra response to correct breathing and Einsatz by a user. It also simulates sounding delay and autonomy of actual orchestra. When the autonomous degree is 100%, the system ignores the conducting input and plays with tempo on music data. When the autonomous degree is 0%, the system is glued to the conducting. In this case, when the conducting gesture stops, the system stops replay simultaneously and waits until the next beat input.