Language-queried target speech extraction using para-linguistic and non-linguistic prompts

Kentaro Seki; Nobutaka Ito; Kazuki Yamauchi; Yuki Okamoto; Kouei Yamaoka; Yuki Saito; Shinnosuke Takamichi; Hiroshi Saruwatari

doi:10.1250/ast.e25.27

Kentaro Seki, Nobutaka Ito, Kazuki Yamauchi, Yuki Okamoto, Kouei Yamaoka, Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari

Author information

Kentaro Seki
Graduate School of Information Science and Technology, The University of Tokyo
Nobutaka Ito
Graduate School of Frontier Sciences, The University of Tokyo
Kazuki Yamauchi
Graduate School of Information Science and Technology, The University of Tokyo
Yuki Okamoto
Graduate School of Information Science and Technology, The University of Tokyo
Kouei Yamaoka
Graduate School of Information Science and Technology, The University of Tokyo
Yuki Saito
Graduate School of Information Science and Technology, The University of Tokyo
Shinnosuke Takamichi
Graduate School of Information Science and Technology, The University of Tokyo
Graduate School of Science and Technology, Keio University
Hiroshi Saruwatari
Graduate School of Information Science and Technology, The University of Tokyo

Keywords: Target speech extraction, Language-queried audio source separation, Non-linguistic information, Para-linguistic information

JOURNAL OPEN ACCESS Advance online publication

Article ID: e25.27

DOI https://doi.org/10.1250/ast.e25.27

Details

Abstract

This paper proposes a new language-queried target speech extraction (TSE) task called para-linguistic and non-linguistic text prompts-based TSE (PNTP-TSE), which uses text prompts that describe para-linguistic and non-linguistic information. This framework addresses the limitations of conventional TSE methods, such as privacy concerns in voiceprint-based systems and dependency on dedicated microphone arrays or video cameras. To support this framework, we construct and provide a new dataset, PromptTSE, which is specifically designed to facilitate various types of language-queried TSE, including PNTP-TSE. We develop a baseline method for PNTP-TSE and conduct experimental evaluations. The experimental results show that PNTP-TSE overcomes the performance degradation issue of voiceprint-based systems caused by the gap in speaking style between enrollment speech and target speech.

Corresponding author

Register with J-STAGE for free!