오픈AI의 Whisper ASR에서 단어 수준의 타임스탬프를 어떻게 얻을 수 있나요?

나는 OpenAI의 'Whisper' 파이썬 라이브러리를 사용하여 음성 인식을 합니다. 어떻게 단어 수준의 타임스탬프를 얻을 수 있을까요?

오픈AI의 'Whisper'로 전사하는 방법 (Ubuntu 20.04 x64 LTS 및 Nvidia GeForce RTX 3090에서 테스트됨):

'conda create -y --name whisperpy39 python==3.9

conda activate whisperpy39

pip install git+https://github.com/openai/whisper.git

sudo apt update && sudo apt install ffmpeg

whisper recording.wav

whisper recording.wav --model large

'

만약 Nvidia GeForce RTX 3090을 사용하는 경우, 다음을 추가합니다. 'conda activate whisperpy39' :

'pip install -f https://download.pytorch.org/whl/torch_stable.html

conda install pytorch==1.10.1 torchvision torchaudio cudatoolkit=11.0 -c pytorch

'

답변 1

나는 단어 수준의 타임스탬프(그리고 확신도)를 복구하기 위한 리포지토리를 생성했습니다. 또한 보다 정확한 세그먼트 타임스탬프도 포함되어 있습니다:

'https://github.com/Jeronymous/whisper-timestamped'

이것은 Whisper의 교차-주의 가중치를 기반으로 만들어진 것입니다. 'this notebook in the Whisper repo'와 같이 작성되었습니다. 제가 약간 접근 방식을 조정하여 더 나은 위치를 얻을 수 있도록 하였으며, 교차-주의를 실시간으로 받아볼 수 있는 기능을 추가하여 Whisper 모델을 두 번 실행할 필요가 없습니다. 긴 오디오를 처리할 때 메모리 문제는 없습니다.

주의 : 먼저, 와브투벡스 모델을 사용하여 휘스퍼의 변환된 단어를 입력 오디오에 맞게 다시 정렬하는 방법을 시도했습니다. 이 방법은 상당히 잘 작동하지만 여러 가지 단점이 있습니다. (와브투벡스) 모델을 처리할 별도의 모델을 사용해야 하며, 전체 신호에 대해 다른 추론을 수행하고 언어당 하나의 와브투벡스 모델을 가져와야 하며, 변환된 텍스트를 정규화하여 문자 집합이 와브투벡스 모델의 문자 집합과 일치하도록 해야 합니다(예: 숫자를 문자로 변환, %와 같은 기호, 통화...). 또한 정렬에는 보통 휘스퍼에서 제거되는 어색한 부분에 문제가 발생할 수 있습니다(따라서 문장의 시작 부분은 다시 구성됩니다).

답변 2

Title: Whisper ASR: Unlocking Word-Level Timestamps and Enhancing Speech Recognition Accuracy

Introduction:

As technology continues to advance, automatic speech recognition (ASR) systems have become increasingly valuable in various applications. OpenAI's Whisper ASR is a remarkable advancement in this domain, offering accurate speech recognition capabilities. To enhance its functionality and usability, many users are interested in obtaining word-level timestamps. In this essay, we will explore the significance of word-level timestamps in Whisper ASR and propose effective methods to obtain them.

1. The Importance of Word-Level Timestamps in Whisper ASR:

Word-level timestamps provide precise timing information for each word in an audio transcription. By incorporating these timestamps in Whisper ASR, numerous advantages can be attained:

a) Improved Transcription Analysis: Word-level timestamps enable researchers, content creators, and language learners to analyze spoken content more comprehensively. They can easily locate specific words or phrases within a transcript, facilitating deeper understanding and effective analysis.

b) Enhanced Accessibility: Word-level timestamping is invaluable for individuals with hearing impairments or those who require text-based captions to comprehend spoken content. This feature ensures accurate synchronization of the transcript with the audio, enabling a superior user experience.

c) Efficient Document Navigation: Word-level timestamps enable easy navigation within the transcription. Users can quickly jump to desired sections, facilitating efficient data retrieval, editing, and sharing of relevant speech segments.

2. Methods to Obtain Word-Level Timestamps in Whisper ASR:

a) Utilizing External Speech Recognition Tools: While Whisper ASR itself generates sentence-level timestamps, word-level timestamps can be obtained by utilizing external tools such as Gentle, Gentle Forced Aligner, or Kaldi. These tools align the audio with the transcript, resulting in accurate word-level timestamps.

b) Manual Annotation: Another approach involves manually annotating the audio by aligning each word with its corresponding timestamp. Although this method requires more effort, it guarantees precise word-level timestamps when performed meticulously.

c) Post-Processing Techniques: Post-processing techniques can be applied to sentence-level timestamp data from Whisper ASR to approximate word-level timestamps. By leveraging natural language processing algorithms, speech patterns, and pauses, one can infer word boundaries and assign timestamps accordingly.

3. The Future of Whisper ASR and Word-Level Timestamps:

The integration of word-level timestamps in Whisper ASR is an essential feature that OpenAI may consider in future updates. OpenAI has consistently exhibited its commitment to improving user experience by actively listening to user feedback. By incorporating this valuable feature, Whisper ASR has the potential to cement its position as a leading ASR system.

Conclusion:

Word-level timestamps are instrumental in maximizing the potential of Whisper ASR, enabling users to extract valuable insights, improving accessibility, and facilitating efficient data navigation. While external tools, manual annotation, and post-processing techniques can currently assist in obtaining word-level timestamps, their seamless integration into Whisper ASR would undoubtedly enhance the overall user experience. OpenAI's continued innovation and responsiveness make it highly likely that such advancements will be introduced in the near future. With word-level timestamps, Whisper ASR's utility and versatility would reach new heights, benefiting users across diverse domains.