Accuracy of Commercially-Available Speech Recognition Systems in Identifying PIREP Terminology

<p>Pilot Reports (PIREPs) are an important source of information that aids, other pilots, air traffic control, and operational aviation meteorologists in terms of forecasting and updating weather advisories such as SIGMETs. Pilots rely upon PIREPs so they can avoid hazardous weather and fly their aircraft in the safest manner possible. However, many PIREPs are not successfully submitted or transmitted to the many end users which impedes their ability to be used to keep the NAS safe. The National Transportation Safety Board (NTSB) made several recommendations for increasing the effectiveness and distribution of PIREPs, including receiving PIREPs from pilots directly and automatically (NTSB, 2017). We recruited eighty-four native-speaking participants to read a short, average, and long PIREP scripts in order to test the performance of various speech recognition systems (SRSs). The spoken PIREPs were transcribed by SRSs and compared to the original PIREP scripts. The words that were deleted, substituted, and inserted were identified and used to calculate the word error rate (WER) and word information loss (WIL). The WERs and WILs were separately analyzed with a repeated-measures marginal model to compare the accuracy between each of the SRSs. Also, the interaction between each SRS and gender was analyzed. The results demonstrated that Google, LilySpeech, and Transcribe had the same and superior performance when transcribing the average-length PIREPs than Braina and Dragon. All SRSs had equal performance at transcribing the short-length PIREPs. Dragon, Google, LilySpeech, and Transcribe had the same performance and superior when transcribing the long-length PIREPs than Braina. Additionally, we found that the short, average, and long-length transcriptions for all 5 commercial off-the-shelf (COTS) SRSs provided readable information for flight service stations (FSS) to enter valuable weather information into the PIREP system.</p>