AM31RAT - Řečové aplikace v telekomunikacích Robustní řečové parametrizace Doc. Ing. Petr Pollák, CSc. 3. března 11-1:34
Obsah přednášky Příznaky pro rozpoznávání řeči Vlastnosti příznaků na bázi DFT, LPC a kepstra MFCC a PLP Statické a dynamické příznaky Techniky robustní parametrizace (příznaky pro rozpoznávání) Spektrální odečítání CMS VTLN Kompenzace Lombardova jevu
I. část Příznaky pro rozpoznávání řeči
Parametrizace řečového signálu parametrizace = extrakce příznaků popisujících signál pro účely následné klasifikace používané spektrální (kepstrální příznaky) DFT spektrum - méně používané pro množství redundantní informace AR koeficienty - nevhodné, koeficienty polynomu AR model, koeficienty odrazu - možné, stabilnější Kepstrum - tvoří shluky v n-dimenzionální prostoru - OK LPC kepstrum - méně používané pro malou robustnost MFCC = mel-kepstrum - nejpoužívanější příznaky (modelování nelinearity lidského slyšení) PLP - alternativní způsob modelování nelinearity slyšení dynamické a akcelerační parametry (delta, delta-delta) energetické parametry ( E, ln E, c ) Další parametrizace výpočet pomocí neuronových sítí TRAPs - časové trajektorie příznaků v delších kontextech
MFCC - Melovské kepstrální koeficienty Blokové schéma výpočtu mel-kepstrálních koeficientů: sn DFT Xk Mel BF mk ln(.) ln(mk) IDCT cn Výpočet energie v jednom pásmu N/ g j = ln S[k] H mel,j [k]. k= Výpočet kepstra pomocí DCT c i = P P j=1 ( ) πi g j cos (j.5) P
PLP - kepstrální koeficienty Blokové schéma výpočtu PLP kepstrálních koeficientů: sn DFT Xk E(f) Xsk CBA Bsj.3 Bj IDFT Rk LPC ak ac cn aplikuje se dříve diskutovaná PLP banka filtrů kepstrum se počítá na bázi lineární predikce rozdílné šumové vlastnosti
Statické a diferenciální parametry c 1 [i] 1 [i] δ 1 [i] c [i] c[i] =., [i] = [i]., δ[i] = δ [i]. c p [i] n [i] δ n [i] Dynamické parametry (odhad 1. derivace základních příznaků) MX m (c k [i + m] c k [i m]) m=1 k [i] = pro 1 k n MX m m=1 Akcelerační parametry (odhad. derivace základních příznaků) MX m ( k [i + m] k [i m]) m=1 δ k [i] = pro 1 k n MX m m=1
II. část Základní robustní parametrizační techniky
pravdepod pro kazd neuro 9 tody jsou zaloz eny na spektra lnı m odec ı ta nı vloz eny m ^Xi GMM/ Robustní parametrizace řeči pro kompenzaci aditivního šumu 14 HMM do standardnı ch parametrizac nı ch postupu, viz pr ı klad Podrobne js ı info vy poc tu melovsky ch kepstra lnı ch koeficientu. x c Ni + - standardnı +pr ı znaky Mel-kepstrum r ec ove MFCC -DFT mel-frekvenční kepstrální koeficienty (DCT) WF IDFT p Ni Si Modifikovane spektra lnı odec ı ta nı z 1 Metoda pro potlac enı. aditivnı ho s umu + ^Xi sbi Bor il, H., Polla k, P.: De /u/ Bor il, H.,1 Polla k, P.: C ASIDE 5,3 COST7 DCT 1.8.6.1. +.3.4.5 Blok MSO.6.7 Xi.9. 4 3 1 ff Podrobne Bor il, H., P Bor il, H., P ASIDE Si MBF ln DCT Ni 1 1 p Blokz 1MSO + p Blok MSO lze umístit Ni Si i za MBF či za logaritmus (z hlediska vyhlazení je vhodné 1 umístění přímo za DFT) z databa zı MSO Si 1 5 7.8 b xmfcc c + pozadí i Robustnı Xi vlivu aditivního N mel-kepstrum - standardnı pr ı znakysi s eliminací rušivého i r ec ove + WFc IDFTcbs,i xi DFT DFT 6. 7 1.4 8 cx,i Počet segmentů (x 1 ) Xi Si 1 M N 1 pmi i i 1 DFT MBF ln 1 + z Habs pravdepodobnosti fonemu pro kazdy segment xi 17 1 Xi i neuronova sit 16 7 Hardwarove impleme
Výsledky rozpoznávání řeči se standardní parametrizací 43 1 Rozpoznávač izolovaných slov, standardní MFCC parametrizace Trénování i testování - zašuměná řeč s různou úrovní šumu -/., -/. +* ')(* $% &!#" ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) 4 ) ) - ) )! " #$ #%" &(' (*,+.- +/*1 *3 45.*36 --.*1-(*78 54.*74:9! -.*74 -.*364 9*39.*3 *1-8 -4(*5.*75- +;*79 98*3<+ -(*7 98*3<+ +/*1=+ 4:98*,+ -5.*745 49*3-.* (*39.*754 +/*1=+ (*>9-54.*396 #? +. (*3@ 958*7 --(* 45.*7-5 *14=+ :*7 *7-
Výsledky rozpoznávání řeči s předzpracováním řeči s šumem 3 1 / Rozpoznávač izolovaných slov, standardní MFCC parametrizace Trénování i testování - zašuměná řeč, spektrální odečítání,.-+,.- *) &(') #$ %!"! " #$ #%" & ')(*'+ ',-(*',+ ')(*+ ',.-(*/ +-(1 /3(*. /3(*.,! ')( & '/3(*+,' '4(5 & ',-(5 '.-(6.' +)(*.3 /3(*',. ')(5 ',-( & ')(* '/3(5 & ' & (1' ')(*'. +,-( &,& ' & (* ',-(*',+ ')(* '/3(5 & ')(5 ' & (* & +,-(5, #7 +)(5' +,'-(*.,+ '3/(*. '/3(*./ '.-(6'3 '.)(5 +,'-(*4
Robustní parametrizace řeči pro kompenzaci aditivního šumu Spektrální odečítání lze použít různé techniky pro zvýrazňování řeči (obvykle spíše jednokanálové systémy) odstranění šumu se provede pouze ve frekvenční oblasti zpětný převod do časové oblasti není nutný otázka vlivu zkreslení řečového signálu (potlačení informace) Možnost zařazení potlačení aditivního rušení ve frekvenční oblasti standardní MFCC, PLP různý vliv na jednotlivé techniky (LPC vs. DCT) TRAPs (základem je též banka filtrů) - nadějné, zatím spíše v základním výzkumu
CMS - Cepstral Mean Subtraction Potlačení konvolučního rušení (především vlivu kanálu) Vstupní předpoklad: x[n] = s[n] h[n] c x = c s + c h Průměrné kepstrum všech L segmentů signálu je dané vztahem c x = 1 L 1 c s [i] + c h = c s + c h L i= pro dostatečné množství segmentů platí c s. c x = ĉ h ĉ s [i] = c x [i] c x = c s [i] + c h ĉ h c s [i]
VTLN - Vocal Tract Length Normalization Zvýšení robustnosti kompenzací variability mezi mluvčími Výchozí předpoklad - délka vokálního traktu je nepřímo úměrná formantovým frekvencím VTL = (i 1) c 4F i Řešení - warpovací funkce a warpovací faktor pro transformaci frekvenční osy
VTLN - Vocal Tract Length Normalization Ilustrační obrázek vlivu warpování frekvenční osy Převzato z: David Suendermann, Guntram Strecha, Antonio Bonafonte, Harald Hoege, Hermann Ney: Evaluation of VTLN-Based Voice Conversion for Embedded Speech Synthesis, In Interspeech 5.
VTLN - Vocal Tract Length Normalization Ilustrace principu warpovacích funkcí Převzato z: Xiaodong Cui and Abeer Alwan: MLLR-Like Speaker Adaptation Based on Linearization of VTLN with MFCC Features. In Interspeech 5.
VTLN - Vocal Tract Length Normalization Nelineární zkreslení - warpovací faktor α :, 88 α 1, 1 ( ) (1 α) sin π f f η α (f ) = f 1 + arctan mez 1 (1 α) cos π f f mez 4 35 3 Warped frequency 5 15 1 5 5 1 15 5 3 35 4 Linear frequency warpují se často jen mezní frekvence použité BF (MFCC, PLP) parametr α se odhaduje maximalizací pravděpodobnosti (analogie trénování)
VTLN - Vocal Tract Length Normalization Lineární zkreslení - warpovací faktor α :, 88 α 1, 1 αf pro f < f o f η α (f ) = mez αf o + αf o f mez (f f o ) f o pro f o f < f mez 45 4 35 3 Warped frequency 5 15 1 5 5 1 15 5 3 35 4 Linear frequency aproximace nelineární warpovací funkce parametr α se odhaduje opět trénováním
Lombardův jev (LE) Změny v produkci řeči pod vlivem šumu zvýšení intezity promluvy posun základního tónu řeči (při rozpoznávání menší vliv) posun formantových kmitočtů VÝRAZNÝ VLIV NA ROZPOZNÁVÁNÍ podobný efekt má vliv stresu či jiných emocí (emotional speech recognition)
ree Car 7 gine Off ngine Off gine On ngine On 7 s. voice intensity rises significantly for the Lombard speech, see possible Fig.. evaluate only the power of mixture σ, as the 4 Přítomnost Lombardova jevu v dostupných DB xi signal is supposed to contain noise all the time. Powers of 3.. Fundamental frequency speech and noise ( ˆ σ s, i and ˆn σ, i ) have to be estimated. Noise f power was is tracked estimated in in the speech WaveSurfer pause by standard [1]. Tracking exponential was performed estimation in voiced parts of all neutral and noisy speech utterances. ˆ = In p the ˆ graph + (1 descriptions, p) for letters =, F and () M σ σ σ VAD ni, ni, 1 xi, i represent female and male data respectively. ˆ σ ˆ ni, = σni, 1 for VADi = 1. (3) 1 If the speech SPEECON and noise - Fundamental signals can Frequency be considered Distribution to be uncorrelated, 1 speech power can be estimated by subtraction of noise power from the mixture power 8 ˆ σ ˆ Office_F s, i = σx, i σ,. (4) n i Car_F In 6principle, the algorithm estimates standard global Office_M SNR evaluated over speech activity regions only. The segmental Car_M approach 4 and the averaging of linear power ratios give lower estimation error [8]. Precision of the estimation is very sensitive to correct VAD classification. Detector based on differential cepstral analysis was used. The details are described 7 in [9]. 17 7 37 47 57 Frequency (Hz) Number of Frames (x 1 ) er of Utterances Number of Utterances 3 5 15 1 5 Figure SPEECON 3: SPEECON Channel f SNR distribution. Histograms Close-talk Office Hands-free Office Close-talk Car Hands-free Car -1 1 3 5 7 SNR (db) 5 CZKCC Channel SNR Histograms 15 Close-talk Engine Off Distant Mike Engine Off Hands-free LE evaluated over speech activity regions only. The segmental approach and the averaging of linear power ratios give lower estimation error [8]. Precision of the estimation is very sensitive to correct VAD classification. Detector based on -1 1 3 5 7 differential cepstral analysis was SNR (db) used. The details are described in [9]. Figure : CLSD channel SNRs. Number of Frames (x 1) Number of Utterances Number Number of of Frames Utterances (x 1 ) N Since CZKCC and CLSD were recorded by two 3 SPEECON Channel SNR Histograms microphones, SPEECON SNR distributions are also 16depicted only CZKCC for - the Fundamental first two Frequency channels. Distribution In CZKCC a 5 directional microphone was used in the distant position, 14 which explains higher average SNR in the distant 1microphone engine on channel than in SPEECON. 1 15 Sometimes it is necessary to modify gain of the Close-talk Office Off_F 8microphone preamplifier during the recording Hands-free session Drv_F Office to avoid Off_M 6 1 signal clipping when speaker changes Close-talk voice Car intensity. Drv_M In consequence, it becomes impossible Hands-free to evaluate Car voice 4 intensity 5 changes directly from the amplitude of the recorded speech signal. In case the ambient noise can be considered stationary, relative voice intensity changes can be estimated -1 from 7 the SNR 17 1 even 7 with 3 gain 37 being 5 changed 47 7during 57the Frequency SNR (db) (Hz) session. Moreover, if the absolute level of the ambient noise was 5known, absolute level of vocal intensity could be 7 CZKCC Channel SNR Histograms estimated (but CLSD it was - not Fundamental our case). Frequency Distribution 6 In SPEECON and CZKCC environmental characteristics changed significantly when comparing office and car or 5 standing car with engine off and moving car scenarios, but in 4case 15of CLSD ambient noise can be considered Clean_F stationary Close-talk Engine LE_F Off and thus SNR histograms relate to overall vocal intensity 3 Clean_M changes in neutral and Lombard speech. Distant It Mike is Engine obvious Off LE_M that 1 voice intensity rises significantly for the Lombard speech, Close-talk Engine On see Fig.. 1 5 Distant Mike Engine On 3.. Fundamental frequency 7 17 7 37 47 57 f was tracked in the Frequency WaveSurfer (Hz) [1]. Tracking was -1 1 3 5 7 performed in voiced parts SNR of (db) all neutral and noisy speech utterances. Figure In 4: CZKCC the graph and descriptions, CLSD f distribution. letters F and M represent Figure 1: female SPEECON and male and data CZKCC respectively. channel SNRs. In case 1 of SPEECON, see Fig. 3, and CZKCC, Fig. 4, shifts SPEECON - Fundamental Frequency Distribution in f distribution are observable but not significant. In case of 1 CLSD, Fig. 4, maximum of the LE male f distribution appears 8at the higher frequency than maximum of neutral Office_F female distribution while female maximum moves to ames (x 1 ) 6 Car_F microphone avoid signal In consequen intensity cha speech signa stationary, re from the SN session. Mor was known, estimated 4(bu In SPEEC changed sign standing car case of CLS 18 and thus SN changes 16 in n voice intensi 14 see Fig.. F (Hz) 1 3.. Fundam 1 f was track 3 performed in utterances. I 4 represent fem Number of Frames (x 1 ) 1 1 F (Hz) 8 6 4 7 18 16 14 1 1 3 F Changes observed Formant changes i be observ of first tw
In case of SPEECON, see Fig. 3, and CZKCC, Fig. 4, shifts in f distribution are observable but not significant. In case of CLSD, Fig. 4, maximum of the LE male f distribution appears at the higher frequency than maximum of neutral Changes in first two formant F 1, F locations can be observed for SPEECON, Fig. 5, and CZKCC, Fig. 6. Formant bandwidths did not display any systematical changes in different scenarios. Significant formant shifts can 3.3. Formants Přítomnost Formant analysis Lombardova was performed on utterances jevu containing v dostupných DB Number of Frames (x 1) Number of Frames (x 1 ) digits. Monophone HTK [11] recognizer trained on 7 SPEECON office sessions was used for the forced alignment. 1 th order LPC was chosen for formant tracking performed by the WaveSurfer. Information about first four formant frequencies and bandwidths were assigned to corresponding phonemes. In the following figures, positions of first two female formants of the selected vowels appearing in Czech digits are presented. CZKCC - Fundamental Frequency Distribution 4 16 SPEECON - Female Vowel Formants /i/ Office vs. Car 14 /i'/ /e/ 1 Neutral 1 18 Off_F /e'/ LE 8 Drv_F 16 Off_M /a'/ 6 Drv_M /o'/ 14 4 /u'/ /a/ /o/ 1 /u/ 1 7 17 7 37 47 57 3 4 45 55 65 35 5 6 7 Frequency (Hz) F1 (Hz) 7 6 5 4 3 1 F (Hz) Figure 5: Positions of female F 1, F SPEECON. CLSD - Fundamental Frequency Distribution containing digits. Difference in phoneme duration in the same word uttered in two different scenarios was evaluated as shown in Eq. 5. TC TC = T 1 (%), (5) 1 C1 T Cx represents average phoneme duration in scenario x. In SPEECON, phoneme duration differences did not exceed 38 %. In case of CZKCC, greatest duration changes were observed in the word 'štiri' (phoneme /r/ 79 %) and in the word 'sedm' (phoneme CZKCC - Female /e/ 73 Vowel %). Formants Most significant phoneme duration /i/ /i'/ differences Engine Off were vs. Engine observed Drivein the CLSD database, e.g. in word jedna (/e/ 161 %), pjet (/e/ 174 %), devjet ( nd /e/ 177 %). No /e'/ systematical changes in /e/ Neutral word duration were observed in SPEECON. LE Word # N TN (s) σtn (%) # LE TLE (s) /a/ σtle (%) (%) Nula 349,475 11,68 /o'/ 36,56 34,48 17,8 /u'/ /a'/ /o/ Jedna 69,559 /u/ 13,6 51,67 6,7 8,58 1 Dvje 35 454,46 45 1,56 5 55 55,483 6 653,517 13,57 3 Figure 7: CZKCC - word duration changes. 4 CLSD - Female Vowel Formants /i/ /i'/ /e'/ Oproti předpokladu je LE přítomen v menší /e/ míře Clean_F 18 LE_F (Čtené promluvy, Clean_M bez zpětné vazby) /a'/ 16 LE_M /a/ 14 /u'/ /o'/ Neutral 1 /o/ LE /u/ 1 7 17 7 37 47 57 3 4 5 6 7 8 9 Frequency (Hz) F1 (Hz) F (Hz) F (Hz) 4 18 16 14 1 F1 (Hz) Figure 4: CZKCC and CLSD f distribution. Figure 6: Female F 1, F CZKCC and CLSD.
1 Off_F Off_F 18 8 8 Drv_F Drv_F Analýza přítomnosti LE Off_M v CLSD5 16 Off_M Number of Frame 6 4 Number of Frame 6 4 Drv_M Drv_M 7 17 7 37 47 57 7 17 7 37 47 57 Frequency (Hz) Frequency (Hz) F (Hz) 14 1 1 F (Hz) 18 16 14 1 1 /u'/ /u/ /u'/ /u/ 3 35 3 435 45 4 5 45 555 6 55 656 7 65 7 F1 (Hz) /o'/ /o/ F1 (Hz) /a/ /o'/ /a'/ /o/ LE /a/ /a'/ LE Number of Frames (x 1 ) 7 6 5 4 3 1 Number of Frames (x 1 ) 7 CLSD - Fundamental CLSD - Fundamental Frequency Frequency Distribution Distribution 6 5 4 Clean_F Clean_F LE_F LE_F 3 Clean_M Clean_M LE_M LE_M 1 7 17 7 37 47 57 7 17 7 37 47 57 Frequency (Hz) Frequency (Hz) F (Hz) 4 4 CLSD - Female CLSD - Vowel Female Formants Vowel Formants /i/ /i'/ /i/ /i'/ /e'/ /e'/ /e/ /e/ 18 18 /a'/ /a'/ 16 16 /a/ /a/ 14 14 /u'/ /o'/ /u'/ /o'/ Neutral Neutral 1 1 /o/ /o/ LE LE /u/ /u/ 1 1 F (Hz) 3 3 4 4 5 5 6 6 7 7 8 8 9 9 F1 (Hz) F1 (Hz) Figure 4: Figure CZKCC 4: CZKCC and CLSD and f CLSD distribution. f distribution. Figure 6: Figure Female 6: Female 1, F FCZKCC 1, F CZKCC and CLSD. and CLSD. nnel distribution 16 CLSD Channel SNR Histograms In case of In SPEECON, case of SPEECON, see Fig. see 3, and Fig. CZKCC, 3, and CZKCC, Fig. 4, shifts Fig. 4, shifts Changes Changes in first in two first formant two formant F 1, F Flocations 1, F locations can be can be egmental SNR was evaluated as 14 in f distribution in f distribution are observable are observable but not significant. but not significant. In case of In case of observed observed for SPEECON, for SPEECON, Fig. 5, Fig. and 5, CZKCC, and CZKCC, Fig. 6. Fig. 6. L CLSD, ˆ 1 σ sclsd, Fig., j 4, Fig. maximum 4, maximum of the LE of the male LE f male distribution f distribution Formant Formant bandwidths bandwidths did not did display not display any systematical any systematical SNR = 1log appears, (1) appears at the higher at the frequency higher frequency than maximum than maximum of neutral of neutral changes changes in different in different scenarios. scenarios. Significant Significant formant formant shifts can shifts can j 1 ˆ σ 1 = n, j female female distribution distribution while female while female maximum maximum moves moves to to be observed be observed in the CLSD, in the Fig. CLSD, 6. Also Fig. significant 6. Also significant narrowing narrowing index of frames with speech activity since the 8 Close-talk Clean location location of typical of typical first formant first formant appearance appearance of certain of certain of first two of first formant two formant bandwidths bandwidths has been has observed. aluated only for short-time frames containing been observed. Hands-free Clean 6 =1 for each phonemes j. For each phonemes in short-time neutral in frame neutral speech. it is speech. During During the recognition, the recognition, f f Close-talk LE component component may be wrongly may be wrongly interpreted interpreted as F1. as F1. 3.4. Phoneme 3.4. Phoneme Hands-free and word LE and durations word durations aluate only the power of mixture σ xi, as the 4 osed to contain Average phoneme durations were evaluated for utterances 3.3. noise Formants 3.3. all Formants the time. Powers of Average phoneme durations were evaluated for utterances ise ( ˆ σ containing containing digits. Difference digits. Difference in phoneme in phoneme duration duration in the in the s, i and ˆn σ, i ) have to be estimated. Noise Formant Formant analysis analysis was performed was performed on utterances on utterances containing containing same word same uttered word in uttered two different in two different scenarios scenarios was evaluated -1 1 3 5 7 was evaluated ated in speech pause by standard exponential SNR digits. Monophone digits. Monophone HTK [11] HTK recognizer [11] recognizer trained trained on 7 on 7 as (db) shown as in shown Eq. 5. in Eq. 5. SPEECON SPEECON office sessions office sessions was used was for used the for forced Figure the : forced CLSD channel SNRs. ˆ σ alignment. 1 th TC T alignment. 1 th TC T ni, 1 + (1 p) σxi, for VADi C1 order =, () LPC C1 order was LPC chosen was for chosen formant formant tracking tracking = = 1 (%), 1 (%), (5) (5) Since CZKCC and CLSD were recorded by performed performed by the WaveSurfer. by the WaveSurfer. Information Information about first about four first four Ttwo ˆ σ ˆ C1 T ni, = σni, 1 VADi = 1. (3) C1 microphones, SPEECON SNR distributions are also and noise formant signals can formant frequencies be considered frequencies and to bandwidths be bandwidths depicted were assigned were only for assigned to the first to two T Cx channels. represents T Cx represents In average CZKCC average phoneme a phoneme duration duration in scenario in scenario x. x. speech power corresponding can be corresponding estimated phonemes. by subtraction phonemes. In the following In the following directional figures, figures, positions microphone positions was used in In the SPEECON, distant In SPEECON, position, phoneme phoneme duration duration differences differences did not did not r from the mixture of first power of two first female two female formants formants of the which of selected the explains selected vowels higher vowels average exceed SNR 38 exceed %. in In the 38 case %. distant In of case CZKCC, of CZKCC, greatest greatest duration duration changes changes Number of Utterances
Možnosti kompenzace LE Konverze hlasu - změna f o resp. dalších parametrů Speciální parametrizace - daty řízený návrh BF s optimalizovaným rozložením pásem Frekvenční transformace formantových kmitočtů - obdoba VTLN, počítáno na základě intezity LE
Děkuji vám za pozornost!