Метод стилевых кодов для многостилевого параметрического синтеза речи по тексту
Ключевые слова:
синтез речи по тексту, экспрессивный синтез речи, глубокие нейронные сети, стиль речи, стилевой код, прямой унитарный векторАннотация
Современные системы преобразования текста в речь обычно обеспечивают хорошую разборчивость. Одним из главных недостатков этих систем является отсутствие выразительности по сравнению с естественной человеческой речью. Очень неприятно, когда автоматическая система передает утвердительные и отрицательные предложения совершенно одинаково. Введение параметрических методов в синтезе речи дало возможность легко изменять характеристики говорящего и стили речи. В этой статье представлен простой способ включения стилей в синтезированную речь, используя стилевые коды. Предлагаемый метод требует всего лишь пару минут заданного стиля, чтобы смоделировать нейтральную речь. Он успешно применяется как в скрытых марковских моделях, так и в синтезе на основе глубоких нейронных сетей, предоставляя стилевой код как дополнительный вклад в модель. Аудирование подтвердило, что наибольшая выразительность достигается за счет синтеза глубоких нейронных сетей по сравнению с синтезом скрытых марковских моделей. Также доказано, что качество речи, синтезированное глубокими нейронными сетями в определенном стиле, сопоставимо с речью, синтезированной в нейтральном стиле, хотя база данных нейтральной речи примерно в 10 раз больше. Глубокие нейронные сети на основе синтеза речи по тексту со стилевыми кодами изучаются путем сравнения качества речи, создаваемой системами одностилевого моделирования и многостилевого моделирования. Объективные и субъективные измерения подтвердили, что между этими двумя подходами нет существенной разницы.Литература
1. Hunt A.J., Black A.W. Unit selection in a concatenative speech synthesis system using a large speech database // Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP-96). 1996. vol. 1. pp. 373–376.
2. Tokuda K. et al. Speech synthesis based on hidden Markov models // Proceedings of the IEEE. 2013. vol. 101. no. 5. pp. 1234–1252.
3. Watts O. et al. From HMMs to DNNs: where do the improvements come from? // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2016. pp. 5505–5509.
4. Ling Z.H. et al. Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends // IEEE Signal Processing Magazine. 2015. vol. 32. no. 3. pp. 35–52.
5. Yu D., Deng L. Deep learning: methods and applications // Foundations and Trends® in Signal Processing. 2014. vol. 7. no. 3-4. pp. 198–387.
6. Qian Y., Fan Y., Hu W., Soong F.K. On the Training Aspects of Deep Neural Network (DNN) for Parametric TTS Synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2014. pp. 3829–3833.
7. Delić T., Sečujski M. Sinteza govora na srpskom jeziku zasnovana na veštačkim neuralnim mrežama // Telecommunication forum (TELFOR 2016). 2016. pp. 403–406.
8. Solomennik A.I., Chistikov P.G. Evaluation of naturalness of synthesized speech with different prosodic models // Proceedeings International conference on Computational Linguistics and Intellectual Technologies “Dialogue 2013”. 2013. 7 p.
9. Abe M. Speaking Styles: Statistical Analysis and Synthesis by a Text-to-Speech System // Progress in speech synthesis. 1997. pp. 495–510.
10. Rusko M. et al. Expressive Speech Synthesis for Critical Situations // Computing and Informatics. 2015. vol. 33. no. 6. pp. 1312–1332.
11. Delić Т. et al. Multi- style Statistical Parametric TTS // Proceedings Digital speech and image processing (DOGS 2017). 2017. pp. 5–8.
12. Wu Z., Valentini-Botinhao C., Watts O., King S. Deep Neural Networks employing multi-task learning and stacked bottleneck features for speech synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2015. pp. 4460–4464.
13. Watts O. et al. From HMMs to DNNs: Where do the improvements come from? // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2016. pp. 5505–5509.
14. Delić T., Sečujski M., Suzić S. A review of Serbian parametric speech synthesis based on deep neural networks // Telfor Journal. 2017. vol. 9. no. 1. pp. 32–37.
15. Zen H., Tokuda K., Black A.W. Statistical parametric speech synthesis // Speech Communication. 2009. vol. 51. no. 11. pp. 1039–1064.
16. Zen H. et al. A hidden semi-Markov model-based speech synthesis system // IEICE transactions on information and systems. 2007. vol. 90. no. 5. pp. 825–834.
17. Yoshimura T. et al. Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis // Sixth European Conference on Speech Communication and Technology. 1999. 4 p.
18. Toda T., Tokuda K. A speech parameter generation algorithm considering global variance for HMM-based speech synthesis // IEICE transactions on information and systems. 2007. vol. E90-D. no. 5. pp. 816–824.
19. Tokuda K. et al. Speech Parameter Generation Algorithms for HMM-based Speech Synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2000. pp. 1315–1318.
20. Rabiner L.R. A tutorial on hidden Markov models and selectedapplications in speech recognition // Proceedings of the IEEE. 1989. vol. 77. no. 2. pp. 257–286.
21. Schröder M. Expressive speech synthesis: Past, present, and possible futures // Affective information processing. 2009. pp. 111–126.
22. Tao J., Kang Y., Li A. Prosody conversion from neutral speech to emotional speech // IEEE Transactions on Audio, Speech, and Language Processing. 2006. vol. 14. no. 4. pp. 1145–1153.
23. Iida A., Campbell N., Higuchi F., Yasumura M. A corpus-based speech synthesis system with emotion // Speech Communication. 2003. vol. 40. no. 1-2. pp. 161–187.
24. Yamagishi J., Onishi K., Masuko T., Kobayashi T. Modeling of various speaking styles and emotions for HMM-based speech synthesis // Eighth European Conference on Speech Communication and Technology. 2003. pp. 2461–2464.
25. Yamagishi J. et al. Model adaptation approach to speech synthesis with diverse voices and styles // International Conference on Acoustics, Speech and Signal Processing (ICASSP 2007). 2007. vol. 4. p. IV-1233–IV-1236.
26. Barra-Chicote R. et al. Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech // Speech Communication. 2010. vol. 52. no. 5. pp. 394–404.
27. Inoue K. et al. An investigation to transplant emotional expressions in DNN-based TTS synthesis // Proc. APSIPA Annual Summit and Conference. 2017. pp. 1253–1258.
28. An S., Ling Z., Dai L. Emotional statistical parametric speech synthesis using LSTM-RNNs // Proc. APSIPA Annual Summit and Conference. 2017. pp. 1613–1616.
29. Lorenzo-Trueba J. et al. Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis // Speech Communication. 2018. vol. 99. pp. 135–143.
30. Stylianou Y., Cappe O., Moulines E. Statistical Methods for Voice Quality Transformation // Fourth European Conference on Speech Communication and Technology. 1995. pp. 447–450.
31. Hojo N., Ijima Y., Mizuno H. An Investigation of DNN-Based Speech Synthesis Using Speaker Codes // INTERSPEECH. 2016. pp. 2278–2282.
32. Fan Y., Qian Y., Soong F.K., He L. Multi-speaker modeling and speaker adaptation for DNN-based TTS synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2015. pp. 4475–4479.
33. Luong H.T., Takaki S., Henter G.E., Yamagishi J. Adapting and controlling DNN-based speech synthesis using input codes // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017. pp. 4905–4909.
34. Yang S., Wu Z., Xie L. On the Training of DNN-based Average Voice Model for Speech Synthesis // Proc. Signal and Information Processing Association Annual Summit and Conference (APSIPA). 2016. pp. 1–6.
35. Wu Z., Watts O., King S. Merlin: An Open Source Neural Network Speech Synthesis System // Proc. 9th ISCA Speech Synthesis Workshop (SSW9). 2016. pp. 218–223.
36. Fan Y., Qian Y., Xie F.L., Soong F.K. TTS synthesis with bidirectional LSTM based recurrent neural networks // Fifteenth Annual Conference of the International Speech Communication Association INTERSPEECH. 2014. pp. 1964–1968.
37. Silverman K. et al. ToBI: A standard for labeling English prosody // Proceedings of International Conference on Spoken Language Processing (ICSLP). 1992. pp. 867–870.
38. Morise M., Yokomori F., Ozawa K. WORLD: a vocoder-based high-quality speech synthesis system for real-time applications // IEICE transactions on information and systems. 2016. vol. E99-D. no. 7. pp. 1877–1884.
39. Zen H. et al. The HMM-based speech synthesis system (HTS) version 2.0 // Proceedings of Sixth ISCA Workshop on Speech Synthesis. 2007. pp. 294–299.
2. Tokuda K. et al. Speech synthesis based on hidden Markov models // Proceedings of the IEEE. 2013. vol. 101. no. 5. pp. 1234–1252.
3. Watts O. et al. From HMMs to DNNs: where do the improvements come from? // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2016. pp. 5505–5509.
4. Ling Z.H. et al. Deep learning for acoustic modeling in parametric speech generation: A systematic review of existing techniques and future trends // IEEE Signal Processing Magazine. 2015. vol. 32. no. 3. pp. 35–52.
5. Yu D., Deng L. Deep learning: methods and applications // Foundations and Trends® in Signal Processing. 2014. vol. 7. no. 3-4. pp. 198–387.
6. Qian Y., Fan Y., Hu W., Soong F.K. On the Training Aspects of Deep Neural Network (DNN) for Parametric TTS Synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2014. pp. 3829–3833.
7. Delić T., Sečujski M. Sinteza govora na srpskom jeziku zasnovana na veštačkim neuralnim mrežama // Telecommunication forum (TELFOR 2016). 2016. pp. 403–406.
8. Solomennik A.I., Chistikov P.G. Evaluation of naturalness of synthesized speech with different prosodic models // Proceedeings International conference on Computational Linguistics and Intellectual Technologies “Dialogue 2013”. 2013. 7 p.
9. Abe M. Speaking Styles: Statistical Analysis and Synthesis by a Text-to-Speech System // Progress in speech synthesis. 1997. pp. 495–510.
10. Rusko M. et al. Expressive Speech Synthesis for Critical Situations // Computing and Informatics. 2015. vol. 33. no. 6. pp. 1312–1332.
11. Delić Т. et al. Multi- style Statistical Parametric TTS // Proceedings Digital speech and image processing (DOGS 2017). 2017. pp. 5–8.
12. Wu Z., Valentini-Botinhao C., Watts O., King S. Deep Neural Networks employing multi-task learning and stacked bottleneck features for speech synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2015. pp. 4460–4464.
13. Watts O. et al. From HMMs to DNNs: Where do the improvements come from? // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2016. pp. 5505–5509.
14. Delić T., Sečujski M., Suzić S. A review of Serbian parametric speech synthesis based on deep neural networks // Telfor Journal. 2017. vol. 9. no. 1. pp. 32–37.
15. Zen H., Tokuda K., Black A.W. Statistical parametric speech synthesis // Speech Communication. 2009. vol. 51. no. 11. pp. 1039–1064.
16. Zen H. et al. A hidden semi-Markov model-based speech synthesis system // IEICE transactions on information and systems. 2007. vol. 90. no. 5. pp. 825–834.
17. Yoshimura T. et al. Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis // Sixth European Conference on Speech Communication and Technology. 1999. 4 p.
18. Toda T., Tokuda K. A speech parameter generation algorithm considering global variance for HMM-based speech synthesis // IEICE transactions on information and systems. 2007. vol. E90-D. no. 5. pp. 816–824.
19. Tokuda K. et al. Speech Parameter Generation Algorithms for HMM-based Speech Synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2000. pp. 1315–1318.
20. Rabiner L.R. A tutorial on hidden Markov models and selectedapplications in speech recognition // Proceedings of the IEEE. 1989. vol. 77. no. 2. pp. 257–286.
21. Schröder M. Expressive speech synthesis: Past, present, and possible futures // Affective information processing. 2009. pp. 111–126.
22. Tao J., Kang Y., Li A. Prosody conversion from neutral speech to emotional speech // IEEE Transactions on Audio, Speech, and Language Processing. 2006. vol. 14. no. 4. pp. 1145–1153.
23. Iida A., Campbell N., Higuchi F., Yasumura M. A corpus-based speech synthesis system with emotion // Speech Communication. 2003. vol. 40. no. 1-2. pp. 161–187.
24. Yamagishi J., Onishi K., Masuko T., Kobayashi T. Modeling of various speaking styles and emotions for HMM-based speech synthesis // Eighth European Conference on Speech Communication and Technology. 2003. pp. 2461–2464.
25. Yamagishi J. et al. Model adaptation approach to speech synthesis with diverse voices and styles // International Conference on Acoustics, Speech and Signal Processing (ICASSP 2007). 2007. vol. 4. p. IV-1233–IV-1236.
26. Barra-Chicote R. et al. Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech // Speech Communication. 2010. vol. 52. no. 5. pp. 394–404.
27. Inoue K. et al. An investigation to transplant emotional expressions in DNN-based TTS synthesis // Proc. APSIPA Annual Summit and Conference. 2017. pp. 1253–1258.
28. An S., Ling Z., Dai L. Emotional statistical parametric speech synthesis using LSTM-RNNs // Proc. APSIPA Annual Summit and Conference. 2017. pp. 1613–1616.
29. Lorenzo-Trueba J. et al. Investigating different representations for modeling and controlling multiple emotions in DNN-based speech synthesis // Speech Communication. 2018. vol. 99. pp. 135–143.
30. Stylianou Y., Cappe O., Moulines E. Statistical Methods for Voice Quality Transformation // Fourth European Conference on Speech Communication and Technology. 1995. pp. 447–450.
31. Hojo N., Ijima Y., Mizuno H. An Investigation of DNN-Based Speech Synthesis Using Speaker Codes // INTERSPEECH. 2016. pp. 2278–2282.
32. Fan Y., Qian Y., Soong F.K., He L. Multi-speaker modeling and speaker adaptation for DNN-based TTS synthesis // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2015. pp. 4475–4479.
33. Luong H.T., Takaki S., Henter G.E., Yamagishi J. Adapting and controlling DNN-based speech synthesis using input codes // International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2017. pp. 4905–4909.
34. Yang S., Wu Z., Xie L. On the Training of DNN-based Average Voice Model for Speech Synthesis // Proc. Signal and Information Processing Association Annual Summit and Conference (APSIPA). 2016. pp. 1–6.
35. Wu Z., Watts O., King S. Merlin: An Open Source Neural Network Speech Synthesis System // Proc. 9th ISCA Speech Synthesis Workshop (SSW9). 2016. pp. 218–223.
36. Fan Y., Qian Y., Xie F.L., Soong F.K. TTS synthesis with bidirectional LSTM based recurrent neural networks // Fifteenth Annual Conference of the International Speech Communication Association INTERSPEECH. 2014. pp. 1964–1968.
37. Silverman K. et al. ToBI: A standard for labeling English prosody // Proceedings of International Conference on Spoken Language Processing (ICSLP). 1992. pp. 867–870.
38. Morise M., Yokomori F., Ozawa K. WORLD: a vocoder-based high-quality speech synthesis system for real-time applications // IEICE transactions on information and systems. 2016. vol. E99-D. no. 7. pp. 1877–1884.
39. Zen H. et al. The HMM-based speech synthesis system (HTS) version 2.0 // Proceedings of Sixth ISCA Workshop on Speech Synthesis. 2007. pp. 294–299.
Style-Code Method for Multi-Style Parametric Text-to-Speech Synthesis
Abstract
Modern text-to-speech systems generally achieve good intelligibility. The one of the main drawbacks of these systems is the lack of expressiveness in comparison to natural human speech. It is very unpleasant when automated system conveys positive and negative message in completely the same way. The introduction of parametric methods in speech synthesis gave possibility to easily change speaker characteristics and speaking styles. In this paper a simple method for incorporating styles into synthesized speech by using style codes is presented. The proposed method requires just a couple of minutes of target style and moderate amount of neutral speech. It is successfully applied to both hidden Markov models and deep neural networks-based synthesis, giving style code as additional input to the model. Listening tests confirmed that better style expressiveness is achieved by deep neural networks synthesis compared to hidden Markov model synthesis. It is also proved that quality of speech synthesized by deep neural networks in a certain style is comparable with the speech synthesized in neutral style, although the neutral-speech-database is about 10 times bigger. DNN based TTS with style codes are further investigated by comparing the quality of speech produced by single-style modeling and multi-style modeling systems. Objective and subjective measures confirmed that there is no significant difference between these two approaches.
Опубликован
2018-10-01
Как цитировать
Сузич, С., Делич, Т. В., Острогонац, С., Джурич, С., & Пекар, Д. Й. (2018). Метод стилевых кодов для многостилевого параметрического синтеза речи по тексту. Труды СПИИРАН, 5(60), 216-240. https://doi.org/10.15622/sp.60.8
Раздел
Искусственный интеллект, инженерия данных и знаний
Авторы, которые публикуются в данном журнале, соглашаются со следующими условиями:
Авторы сохраняют за собой авторские права на работу и передают журналу право первой публикации вместе с работой, одновременно лицензируя ее на условиях Creative Commons Attribution License, которая позволяет другим распространять данную работу с обязательным указанием авторства данной работы и ссылкой на оригинальную публикацию в этом журнале.
Авторы сохраняют право заключать отдельные, дополнительные контрактные соглашения на неэксклюзивное распространение версии работы, опубликованной этим журналом (например, разместить ее в университетском хранилище или опубликовать ее в книге), со ссылкой на оригинальную публикацию в этом журнале.
Авторам разрешается размещать их работу в сети Интернет (например, в университетском хранилище или на их персональном веб-сайте) до и во время процесса рассмотрения ее данным журналом, так как это может привести к продуктивному обсуждению, а также к большему количеству ссылок на данную опубликованную работу (Смотри The Effect of Open Access).
