A Distributed Dictionary - Based Morphological Analysis Framework for Russian Language Processing

This article describes an approach to scaling service morphological parsing of words of natural language processing of various collections of documents in Russian. An overview and critical analysis of existing solutions. The requirements workbench vocabulary morphological analyzer were established. The distributed architecture of the web service morphological analysis, designed to a handle large collections of documents in Russian, presented the form of a structural model. This architecture is implemented as a prototype system in the programming language Ruby. The structure used in the morphological dictionary of a relational schema. Tests of this method in a distributed computing environment showed linear scalability of the proposed solutions. The configuration of the experiment involves the generation of the system load as a HTTP requests, system load balancing working nodes of a distributed system, application servers with a functioning database analyzer and morphological dictionary, as well as a caching node to reduce costs when you run queries to the dictionary. Applying this approach provides a linear increase in performance in distributed systems, automated processing of large volumes of text.

Keywords

distributed computing, natural language processing, corpus linguistics, data-intensive computing, morphological analysis.

References

1. Corpus Linguistics. Available at: http://www.corpling-ran.ru/index.html (accessed 20 May 2012).
2. GATE Cloud - a New Way to Mine the Web. Available at: http://gatecloud.net (accessed 20 May 2012).
3. Shestakov A.L., Sidorov A.I., Shaefer L.A., Gichkina E.V. The System of the Management Quality, Operational Control and Analysis of the Educational Process. Vestnik Leningradskogo gosudarstvennogo universiteta imeni A.S. Pushkina [Herald of the Leningrad State University], 2009, no. 1, pp. 177-194. (in Russian)
4. Mystem. Available at: http://company.yandex.ru/technologies/mystem (accessed 20 May 2012).
5. Snowball. Available at: http://snowball.tartarus.org (accessed 20 May 2012).
6. Stemka. Available at: http://www.keva.ru/stemka/stemka.html (accessed 20 May 2012).
7. Gearman. Available at: http://gearman.org (accessed 20 May 2012).
8. Erjavec T. MULTEXT-East Version 4: Multilingual Morphosyntactic Specifications, Lexicons and Corpora. Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC'10. Malta, 2010, pp. 2544-2547.
9. Myaso. Available at: http://myaso.eveel.ru (accessed 20 May 2012).
10. AOT :: Technologies. Available at: http://aot.ru/technology.html (accessed 20 May 2012).
11. Segalovich I. A Fast Morphological Algorithm with Unknown Word Guessing Induced by a Dictionary for a Web Search Engine. Proceedings of the International Conference on Machine Learning; Models, Technologies and Applications. MLMTA'03. Las Vegas, 2003, pp. 273-280.
12. HAProxy - The Reliable, High Performance TCP/HTTP Load Balancer. Available at: http://haproxy.1wt.eu (accessed 20 May 2012).
13. Blog Posts Collection. Available at: http://plove.eveel.ru (accessed 20 May 2012).
14. Tokyo Cabinet: a Modern Implementation of DBM. Available at: http://fallabs.com/tokyocabinet (accessed 20 May 2012).
15. Memcached - a Distributed Memory Object Caching System. Available at: http://memcached.org (accessed 20 May 2012).
16. Software. Russian National Corpus Language. Available at: http://www.ruscorpora.ru/corpora-progr.html (accessed 20 May 2012).