Technical Report Series on Corpus Building

Transkript

1 Technical Report Series on Corpus Building Vol. 8 (June 2013) Czech Corpora Uwe Quasthoff Dirk Goldhahn Abteilung Automatische Sprachverarbeitung, Institut für Informatik, Universität Leipzig

2 Affiliation of the authors: Uwe Quasthoff, Dirk Goldhahn: Institut für Informatik,Universität Leipzig {quasthoff, Copyright: Abteilung Automatische Sprachverarbeitung, Institut für Informatik, Universität Leipzig, Technical Report Series on Corpus Building Vol. 1: Deutscher Wortschatz 2013 Vol. 2: Danish Corpora Vol. 3: Dutch Corpora Vol. 4: Icelandic Corpora Vol. 5: Hungarian Corpora Vol. 6: Ukrainian Corpora Vol. 7: Indonesian Corpora Vol. 8: Czech Corpora This PDF document was created using the open source tool mwlib. For more infotmation, see PDF generated at: 25. June 2013

3 Czech corpora 1 Introduction to corpus creation 1 CES - a processing related language description 2 CES corpora 3 CES corpus comparison 8 Processing details 9 Appendix to ces news : Database summary 9 Appendix to ces news 2008: Database summary 9 Appendix to ces news 2009: Database summary 10 Appendix to ces news 2010: Database summary 10 Appendix to ces news 2011: Database summary 11 Appendix to ces news 2012: Database summary 11 Appendix to ces newscrawl 2011: Database summary 12 Appendix to ces newscrawl 2012: Database summary 12 Appendix to ces wikipedia 2007: Database summary 13 Appendix to ces wikipedia 2012: Database summary 13 Appendix to ces web 2002: Database summary 14 Appendix to ces web 2011: Database summary 14 Appendix to ces web 2012: Database summary 15 Appendix to ces mixed 2012: Database summary 15 Content details 16 Appendix to ces news : Size of different TLDs 16 Appendix to ces news 2008: Size of different TLDs 16 Appendix to ces news 2009: Size of different TLDs 17 Appendix to ces news 2010: Size of different TLDs 17 Appendix to ces news 2011: Size of different TLDs 17 Appendix to ces news 2012: Size of different TLDs 18 Appendix to ces newscrawl 2011: Size of different TLDs 18 Appendix to ces newscrawl 2012: Size of different TLDs 18 Appendix to ces web 2002: Size of different TLDs 19 Appendix to ces web 2011: Size of different TLDs 19

4 Appendix to ces web 2012: Size of different TLDs 19 Appendix to ces mixed 2012: Size of different TLDs 20 Appendix to ces news : Size of largest domains 20 Appendix to ces news 2008: Size of largest domains 21 Appendix to ces news 2009: Size of largest domains 21 Appendix to ces news 2010: Size of largest domains 22 Appendix to ces news 2011: Size of largest domains 23 Appendix to ces news 2012: Size of largest domains 23 Appendix to ces newscrawl 2011: Size of largest domains 24 Appendix to ces newscrawl 2012: Size of largest domains 25 Appendix to ces web 2002: Size of largest domains 25 Appendix to ces web 2011: Size of largest domains 26 Appendix to ces web 2012: Size of largest domains 27 Appendix to ces mixed 2012: Size of largest domains 27 Appendix to ces news : Number of sources by time period 28 Appendix to ces news 2008: Number of sources by time period 29 Appendix to ces news 2009: Number of sources by time period 31 Appendix to ces news 2010: Number of sources by time period 32 Appendix to ces news 2011: Number of sources by time period 33 Appendix to ces news 2012: Number of sources by time period 35 Word details 37 Appendix to ces news : Words by length without multiplicity 37 Appendix to ces news 2008: Words by length without multiplicity 39 Appendix to ces news 2009: Words by length without multiplicity 41 Appendix to ces news 2010: Words by length without multiplicity 43 Appendix to ces news 2011: Words by length without multiplicity 45 Appendix to ces news 2012: Words by length without multiplicity 47 Appendix to ces newscrawl 2011: Words by length without multiplicity 49 Appendix to ces newscrawl 2012: Words by length without multiplicity 51 Appendix to ces wikipedia 2007: Words by length without multiplicity 53 Appendix to ces wikipedia 2012: Words by length without multiplicity 55 Appendix to ces web 2002: Words by length without multiplicity 57 Appendix to ces web 2011: Words by length without multiplicity 59 Appendix to ces web 2012: Words by length without multiplicity 61 Appendix to ces mixed 2012: Words by length without multiplicity 63 Appendix to ces news : Words by length with multiplicity 65 Appendix to ces news 2008: Words by length with multiplicity 67

5 Appendix to ces news 2009: Words by length with multiplicity 69 Appendix to ces news 2010: Words by length with multiplicity 71 Appendix to ces news 2011: Words by length with multiplicity 73 Appendix to ces news 2012: Words by length with multiplicity 75 Appendix to ces newscrawl 2011: Words by length with multiplicity 77 Appendix to ces newscrawl 2012: Words by length with multiplicity 79 Appendix to ces wikipedia 2007: Words by length with multiplicity 81 Appendix to ces wikipedia 2012: Words by length with multiplicity 83 Appendix to ces web 2002: Words by length with multiplicity 85 Appendix to ces web 2011: Words by length with multiplicity 87 Appendix to ces web 2012: Words by length with multiplicity 89 Appendix to ces mixed 2012: Words by length with multiplicity 91 Appendix to ces news : The most frequent 50 words 92 Appendix to ces news 2008: The most frequent 50 words 93 Appendix to ces news 2009: The most frequent 50 words 94 Appendix to ces news 2010: The most frequent 50 words 95 Appendix to ces news 2011: The most frequent 50 words 96 Appendix to ces news 2012: The most frequent 50 words 97 Appendix to ces newscrawl 2011: The most frequent 50 words 98 Appendix to ces newscrawl 2012: The most frequent 50 words 99 Appendix to ces wikipedia 2007: The most frequent 50 words 100 Appendix to ces wikipedia 2012: The most frequent 50 words 101 Appendix to ces web 2002: The most frequent 50 words 102 Appendix to ces web 2011: The most frequent 50 words 103 Appendix to ces web 2012: The most frequent 50 words 104 Appendix to ces mixed 2012: The most frequent 50 words 105 Appendix to ces news : Longest words in top by rank 106 Appendix to ces news 2008: Longest words in top by rank 107 Appendix to ces news 2009: Longest words in top by rank 108 Appendix to ces news 2010: Longest words in top by rank 109 Appendix to ces news 2011: Longest words in top by rank 110 Appendix to ces news 2012: Longest words in top by rank 111 Appendix to ces newscrawl 2011: Longest words in top by rank 112 Appendix to ces newscrawl 2012: Longest words in top by rank 113 Appendix to ces wikipedia 2007: Longest words in top by rank 114 Appendix to ces wikipedia 2012: Longest words in top by rank 115 Appendix to ces web 2002: Longest words in top by rank 116 Appendix to ces web 2011: Longest words in top by rank 117

6 Appendix to ces web 2012: Longest words in top by rank 118 Appendix to ces mixed 2012: Longest words in top by rank 119 Character N-gram details 120 Appendix to ces news : Alphabet as used in the top words 120 Appendix to ces news 2008: Alphabet as used in the top words 121 Appendix to ces news 2009: Alphabet as used in the top words 123 Appendix to ces news 2010: Alphabet as used in the top words 124 Appendix to ces news 2011: Alphabet as used in the top words 126 Appendix to ces news 2012: Alphabet as used in the top words 127 Appendix to ces newscrawl 2011: Alphabet as used in the top words 129 Appendix to ces newscrawl 2012: Alphabet as used in the top words 130 Appendix to ces wikipedia 2007: Alphabet as used in the top words 132 Appendix to ces wikipedia 2012: Alphabet as used in the top words 133 Appendix to ces web 2002: Alphabet as used in the top words 135 Appendix to ces web 2011: Alphabet as used in the top words 136 Appendix to ces web 2012: Alphabet as used in the top words 138 Appendix to ces mixed 2012: Alphabet as used in the top words 139 Abbreviation details 141 Appendix to ces news : Most frequent abbreviations 141 Appendix to ces news 2008: Most frequent abbreviations 142 Appendix to ces news 2009: Most frequent abbreviations 143 Appendix to ces news 2010: Most frequent abbreviations 144 Appendix to ces news 2011: Most frequent abbreviations 145 Appendix to ces news 2012: Most frequent abbreviations 146 Appendix to ces newscrawl 2011: Most frequent abbreviations 146 Appendix to ces newscrawl 2012: Most frequent abbreviations 147 Appendix to ces wikipedia 2007: Most frequent abbreviations 147 Appendix to ces wikipedia 2012: Most frequent abbreviations 148 Appendix to ces web 2002: Most frequent abbreviations 149 Appendix to ces web 2011: Most frequent abbreviations 150 Appendix to ces web 2012: Most frequent abbreviations 151 Appendix to ces mixed 2012: Most frequent abbreviations 152 Appendix to ces news : Left neighbors of the full stop 153 Appendix to ces news 2008: Left neighbors of the full stop 154 Appendix to ces news 2009: Left neighbors of the full stop 155 Appendix to ces news 2010: Left neighbors of the full stop 156

7 Appendix to ces news 2011: Left neighbors of the full stop 157 Appendix to ces news 2012: Left neighbors of the full stop 158 Appendix to ces newscrawl 2011: Left neighbors of the full stop 159 Appendix to ces newscrawl 2012: Left neighbors of the full stop 160 Appendix to ces wikipedia 2007: Left neighbors of the full stop 161 Appendix to ces wikipedia 2012: Left neighbors of the full stop 162 Appendix to ces web 2002: Left neighbors of the full stop 163 Appendix to ces web 2011: Left neighbors of the full stop 164 Appendix to ces web 2012: Left neighbors of the full stop 165 Appendix to ces mixed 2012: Left neighbors of the full stop 166 Appendix to ces news : Left neighbors of the full stop with additional internal full stops 167 Appendix to ces news 2008: Left neighbors of the full stop with additional internal full stops 168 Appendix to ces news 2009: Left neighbors of the full stop with additional internal full stops 169 Appendix to ces news 2010: Left neighbors of the full stop with additional internal full stops 170 Appendix to ces news 2011: Left neighbors of the full stop with additional internal full stops 171 Appendix to ces news 2012: Left neighbors of the full stop with additional internal full stops 172 Appendix to ces newscrawl 2011: Left neighbors of the full stop with additional internal full stops 173 Appendix to ces newscrawl 2012: Left neighbors of the full stop with additional internal full stops 174 Appendix to ces wikipedia 2007: Left neighbors of the full stop with additional internal full stops 175 Appendix to ces wikipedia 2012: Left neighbors of the full stop with additional internal full stops 176 Appendix to ces web 2002: Left neighbors of the full stop with additional internal full stops 177 Appendix to ces web 2011: Left neighbors of the full stop with additional internal full stops 178 Appendix to ces web 2012: Left neighbors of the full stop with additional internal full stops 179 Appendix to ces mixed 2012: Left neighbors of the full stop with additional internal full stops 180 Sentences details 181 Appendix to ces news : Shortest sentences 181 Appendix to ces news 2008: Shortest sentences 182 Appendix to ces news 2009: Shortest sentences 184 Appendix to ces news 2010: Shortest sentences 185 Appendix to ces news 2011: Shortest sentences 187 Appendix to ces news 2012: Shortest sentences 188 Appendix to ces newscrawl 2011: Shortest sentences 190 Appendix to ces newscrawl 2012: Shortest sentences 191 Appendix to ces wikipedia 2007: Shortest sentences 193 Appendix to ces wikipedia 2012: Shortest sentences 194 Appendix to ces web 2002: Shortest sentences 196 Appendix to ces web 2011: Shortest sentences 197

8 Appendix to ces web 2012: Shortest sentences 199 Appendix to ces mixed 2012: Shortest sentences 200 Appendix to ces news : Longest sentences 202 Appendix to ces news 2008: Longest sentences 204 Appendix to ces news 2009: Longest sentences 206 Appendix to ces news 2010: Longest sentences 208 Appendix to ces news 2011: Longest sentences 210 Appendix to ces news 2012: Longest sentences 212 Appendix to ces newscrawl 2011: Longest sentences 214 Appendix to ces newscrawl 2012: Longest sentences 216 Appendix to ces wikipedia 2007: Longest sentences 218 Appendix to ces wikipedia 2012: Longest sentences 220 Appendix to ces web 2002: Longest sentences 222 Appendix to ces web 2011: Longest sentences 224 Appendix to ces web 2012: Longest sentences 226 Appendix to ces mixed 2012: Longest sentences 228 Appendix to ces news : Length of sentences in characters 230 Appendix to ces news 2008: Length of sentences in characters 231 Appendix to ces news 2009: Length of sentences in characters 232 Appendix to ces news 2010: Length of sentences in characters 233 Appendix to ces news 2011: Length of sentences in characters 234 Appendix to ces news 2012: Length of sentences in characters 235 Appendix to ces newscrawl 2011: Length of sentences in characters 236 Appendix to ces newscrawl 2012: Length of sentences in characters 237 Appendix to ces wikipedia 2007: Length of sentences in characters 238 Appendix to ces wikipedia 2012: Length of sentences in characters 239 Appendix to ces web 2002: Length of sentences in characters 240 Appendix to ces web 2011: Length of sentences in characters 241 Appendix to ces web 2012: Length of sentences in characters 242 Appendix to ces mixed 2012: Length of sentences in characters 243 Appendix to ces news : Length of sentences in words 244 Appendix to ces news 2008: Length of sentences in words 245 Appendix to ces news 2009: Length of sentences in words 246 Appendix to ces news 2010: Length of sentences in words 247 Appendix to ces news 2011: Length of sentences in words 248 Appendix to ces news 2012: Length of sentences in words 249 Appendix to ces newscrawl 2011: Length of sentences in words 250 Appendix to ces newscrawl 2012: Length of sentences in words 251

9 Appendix to ces wikipedia 2007: Length of sentences in words 252 Appendix to ces wikipedia 2012: Length of sentences in words 253 Appendix to ces web 2002: Length of sentences in words 254 Appendix to ces web 2011: Length of sentences in words 255 Appendix to ces web 2012: Length of sentences in words 256 Appendix to ces mixed 2012: Length of sentences in words 257 Oddities details 258 Appendix to ces news : Longest words 258 Appendix to ces news 2008: Longest words 258 Appendix to ces news 2009: Longest words 259 Appendix to ces news 2010: Longest words 259 Appendix to ces news 2011: Longest words 260 Appendix to ces news 2012: Longest words 260 Appendix to ces newscrawl 2011: Longest words 261 Appendix to ces newscrawl 2012: Longest words 261 Appendix to ces wikipedia 2007: Longest words 262 Appendix to ces wikipedia 2012: Longest words 262 Appendix to ces web 2002: Longest words 263 Appendix to ces web 2011: Longest words 263 Appendix to ces web 2012: Longest words 264 Appendix to ces mixed 2012: Longest words 264 Appendix to ces news : Sentences with high average word length 265 Appendix to ces news 2008: Sentences with high average word length 266 Appendix to ces news 2009: Sentences with high average word length 267 Appendix to ces news 2010: Sentences with high average word length 268 Appendix to ces news 2011: Sentences with high average word length 269 Appendix to ces news 2012: Sentences with high average word length 270 Appendix to ces newscrawl 2011: Sentences with high average word length 271 Appendix to ces newscrawl 2012: Sentences with high average word length 272 Appendix to ces wikipedia 2007: Sentences with high average word length 273 Appendix to ces wikipedia 2012: Sentences with high average word length 274 Appendix to ces web 2002: Sentences with high average word length 275 Appendix to ces web 2011: Sentences with high average word length 276 Appendix to ces web 2012: Sentences with high average word length 277 Appendix to ces mixed 2012: Sentences with high average word length 278 Appendix to ces news : Problems with sentence segmentation - words ending in a stopword 279

10 Appendix to ces news 2008: Problems with sentence segmentation - words ending in a stopword 279 Appendix to ces news 2009: Problems with sentence segmentation - words ending in a stopword 279 Appendix to ces news 2010: Problems with sentence segmentation - words ending in a stopword 280 Appendix to ces news 2011: Problems with sentence segmentation - words ending in a stopword 281 Appendix to ces news 2012: Problems with sentence segmentation - words ending in a stopword 281 Appendix to ces newscrawl 2011: Problems with sentence segmentation - words ending in a stopword 282 Appendix to ces newscrawl 2012: Problems with sentence segmentation - words ending in a stopword 283 Appendix to ces wikipedia 2007: Problems with sentence segmentation - words ending in a stopword 284 Appendix to ces wikipedia 2012: Problems with sentence segmentation - words ending in a stopword 284 Appendix to ces web 2002: Problems with sentence segmentation - words ending in a stopword 285 Appendix to ces web 2011: Problems with sentence segmentation - words ending in a stopword 286 Appendix to ces web 2012: Problems with sentence segmentation - words ending in a stopword 287 Appendix to ces mixed 2012: Problems with sentence segmentation - words ending in a stopword 288

11 1 Czech corpora Introduction to corpus creation The Leipzig Corpora Collection (LCC) collects Web based corpora for many different languages. The main text genres are newspaper texts, Wikipedias and randomly collected web pages. All corpora are processed in the same way: Crawling Web pages HTML stripping Language identifikation Sentence segmentation Cleaning: Removal of ill-formed sentences Duplicate removal Calculation of word frequences and word co-occurrences As result we have a corpus containing only well-formed sentences in the language under consideration. The sentences are in random order; hence, sharing the corpus does not violate copyright law because it is impossible to reconstruct the original texts. The pre-processing steps contain both language independent steps (like HTML stripping and duplicate removal) and language dependent steps (like language identification and sentence segmentation). Especially the language specific parts are vulnerable to specific processing problems. The aim of the paper is to identify possible problems and evaluate the results. The following problems are adressed: A processing-focused language description Language size: How much text is available for this language? What are the biggest sources? Corpus description: Genre, size, crawling and processing date. Possible problems in language identification: Which languages are similar? Character set and alphabet Inspecting the word list: Most frequent words, longer high frequent words and longest words at all. Word length distribution. Can abbreviations confuse sentence segmentation? Information about the abbreviation list. Inspecting sentences: Inspect shortest and longest sentences to identify possible segmentation problems. Sentence length distribution. The paper describes the result of these inspections; the appendices show the exact results for the different corpora. This helps to compare the corpora with respect to quality. In the section quality overview, an overall quality description for each corpus is given. All corpora contain only minor problems which are irrelevant for most applications. Otherwise the corpus creation has been iterated.

12 CES - a processing related language description 2 CES - a processing related language description General properties of the Czech language Native Name: Čeština Classifiation: Indo-European, Slavic, West, Czech-Slovak Total Number of Speakers: 12M Largest countries with number of speakers: Czech Republic (10M) Sources: / www. ethnologue. com, Wikipedia Processing summary Latin alphabet with some additional characters full stop is used as sentence boundary and for abbreviations apostrostophes used rarely Properties important for processing Alphabet and punctuation The alphabet is Latin based, with the following specialities (sources: / en. wikipedia. org/ wiki/ Alphabets_derived_from_the_Latin and / de. wikipedia. org/ wiki/ Tschechische_Sprache#Alphabet): Czech includes all 26 base letters and Á, Č, Ď, É, Ě, Í, Ň, Ó, Ř, Š, Ť, Ú, Ů, Ý, Ž In foreign words some more Diphtongs: au, eu, ou Usual Latin punctuation Usage of uppercase letters: At sentence beginnings and for proper names (of persons, organisations, countries etc.). Sentence segmentation and word tokenization Sentence beginnings Sentences begin with a capitalized first word. Abbreviations Abbreviations confusing with sentence boundaries: Special abbreviation list has to be inspected. Sources for abbreviations: ### Abbreviations with full stop may appear in the word list without full stop. Apostrophes Use of apostrophes: very infrequent (???)

13 CES - a processing related language description 3 Sources and ranking (2012) Estimated number of webpages containing text Google.com top-5 words: results for "a" "se" "na" "v" "je" Google.com top-10 words: results for "a" "se" "na" "v" "je" "že" "to" "z" "s" "o" Rank according to number of speakers (Ethnologue): 75 Rank according to Wikipedia size (see / de. wikipedia. org/ wiki/ Wikipedia:Sprachen, 01/2013): Rank 18 with articles. Rank according to number of newspapers as found by AbyZ (5/2012): 134 newspapers, rank 18. Rank according to number of newspapers with RSS feeds (5/2012): 131 newspapers, rank 12. Rank according to our corpus size (9/2012): 17 CES corpora Quality Overview Quality Ratings A: Very good quality. Ready to use (or already used) for frequency dictionary. Size as large as possible Only minimal errors Multiple genres (if possible) A-: Small problems identified. They should not affect usage. B: Native speaker quality. Information about abbreviations and sentence boundaries by native speaker Resulting statistics checked by native speaker, possible errors corrected C: Non-native speaker quality Obvious problems shown in corpus statistics are corrected D: First version Pre-processing with default abbreviation list and default sentence boundaries E: Poor Quality: Old, outdated or faulty. Corpus Quality The quality of the corpora differes slightly because the corpus processing toolchain changed slightly during several years. Moreover, original data are often no more available. Hence, improvement of quality often means removing incomplete or doubtful sentences. Forthcoming editions of all corpora thus might have a slightly smaller number of sentences. This especially applies to near duplicate sentences which are removed only sparingly. The following table shows the quality of the corpora. Minimal errors are still possible and described in the sections below. All possible major improvements are mentioned here.

14 CES corpora 4 Corpus Quality rating Known problems to-dos ces_news_ A - - ces_news_2008 A - - ces_news_2009 A - - ces_news_2010 A - - ces_news_2011 A - - ces_news_2012 A - - ces_newscrawl_2011 A - - ces_newscrawl_2012 A - - ces_web_2002 A - - ces_web_2011 A - - ces_web_2012 A - - ces_wikipedia_2007 A - - ces_wikipedia_2012 A - - ces_mixed_2012 A - - Processing Overview For more details, see Appendix: Database Summary and Appendix: Number of sources by time period. Corpus Size (M sentences) Size (M running words) Multiwords Crawling date Production date ces_news_ end of 2005-end of ces_news_ daily ces_news_ daily ces_news_ daily ces_news_ daily ces_news_ daily ces_newscrawl_ / ces_newscrawl_ / ces_web_ batch crawl ces_web_ / / ces_web_ / / ces_wikipedia_ / ces_wikipedia_ / ces_mixed_ see above 2013

15 CES corpora 5 Content Overview For more details, see Appendix: Size of different TLDs and Appendix: Size of different domains. Corpus Type of sources Countries Number of sources Publishing date Biggest source ces_news_ News cs(93%), sk(7%) 75 mainly 3/ /2007 ihned.cz/ ces_news_2008 News cs(99%), sk(1%) HN.IHNED.CZ/ ces_news_2009 News cs HN.IHNED.CZ/ ces_news_2010 News cs hn.ihned.cz/ ces_news_2011 News cs zpravy.idnes.cz/ ces_news_2012 News cs(94%), com(6%) isport.blesk.cz/ ces_newscrawl_2011 News cs and before ces_newscrawl_2012 News cs and before ces_web_2002 Web cs (100%) and before ces_web_2011 Web cs(92%), sk(2%), com(2%), eu(2%) and before abc.blesk.cz/ ces_web_2012 Web cs(93%), sk(1%), com(2%), eu(2%) and before darren-shan.ic.cz/ ces_wikipedia_2007 Wikipedia and before wikipedia.org ces_wikipedia_2012 Wikipedia and before wikipedia.org ces_mixed_2012 Mixed Sources cs(95%), sk(1%), com(1%), eu(1%) and before HN.IHNED.CZ/ Words Appendix: Words by Length without multiplicity shows a plot of the corresponding length distribution. A smooth asymetric bell-shaped curve is expected. Appendix: Words by Length with multiplicity shows a plot of the corresponding length distribution. A smooth asymetric bell-shaped curve is expected. Appendix: The Most Frequent 50 Words shows the most frequent stopwords as well as one or more words related to the region. Appendix: Longest Words in Top-1000 by rank shows the 25 longest words within the top The usually give an impression of the main topics treated in the corpus. Appendix: Longest Words with minimum frequency 2 should give an idea of very long words. In the case of processing problems, different types of non-words may appear. This might help to improve the word definition. Corpus Word length graph without multiplicity Word length graph with multiplicity Most Frequent 50 Words Longest Words in Top-1000 Longest Words with minimum frequency 2 ces_news_ okay okay okay okay URLs, missing blanks ces_news_2008 okay okay okay okay URLs ces_news_2009 okay okay okay okay URLs ces_news_2010 okay okay okay okay URLs, missing blanks, junk ces_news_2011 okay okay okay okay URLs, missing blanks ces_news_2012 okay okay okay okay URLs, routes, missing blanks

16 CES corpora 6 ces_newscrawl_2011 okay okay okay okay Missing blanks, routes, chemicals, URLs ces_newscrawl_2012 okay okay okay okay URLs, missing blanks, junk, etc. ces_web_2002 okay okay okay okay URLs, missing blanks ces_web_2011 okay okay okay okay URLs, missing blanks ces_web_2012 okay okay okay okay Routes, missing blanks, URLs ces_wikipedia_2007 okay okay okay okay okay ces_wikipedia_2012 okay okay okay okay chemicals, URLs ces_mixed_2012 okay okay okay okay all of the above Abbreviations Abbreviations are usually not used as sentence boundaries. Conversely, missing abbreviations can overgenerate sentence boundaries. Due to limitations in the processing chain, the list of abbreviations used for sentence boundary detection can differ from the abbreviations in the word list. Appendix: Most Frequent Abbreviations shows possible under-generation of sentence boundaries by wrong abbreviations (i.e. words ending in a full stop) in the word list. Sentences Appendix: Shortest sentences shows the shortest declarative, exclamatory and interrogative sentences. In preprocessing, a minimal length for sentences might be specified. And missing abbreviations are often visible as faulty sentence engings. Appendix: Longest sentences shows the longest declarative, exclamatory and interrogative sentences. Usually, the maximun sentence length is defined as 256 characters (not 256 bytes). Very long exclamatory or interrogative sentences often contain an overseen sentence boundary. Appendix: Length of sentences in characters shows the distribution of the sentence length. A large and balanced corpus will result in a smooth and bell-shaped curve. Isolated local maxima usually result from large sets of near duplicate sentences. Corpus Shortest sentences Longest sentences Length distribution (in characters) Length distribution (in words) ces_news_ okay max. 255 bytes instead characters several near duplicate peaks okay ces_news_2008 okay okay okay okay ces_news_2009 okay okay okay okay ces_news_2010 okay okay okay okay ces_news_2011 okay okay okay okay ces_news_2012 okay okay several near duplicate peaks okay ces_newscrawl_2011 okay okay okay okay ces_newscrawl_2012 okay okay okay okay ces_web_2002 okay okay max. 255 bytes instead characters okay

17 CES corpora 7 ces_web_2011 okay okay okay okay ces_web_2012 okay okay okay okay ces_wikipedia_2007 okay okay several near duplicate peaks several near duplicate peaks ces_wikipedia_2012 okay okay several near duplicate peaks okay ces_mixed_2012 okay okay okay okay Oddities Appendix: Sentences with high average word length: Average sentences contain many stopwords, and these stopwords are usually short. Hence, they restrict the average word length in a sentence. Conversely, sentences with high average word length are often ill formed. They may be used to improve pre-processing. Appendix: Problems with sentence segmentation - Words ending in a stopword: If there are many ill-formed word or sentence boundaries witout a blank between two words, they will generate new ill-formed words. The appendix shows the most frequent words ending in an uppercase stopword. If they are infrequent then the date were of high quality. Corpus Sentences with high average word length Words ending in a stopword ces_news_ URLs okay ces_news_2008 URLs okay ces_news_2009 URLs, missing blanks Numbers like 05:00Na,maxfreq=23 ces_news_2010 URLs, missing blanks, very long words Numbers like 05:00Na,maxfreq=26 ces_news_2011 URLs, missing blanks, very long words, junk Numbers like 05:00Na,maxfreq=8 ces_news_2012 URLs, missing blanks, very long words, junk maxfreq=9 ces_newscrawl_2011 URLs, missing blanks, very long words maxfreq=7 ces_newscrawl_2012 URLs, missing blanks, very long words maxfreq=18 ces_web_2002 URLs, often: missing blanks maxfreq=16 ces_web_2011 URLs, often: missing blanks, routes maxfreq=16 ces_web_2012 URLs, missing blanks, routes okay ces_wikipedia_2007 URLs, chemicals, Russian words okay ces_wikipedia_2012 URLs, missing blanks, Chinese okay ces_mixed_2012 as above as above

18 CES corpus comparison 8 CES corpus comparison Automated Corpus comparison For the following comparisons, the following tests on the top-1000 words are performed: Vectors based on the frequencies of the top-1000 words are created for the analysed languages. The cosine of the angle between these vectors is computed. Identical languages receive a value of 0, distinct languages get a value of 1. The same analysis is conducted using the frequencies of the top-1000 typical letter trigrams of the languages. Monolingual word list comparison (top-1000 words) As one can expect the comparisons show: The different news corpora have different word lists with maximum distance 0.16 (ces_newscrawl_2012 and ces_news_2011) The wikipedia corpora are similar with maximum distance 0.07 The web corpora have maximum distance 0.12 (ces_news_2012 and ces_news_2002) The mixed corpus ces_mixed_2012 holds a central position with maximum distances of 0.32 to the other corpora. Multilingual word list comparison (top-1000 words) Both the comparison of the top-1000 words and the comparison of the letter trigrams used in these words show that there are similar languages in our data, all being members of the slavic family. The distance of the mixed corpus to the next language, Slovak, is 0.54 for the words and 0.59 for the letter trigrams. Both distances are about average. The average value for the most similar language is 0.58 for trigrams. The most similar languages based on words: Slovak, Polish, Crotian source language_short_name language_name cos_logfreq ces slk Slovak ces pol Polish ces hrv Croatian ces hsb Sorbian, Upper ces slv Slovenian The most similar languages based on letter trigrams: Slovak, Croatian, Slovenian source language_short_name language_name cos_logfreq ces slk Slovak ces hrv Croatian ces slv Slovenian ces srp-latn Serbian (Latin) ces pol Polish

19 9 Processing details Appendix to ces news : Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces news 2008: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences

20 Appendix to ces news 2009: Database summary 10 Appendix to ces news 2009: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces news 2010: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences

21 Appendix to ces news 2011: Database summary 11 Appendix to ces news 2011: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces news 2012: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Number of distributional similar word pairs (NOT READY) 0 Number of similar sentence pairs (NOT READY) 0

22 Appendix to ces newscrawl 2011: Database summary 12 Appendix to ces newscrawl 2011: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces newscrawl 2012: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences

23 Appendix to ces wikipedia 2007: Database summary 13 Appendix to ces wikipedia 2007: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces wikipedia 2012: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences

24 Appendix to ces web 2002: Database summary 14 Appendix to ces web 2002: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces web 2011: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences

25 Appendix to ces web 2012: Database summary 15 Appendix to ces web 2012: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences Appendix to ces mixed 2012: Database summary Values for some general parameters Parameter Value Number of sentences Number of running word forms Number of distinct word forms Number of multiwords Percentage of words with frequency= Number of sentence based co-occurrences Number of neighbour co-occurrences

26 16 Content details Appendix to ces news : Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ sk/ Appendix to ces news 2008: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ sk/

27 Appendix to ces news 2009: Size of different TLDs 17 Appendix to ces news 2009: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ Appendix to ces news 2010: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ Appendix to ces news 2011: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/

28 Appendix to ces news 2012: Size of different TLDs 18 Appendix to ces news 2012: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ com/ Appendix to ces newscrawl 2011: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ Appendix to ces newscrawl 2012: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/

29 Appendix to ces web 2002: Size of different TLDs 19 Appendix to ces web 2002: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ Appendix to ces web 2011: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ eu/ sk/ com/ Appendix to ces web 2012: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ com/ eu/ sk/

30 Appendix to ces mixed 2012: Size of different TLDs 20 Appendix to ces mixed 2012: Size of different TLDs TLDs larger than 1% TLD # of sources %.cz/ com/ eu/ Appendix to ces news : Size of largest domains Largest domains Source # of sentences ihned.cz/ HN.IHNED.CZ/ zpravy.idnes.cz/ EKONOM.IHNED.CZ/ HNonline.sk/ pes.eunet.cz/ # of distinct sources 75

31 Appendix to ces news 2008: Size of largest domains 21 Appendix to ces news 2008: Size of largest domains Largest domains Source # of sentences HN.IHNED.CZ/ zpravy.idnes.cz/ deniksport.blesk.cz/ ihned.cz/ Domaci.iHNed.cz/ Zahranicni.iHNed.cz/ # of distinct sources 98 Appendix to ces news 2009: Size of largest domains Largest domains Source # of sentences HN.IHNED.CZ/ zpravy.idnes.cz/ deniksport.blesk.cz/ EKONOM.IHNED.CZ/ Domaci.iHNed.cz/ Sport.iHNed.cz/ # of distinct sources 78

32 Appendix to ces news 2009: Size of largest domains 22 Appendix to ces news 2010: Size of largest domains Largest domains Source # of sentences hn.ihned.cz/ zpravy.idnes.cz/ deniksport.blesk.cz/ EKONOM.IHNED.CZ/ isport.blesk.cz/ Domaci.iHNed.cz/ # of distinct sources 81

33 Appendix to ces news 2011: Size of largest domains 23 Appendix to ces news 2011: Size of largest domains Largest domains Source # of sentences zpravy.idnes.cz/ isport.blesk.cz/ rssfeeds.ihned.cz/ HN.IHNED.CZ/ byznys.ihned.cz/ zpravy.ihned.cz/ sport.ihned.cz/ # of distinct sources 119 Appendix to ces news 2012: Size of largest domains Largest domains Source # of sentences isport.blesk.cz/ zpravy.idnes.cz/ zpravy.ihned.cz/ HN.IHNED.CZ/ byznys.ihned.cz/ sport.ihned.cz/ idnes.cz.feedsportal.com/ # of distinct sources 105

34 Appendix to ces news 2012: Size of largest domains 24 Appendix to ces newscrawl 2011: Size of largest domains Largest domains Source # of sentences ekonom.ihned.cz/ plzensky.denik.cz/ # of distinct sources 35

35 Appendix to ces newscrawl 2012: Size of largest domains 25 Appendix to ces newscrawl 2012: Size of largest domains Largest domains Source # of sentences plzensky.denik.cz/ ekonom.ihned.cz/ # of distinct sources 36 Appendix to ces web 2002: Size of largest domains Largest domains Source # of sentences krystal.op.cz/ osz.cmkos.cz/ # of distinct sources 13471

36 Appendix to ces web 2002: Size of largest domains 26 Appendix to ces web 2011: Size of largest domains Largest domains Source # of sentences abc.blesk.cz/ abicko.avcr.cz/ sw.gurroa.cz/ osz.cmkos.cz/ # of distinct sources 78437

37 Appendix to ces web 2012: Size of largest domains 27 Appendix to ces web 2012: Size of largest domains Largest domains Source # of sentences darren-shan.ic.cz/ sw.gurroa.cz/ oficialnistranky.cz/ synopse.startrek.cz/ web.meulovo.cz/ 5980 # of distinct sources Appendix to ces mixed 2012: Size of largest domains Largest domains Source # of sentences HN.IHNED.CZ/ zpravy.idnes.cz/ EKONOM.IHNED.CZ/ # of distinct sources

38 Appendix to ces mixed 2012: Size of largest domains 28 Appendix to ces news : Number of sources by time period Number of sources by year, month, and day Number of sources per year year # of sources % Number of sources per month

39 Appendix to ces news : Number of sources by time period 29 month # of sources % Appendix to ces news 2008: Number of sources by time period Number of sources by year, month, and day

40 Appendix to ces news 2008: Number of sources by time period 30 Number of sources per year year # of sources % Number of sources per month month # of sources %

41 Appendix to ces news 2009: Number of sources by time period 31 Appendix to ces news 2009: Number of sources by time period Number of sources by year, month, and day Number of sources per year year # of sources % Number of sources per month month # of sources %

42 Appendix to ces news 2009: Number of sources by time period Appendix to ces news 2010: Number of sources by time period Number of sources by year, month, and day Number of sources per year year # of sources % Number of sources per month

43 Appendix to ces news 2010: Number of sources by time period 33 month # of sources % Appendix to ces news 2011: Number of sources by time period Number of sources by year, month, and day

44 Appendix to ces news 2011: Number of sources by time period 34 Number of sources per year year # of sources % Number of sources per month month # of sources %

45 Appendix to ces news 2012: Number of sources by time period 35 Appendix to ces news 2012: Number of sources by time period Number of sources by year, month, and day Number of sources per year year # of sources % Number of sources per month month # of sources %

46 Appendix to ces news 2012: Number of sources by time period

47 37 Word details Appendix to ces news : Words by length without multiplicity Percentage of words of fixed length in characters, counted without multiplicty Average word length word length percentage

48 Appendix to ces news : Words by length without multiplicity

49 Appendix to ces news : Words by length without multiplicity Appendix to ces news 2008: Words by length without multiplicity Percentage of words of fixed length in characters, counted without multiplicty Average word length word length percentage

50 Appendix to ces news 2008: Words by length without multiplicity

51 Appendix to ces news 2008: Words by length without multiplicity Appendix to ces news 2009: Words by length without multiplicity Percentage of words of fixed length in characters, counted without multiplicty Average word length word length percentage