Syntactic annotation of a second-language learner corpus

Transkript

1 Syntactic annotation of a second-language Jirka Hana & Barbora Hladká Charles University Prague ICBLT 2018

2 CzeSL Corpus of L2 Czech ICBLT

3 CzeSL Czech as a Second Language Part of AKCES Acquisition Corpora of Czech Essays written by non-native speakers of Czech A1 C1 CEFR proficiency levels ICBLT

4 CzeSL two releases CzeSL-SGT 8,600 essays, 1.1M tokens CC BY-SA-3.0 CzeSL-man < we work with this here 645 essays, 120K tokens, 11K sentences Manually corrected and annotated for errors CC BY-SA-3.0 ICBLT

5 CzeSL number of documents by CEFR Level Level Documents Basic user A1 57 A1+ 3 A2 111 A Independent user B1 176 B2 124 Proficient user C1 12 Unknown 17 Total 645 ICBLT

6 Non-native and native language are different Non-native langue has: Errors in spelling, grammar, vocabulary, collocations Different distribution of vocabulary and syntactic constructions ICBLT

7 CzeSL: Error Annotation Scheme Tier 0 original text: Myslim že kdy by byl se svim ditem... Tier 1 words correct: Myslím že kdyby byl se svým dítětem... Tier 2 contextually correct: Myslím, že kdybych byl se svým dítětem... think SG1 that if SG1 was MASC with my child... `I think that if I were with my child. corrections ICBLT

8 Sample non-native text: My Family Jmenujese Adam. Ja jsem Mongolska. Mongolska ma 21 kraji. Moje rodina je hezka jeste velka. Mongolska je 3000 million lidi. Ma tradični píseňka, taneční. Mongolska tradicni píseňka je hezka. Ješte ma Morin khuur. Morin Khuur to je muzika. Ten hezka tradični pohádka, píseň. Mongolska má mnoho tradiční svátík. Třiba Naadam, Tsagaarsur. Ješte mnoho Velbloud, Kůn, Kravá, Koza, Ovce. Mongolsky lidi dobrý. Mongolsko ma mnoho hory a nemam ocean. Mongolska hlavní naměsto. Ulaanbaatar. ADAM, 18 Let Bydlim v Cechagh už 6 měsíc. ICBLT

9 Task: Annotate some structure of L2 Czech Motivation: - better understanding of L2 Czech (including its grammar) - better computational processing of L2 Czech Some structure? - the deeper, the better, ideally semantics - dependency syntax for practical purposes Work in progress... ICBLT

10 Universal Dependencies ICBLT

11 Universal Dependencies (UD) 100+ corpora in 60+ languages syntactic annotation based on dependency syntax language agnostic (mostly) ICBLT

12 Universal Dependencies (UD) Yesterday, John invited Mary to his birthday party. ICBLT

13 UD Annotation of Non-native Czech ICBLT

14 Annotating original text Annotate the original text, not corrections Ideal case: use grammar of author s interlanguage Reality: often, not enough data Be conservative, assume as little as possible ICBLT

15 Annotating original text Example 1 Standard oblique (adjunct): Vstoupit do místnosti. enter into room. `Enter a room. Non-native direct object: Vstoupit místnost. enter room. Intended: `Enter a room. ICBLT

16 Annotating original text Example 2 Standard nej `most is a prefix největší město v Kolumbii biggest city in Columbia `the biggest city in Columbia Non-native nej is a word: nej větší město v Kolumbii most bigger city in Columbia `the biggest city in Columbia ICBLT

17 Annotating original text an unclear example - stejné = equal (adjective) - stejně = equally (adverb) Oba jsou stejné důležité. both are equal important `Both are equally important What is it? - spelling error - adj/adv neutralization - other error What is the lemma, POS, syntactic fnc? ICBLT

18 Sometimes UD helps... Jsem Mongolska. `I am Mongolian / a Mongolian / from Mongolia Jsem mongolský. adjective, not in std language Jsem Mongol. inhabitant, noun Jsem z Mongolska. country, preposition + noun The same structure in UD ICBLT

19 Current status 2,100 sentences out of 11,000 annotated so far 100 sentences double annotated with Cohen s kappa: Universal POS: 0.93 Dependency Label: 0.89 Relation: 0.93 ICBLT

20 Future work More double annotated data More annotated data annotate the whole CzeSL Test standard and custom trained parsers ICBLT