Lemmatization of nonstandard words

[harisont]

The guidelines for typos and other errors in underlying text build upon the correct assumption that most token-level "errors" in standard texts are performance errors (hence the feature Typo=Yes). This is not the case in learner data, where a significant fraction of them have to do with incorrect inflection, derivation and/or orthography, as well as nonstandard lexical choices.

In UD_Swedish-SweLL, inspired but UD_Italian-Valico, we decided to lemmatize according to literal criteria, i.e. based on the characteristic of the observed word form (see also #1178).

Examples from UD_Swedish-SweLL (feel free to add):

1. [inflection] förslagor is lemmatized as förslaga: it is clear that the learner means förslag ('proposal(s)'), but the plural is built as if the noun belonged to a declension where the base form would be förslaga
2. [derivation] upplevelsa - an interesting mix of upplevelse, 'experience' (NOUN) and uppleva, 'experience' (VERB) - is lemmatized as upplevelsa
3. [orthography] villja is lemmatized as villja (rather than vilja, 'want'). This is unlikely to be a typo because the source text is handwritten, the most common form of the verb is the present vill (with two Ls) and (at least to Italian ears, I'm an L2 speaker myself!) the word villja arguably sounds like it has a long (double) L, so we think it is an actually interesting error to preserve
4. [lexical choice] brottsligheten is lemmatized as brottslighet ('criminality'), even if the normalized version of the sentence changes that to brott ('crime').

1, 2 and 4 may be more or less controversial, but they do not pose any annotation issues.
3, on the other hand, is an unfortunate case where we end up with an error:

[Line 2552 Sent org-176-test]: [L5 Morpho aux-lemma] 'villja' is not an auxiliary in language [sv]

If I recall correctly, @ElisaDiNuovo solved a similar problem in UD_Italian-Valico by using the "corrected lemma" only in the very few cases where the "descriptive" lemma would be incompatible with the UPOS and/or DEPREL at hand. This is a reasonable quick fix (which we also had to implement for UD 2.17, see UniversalDependencies/UD_Swedish-SweLL#1), but I think it creates unnecessary local inconsistencies.

A simple solution could be to introduce a CorrectLemma field in MISC, e.g.

3	villja	villja	AUX	_	Typo=Yes|VerbForm=Inf|Voice=Act	5	aux	_	CorrectionLabels=O|CorrectLemma=vilja

When available, the validator could check that rather than the LEMMA itself. When unavailable, the usual rules would hold.

[amir-zeldes]

Thanks for bringing this up - I'm honestly a bit unsure which principle to follow: I understand that the learner utterance is the primary data and we want to be faithful to it, but on the other hand the point of lemmatization is at least in part to collapse some of the variability in language down to lexical entries we assume are part of the language's inventory.

For instance, if someone spelled the word "botttle" with three "t"s in a non-learner test, I would not expect it to be lemmatized that way - I would expect the annotations to state that this is a (misspelled) form of the lemma "bottle". And maybe that's OK - after all, lemmas are interpretations and not primary data. In this paper about annotating non-canonical user generated text online, we argued that for example emoji standing in for a word should still be lemmatized to that word (so ❤️ could have the lemma "love" in context), because that is the lexical item being referenced by the form, even though "love" doesn't literally appear.

If you want to record the hypothetical lemma form that would exist if we really had a misspelled inflected form as a word, then maybe that could be put in a MISC annotation as HypotheticalLemma, instead of a CorrectLemma? For me the CorrectX annotations imply that the text contains the mistake in question, but if someone misspelled "knew" as "new", I'm not sure if they actually think that the lemma is "now" instead of "know". All I know for sure is that they misspelled "knew", so I'd prefer to add the CorrectForm and still lemmatize to "know", because it's an instance of that lexical entry in the language, which I could then link to a dictionary, word sense annotation, etc.

[harisont]

You definitely have a point. If I wasn't annotating learner productions, I would probably go for this approach. However, I don't see how to deal with cases like 2 and 4 (which are relatively common in my data) in a way that is consistent with your examples. When someone writes upplevelsa, it is clear to me that the learner thinks that the lemma is upplevelsa itself, and uppleva is my just hypothesis (actually, I would argue that for villja as well, as both of these are infinitives, which is also what we put in the LEMMA field for Swedish verbs). When they write brottsligheten, which is a valid form of brottslighet, saying brott is the lemma seems far-fetched. If we agree on these two cases, I don't see why, in the same corpus, we should treat the remaining two (1 and 3) differently (there is also a fifth case, actually: words that are so misspelled that they become unrecongnizable).

So I wonder, is this an aspect of annotation where learner corpora should (or at least should be given the freedom to) differ from corpora of other domains? My suggested solution deliberately does not force any treebank to switch approach.

[ellepannitto]

I get both points of view and I'm perfectly aware that lemmatization itself is a step of abstraction so any decision we make is a categorization and nothing is gonna tell us what morphological derivation or paradigm the speaker had in mind...
Anyway, I do have a similar situation in particular with cases of interlingua (is this the right english word for the variety that emerges when two similar languages get in contact?), where speakers sort of code switch but not quite. This is data that we're about to treebank as a next step for UD_Italian-KIParlaForest so thanks @harisont for raising the issue before I stumbled upon the problems!

Some examples (the word is marked by $):

1. il francese per forza è la seconda lingua, è come~ per me è come l'arabo, e:: l'inglese, più o meno: ehm uguale al all'italiano perchè e:hm non l'abbiamo studiato $approfodo~ $approfondamente, trad. en. "french of course is the second language, it's like arabic for me, and english more or less the same as italian because we haven't studied it $thoroughly. Here the morphology of the Italian adverb is off, the standard form is "approfonditamente", which is a de-adjectival adverb from the adjective "approfondito" (en. detailed). The -mente` suffix is the right derivational suffix to build adverbs in italian but what is missing is the fact that it should be applied to feminine adjectives. If the base of the adverb is a verb then the suffix should be "-oni".
   In this case it would be easy to provide a "correct" lemma in MISC, and the speaker is Arabic so probably no influence from Arabic can be supposed, but we're not really being faithful to the morphological process but rather to the interpreted meaning.

2. è (già) difficile che il mio compagno c'ha $epilepsia c(io)è non lo puoi lasciare da solo, trad. en. "it is difficult already as my partner suffers from epilepsy I mean you can't leave him alone". Here the speaker is Romanian and the target lexeme in italian would be "epilessia" but the romanian word for it is "epilepsie", so the most likely hypothesis is that the speaker is using a Romanian lexeme with Italian morphology. As I'm treebanking Italian, I want to annotate the fact that the speaker is bringing some repertoire into Italian so from that perspective it is pretty clear that the lemma is "epilepsia". Here again however is pretty much ok to add the "Correct" form in MISC.

3. noi siamo $rivenuti al siamo arrivati all'aeroporto e un fred:do ((ride)) (.) che non potevamo immaginare , trad. en. "we came back (?) to we arrived at the airport and the cold we couldn't imagine". Here the speaker is Spanish and "rivenuti" is really a stange mash up between "venire" (italian) and probably "revenir" (spanish) which has however a completely different meaning. In Italian there exists "ritornare" though as "come back", so that is probably what cause the confusion. Again however the morphology is fully Italian so this cannot be considered a case of code switching, and I don't have a "correct" lemma. I can use "rivenire" as the lemma which is by pure coincidence acceptable in Italian but probably not what the speaker was thinking about when producing this, but at the same time I wouldn't be able to provide an alternative correct form. So ideally what I'd want to do in this case is just to mark that this is a case where I'm reconstructing a lemma from a substandard production, and maybe mark the Spanish supposed lemma rather than the Italian one.

4. poverino si $aggraftafava dalla porta per non andare no:, trad. en. "poor thing he was holding on(?) to the door not to go". Here the speaker is Romanian and the matter is more complicated as again the morphology is Italian so no code switching but at the same time the lemma does not exists, it doesn't seem to come directly from any Romanian verb (a friend is suggesting maybe agăța could be the only option still the strange consonants seem odd). It seems pretty obvious from context that the speaker was meaning "aggrappare" (to hold on to something) in Italian, but then it's not only the verb but also the preposition that is off, as Italian would require "alla" but Romanian has "de" which probably mislead the speaker. So in this case, instead of marking a "Correct" form, I would rather just lemmatize what I see (and use the non existent "aggrafrafare" as a lemma) and then use what is being discussed in Nonstandard constructions: syntactic calques #1181 to mark the strange construction.

To sum up my position, I think that the cases arising can be pretty different from one another and we should come up with a solution that allows us to preserve the morphological process (i.e., I don't want a parser to learn that "approfondamente" is a right derivation as "approfonda" is not an adjective), mark what could be the target lemma in the speaker's L1 (as in cases 2 and 3) or just mark the fact that that lemma is reconstructed but not attested in standard iItalian (as in case 4).

And of course the fact that issues only arise sometimes (i.e., when the validator complains) makes room for hiding a lot of inconsistencies about this in my opinion.

[sylvainkahane]

I am not sure to see how you want to use the reconstructed lemma of an incorrect form (supposing that it is always possible to do that). I think that the only things we need is: the observed form, the corrected form and the lemma of the corrected form. For instance, I can ask the treebank which lexeme/lemma are problematic for learners.
But we don't want a direct link between an incorrect form and the lemma of the correct form. So a solution could be to leave the LEMMA field empty ad to only have in MISC the CorrectForm and CorrectLemma. Another solution could be to fill the LEMMA field with the corrected lemma identified by a special symbol. Example:

form=rivenuti
lemma=*ritornare
CorrectForm=ritornati
CorrectLemma=ritornare
CalquedLang=sp
CalquedLemma=revenir

[harisont]

I am not sure to see how you want to use the reconstructed lemma of an incorrect form

In my case, my hypothesis is that (actually, I know for sure but my treebank still is too small to show that) some orthography, spelling, and derivation errors are relatively common. Reconstructed lemmas allow me to see how often the errors occurs, which learners (e.g. grouped by L1) make the same mistakes etc., independent of the specific word form.