Better way to release internally parallel treebanks (e.g. parallel learner treebanks)

[harisont]

We recently released a parallel learner treebank (learner productions // teacher corrections).
To do that, we followed the same approach as all other treebanks of this kind:

• Parallel: no in the metadata (because Parallel: ITSELF is neither allowed nor particularly clear)
• corrections in a separate file in the not-to-release folder.

I wonder whether we could come up with a solution that:

1. makes it clear from the website that these treebanks are, indeed, parallel
2. allows including both "halves" in the official release.

I don't have strong opinions about what the best way to achieve this would be, so I'm eager to read your suggestions.

[martinpopel]

Udapi supports sentence-aligned parallel treebanks stored in a single file, each language in a separate zone (named usually using the ISO language code, but in practice it can be any string, e.g. "learner" and "teacher"), so that for example the sequence of sent_ids in a English-German treebank would be: s1/en, s1/de, s2/en, s2/de,... as described in #321 (in 2016).
However, UD has decided to keep each language in a separate git repo, so this proposal has never been applied for UD releases (validate.py even checks that sent_id does not contain the slash character, so that it can be used for the zones). Udapi allows to simply load two conllu files (each with one language and the same number of sentences) and merge them into a single file. This can be useful for further analysis (e.g. adding word alignment links and visualizing them; Udapi supports several styles of visualizations with zones side by side). Udapi provides also the opposite procedure of splitting the single-file parallel treebank into two files.

In the case of learner/teacher parallel treebanks, we could say both halves are in the same language, so theoretically both halves could be released in a single git repo and even in a single file. Theoretically, the Udapi style of interleaved sent_ids could be used here: s1/learner, s1/teacher, s2/learner, s2/teacher,... We would just need to add an exception for the slash-in-sent-id test in validate.py. The advantage would be that all other UD metadata could be kept (e.g. #newdoc id). Tools not supporting zones would report double number of sentences in each document (and file), but they could still process such files. That said, I am not sure if this is the best way to go - if multilingual parallel treebanks are still released in separate git repos, while learner/teacher parallel treebanks are released in a different style.

If we want to keep the same style as multilingual parallel treebanks, currently the only option is create two git repos: one for the learner and one for the teacher half of the parallel treebank.

[harisont]

I personally wouldn't mind either of these two solutions, although none of them solves the metadata issue by itself.

Another obvious option could be to allow more than one (parallel) file per split in each repo, e.g. sv_swell-ud-test1.conllu sv_swell-ud-test2.conllu, or better sv_swell-ud-test-learner.conllu sv_swell-ud-test-teacher.conllu, but there are probably too many tools that rely on the current naming conventions.

[ellepannitto]

I think that having everything in the same file can cause confusion: it's good that Udapi supports it but it may end up with people training on files that contain multiple istances of the same sentence. (And sure, it's on people to be careful about what they train on but the whole architecture is made so that you can expect something to be true no matter the treebank, and the train/dev/test split is there for that reason as it was also recently discussed > #1146 )

On the other hand having two repos that are maintained by the same team seems also unnecessary as, as it was pointed out recently (#691 ), adding treebanks also brings load on the infrastructure.

An idea could be to keep the usual filenames (xxx-train.conllu, xxx-dev.conllu, xxx-test.conllu) for the part that is considered to be the core of the work (in case of learner treebanks, I would argue is the learner half because the other half wouldn't exist without the learner part), and have tree more files in the same repo with the other half of the data, with analogous names (something like xxx-train-teacher.conllu, xxx-dev-teacher.conllu, xxx-test-teacher.conllu).
This way tools that are used to the standard file naming conventions should have no problems, general purpose models would be trained on a reasonable split and no data gets lost in the process which is what is really suboptimal at the moment.

[harisont]

...I wouldn't mind this solution either :)
In fact, I think it's an improvement over the two-repo option.