The MERLIN texts are the writings sections of CEFR-related, standardized high-quality tests from telc (Frankfurt/Main, Italian and German tests, homepage) and ÚJOP (Prague, Czech tests, homepage). These institutions are ALTE-audited (ALTE-homepage). The tasks were in use until 2013 and are now freely available on the platform. However, to have explicit and direct information about the CEFR profiles of the written productions themselves (and not only of the tests as a whole), for MERLIN all texts were re-rated independently by two professional raters per language. The reliability of the re-ratings was examined with the help of Classical Test Theory and a Multi-Facet Rasch analysis. The latter is a probabilistic statistical procedure often used in language testing which allows for a correction of rating tendencies (e.g., leniency/harshness) and makes it possible to arrive at a fair average rating for each text. The intra-rater and inter-rater reliability was generally very high in MERLIN, with some exceptions for Italian. Therefore, the whole re-rating process was repeated for Italian resulting in a satisfying rating quality. In MERLIN, the fair average is calculated based on a holistic scale (see 1.2 rating instruments). If you compile your own corpus based on CEFR levels, these are also based on the fair average ratings (Define a subcorpus » Overall CEFR rating). If you are interested in more details regarding the quality of the ratings and the difficulty of the single rating criteria, please consult the technical report.