Introduction

CEFR: goals and principles

The CEFR acronym stands for the Common European Framework of Reference for Languages. It is a framework developed by the Council of Europe and is widely used for describing language proficiency levels and learning outcomes. In this section, we shall sketch the goals and principles of CEFR, before focusing specifically on the relationship between CEFR vocabulary levels and users’ interest in words as reflected in dictionary searches.

The CEFR serves as an important resource in the area of language education, offering a unified basis for outlining language learning objectives and outcomes (Council of Europe, 2001). This framework significantly aids curriculum planners and language teachers in determining the desired proficiency levels for learners, establishing clear objectives, content, and methods. What sets the CEFR apart is its emphasis on communicative competence, aiming to enhance learners’ ability to effectively engage in real-life situations through the practical application of language skills in speaking, listening, reading, and writing (Council of Europe, 2023). This approach shifts the focus from theoretical knowledge to the practical usage of language, fostering real communicative proficiency.

The CEFR framework is widely used worldwide, more so than any other framework (Foley, 2019, 29). Nevertheless, some national alternatives do exist, such as the CES in China (Ministry of Education of the People’s Republic of China, 2018); and there are also national adaptations or extensions of the CEFR, such as the CEFR-J in Japan (Negishi et al., 2013) or FRELE-TH in Thailand (Hiranburana et al., 2017). However, the fact that most of these use CEFR at least as a starting point testifies to the wide applicability and popularity of the CEFR framework.

As a descriptive framework, CEFR characterizes the capabilities of learners at various stages of language proficiency through “can-do statements.” These statements provide a clear depiction of what learners are capable of at each level, rather than dictating specific language content or curricular structures. The utility of the CEFR extends beyond just learner assessment; it plays a crucial role in the design of language curricula, teaching programs, learning materials, and assessment tools. Moreover, it offers a standard for comparing curricula, textbooks, courses, and examinations across different languages (Council of Europe, 2018). In line with contemporary educational priorities, the CEFR principles also underscore the importance of developing plurilingual and intercultural competence. This approach champions the acquisition of multiple languages and the understanding of diverse cultures and perspectives as central objectives of language learning (Çelik, 2013). This holistic view of language education puts emphasis on communicative abilities but also encourages a broader cultural understanding and adaptability among learners.

The CEFR is closely linked to the European Language Portfolio (ELP), an instrument based on the CEFR (Little, 2009). The ELP allows individual learners to reflect upon their learning, record their progress, and self-assess their language proficiency using the CEFR descriptors. As mentioned above, these descriptors are formulated in terms of “can-do statements” appropriate to the language abilities of learners at each level. These statements outline specific tasks and activities that learners are expected to be able to perform, helping learners set realistic goals and track their progress. At the same time, CEFR recognizes that language learning is a lifelong process. It encourages learners to continue improving their language skills beyond the classroom and provides a foundation for further language development and self-assessment (Council of Europe, 2020, 244).

The CEFR framework is not without problems. Critics have pointed out the insufficient nuance and rigidity of the CEFR levels, which are not in principle sensitive to context of use or learning context (for CEFR levels see below; Krumm, 2007, Widdowson, 2015). Further, the CEFR uses speakers as the benchmark (Widdowson, 2015) and has limited usefulness in standardized language testing (Weir, 2005). Alderson (2007) also notes the vague language used in the framework that is difficult to translate into actual practice.

More pertinently to the goals of this study, the CEFR defines six common reference levels: A1, A2, B1, B2, C1, C2 (Council of Europe, 2020, 36–37), where A1 is the most elementary (“breakthrough”) and C2 most advanced (“mastery”). These levels describe the learner’s proficiency in reception, production, interaction, and other language competences. These reference levels provide a basis for comparing language curricula, textbooks, courses, and exams.

A crucial component of any language is its vocabulary stock, forming the essential “building blocks” of texts. The CEFR levels can be, and have been, used to grade vocabulary items, specifying the typical learner level at which specific words (or, sometimes, their specific senses or uses) are appropriately known. Unfortunately—and perhaps surprisingly given the fundamental role of vocabulary in communication—the CEFR initiative within the Council of Europe has stopped short of constructing any “official” inventories of vocabulary levels. Those that do exist appear to be independent efforts, often in the context of projects involving the creation of learning materials, and many of them commercial in nature, which makes it difficult to source CEFR wordlists for research purposes.

CEFR levels and interest in vocabulary items

Given that one of the aims of the CEFR is to assign measures of relevance to vocabulary items, it is worth examining how well its grading system aligns with observable behavior. A possible metric to use is the frequency of online dictionary searches. Essentially, if CEFR grading truly reflects a word’s importance, then lower-level vocabulary items should find more interest from online dictionary users. As it happens, there is only one online dictionary for the English language—the English Wiktionary—that is well-suited to this purpose, because it is freely available, has exceptionally extensive coverage of words (far more than any available alternative; at the time of this writing, January 2024, about 8 million entries overall, of which about 1 million are English entries), and—crucially—has data on usage freely available for download. These statistics of users visiting Wiktionary pages holding specific dictionary entries can be adopted as a useful operationalization of user interest in English vocabulary items.

Turning now to the possible source of CEFR grading for English, there exist at least two CEFR-graded vocabulary lists that appear to enjoy a degree of prestige while being of non-trivial size, that is on the order of at least thousands of items. These are the English Vocabulary Profile (Capel, 2012, 2015, Cambridge University Press, 2015) and the Oxford Learner’s Word Lists (Oxford University Press Oxford Learner’s Word Lists (2023)). Due to copyright considerations, the English Vocabulary Profile was adopted as a source of CEFR levels for English words in this research.

Dictionary logs as a source of data

Logs of dictionary visits have been studied before. One motivation behind such studies has been to examine user behavior (Müller-Spitzer et al., 2015), mostly with an eye to improving dictionaries to better serve their user (Lemnitzer, 2001, Bergenholtz and Johnson, 2005, Lorentzen and Theilgaard, 2012, Trap-Jensen et al., 2014). Another reason for examining dictionary logs has been to explore the relationships between user visits and lexical frequency. This relationship was of relevance in assessing the appropriateness of corpus-based methods in dictionary-making. In particular, a series of studies sought to establish whether corpus frequency—i.e., the relative number of occurrences of a word in a large collection of texts—was a sensible guide in choosing headwords to include in a dictionary. If it could be shown that corpus frequency can predict what users look up in a dictionary, and if this relationship turns out to be positive, then lexicographic work can be optimized to prioritize high-frequency items. This line of research initially brought disappointing findings, with several early studies only finding a weak positive relationship for a few thousand most frequent items, but not much of a pattern beyond that threshold (De Schryver and Joffe, 2004, De Schryver et al., 2006, Verlinde and Binon, 2010). However, further research using more refined methods showed that the effect in the lower frequency ranges was masked by a long tail of very low-frequency items, some of which are nevertheless sometimes looked up (Koplenig et al., 2014a, Müller-Spitzer et al., 2015). Koplenig et al. (2014a) demonstrated that dictionary views are positively related to corpus frequency, while Müller-Spitzer et al. (2015) also found a positive effect of polysemic status (i.e., the word having more than one sense in the dictionary).

Two further possible factors, albeit having less impact, were recently identified by Lew and Wolfer (2024): word prevalence and age of acquisition. This last study, also using logs of Wiktionary views but for a shorter period, suggests that user interest in English entries may be affected by a number of lexical factors, namely (in decreasing measure): (1) a word’s lexical frequency; (2) its polysemy status (whether monosemous or polysemous); (3) word prevalence; and (4) age of acquisition. These findings should also be taken into account in investigating the relationship between a word’s CEFR level and its look-up frequency, and we intend to do just that in the present study.

As we have previously presented the concept of CEFR levels, the reader also deserves some background on the other four factors as identified in related research discussed above.

The first of these is lexical frequency, which is the frequency of a lemma (or, sometimes, a word-form) in a corpus (=a large digital collection) of texts. Frequency can simply be a raw count of how often a word appears in a corpus, although it is often expressed as a standardized frequency per one million tokens to make it independent of corpus size. As discussed earlier, most studies have found that words that are higher in frequency tend to attract more user interest (Koplenig et al., 2014b, Müller-Spitzer et al., 2015, De Schryver et al., 2019).

Polysemy is a property of a lexical item having more than one sense, or distinct meanings (see e.g., Van der Meer 2004). Müller-Spitzer et al. (2015) found that polysemous words tend to attract more views.

The prevalence of a word indicates how widely it is known among speakers (Weizman and Snow, 2001, Longobardi et al., 2015). In simpler terms, a word that has high prevalence is known to a large proportion of speakers. One study (Lew and Wolfer, 2024) found a weak negative effect of a word’s prevalence on the tendency to look it up in a dictionary.

Finally, age of acquisition (Garlock et al., 2001, Juhasz, 2005, Kuperman et al., 2012) refers to the typical age at which a given word is learned in a naturalistic setting. At least one study (Lew and Wolfer, 2024) found that words acquired late in life tend to attract more views than those learned in early life.

Reflecting on the underlying nature of the factors discussed above, we might write that lexical frequency indicates how common a word is in a large collection of texts, which in turn corresponds to the frequency with which people come across a word. Polysemy status describes, as it were, semantic versatility (as seen by creators of the entry for the word). Age of acquisition is a (recalled and reported) typical age at which the word is learned, reflecting a certain developmental sequence that in turn tells us about how important a word is to a developing human speaker at a particular stage. Prevalence is about how widely a given word is known in the population. These measures are not independent. Conversely, it is reasonable to expect that words with many senses, words learned at an early age, and words known to many people are more likely to be more frequent on average, other things being equal. In modeling the relationship between these variables and user interest (as indicated by entry views), these mutual interrelationships should be taken into account.

CEFR vocabulary levels provide information about words that is qualitatively different: these levels reflect expert judgements about a word’s usefulness to language learners at a particular stage. As such, CEFR levels have a pedagogical purpose. In what follows, we shall attempt to assess whether CEFR vocabulary levels correspond to the degree of user interest in specific words, as measured by dictionary consultation.

Study

Aims

We derive the following research questions and hypotheses from our introductory remarks.

Question 1: Does the CEFR vocabulary level of an English word (as defined in the English Vocabulary Profile) carry predictive power with respect to how often an entry for this word is viewed in the English Wiktionary?

We especially focus on the CEFR levels A1 to B2 and, to a lesser extent on the conflated C1/2 levels. We take the views in the English Wiktionary as a quantifiable measure of user interest in vocabulary items.

Hypothesis 1: Our first hypothesis is a directed one: lower-level vocabulary items should find more interest from online dictionary users. In terms of views, we expect a hierarchy of A1 > A2 > B1 > B2 > C1/2.

Question 2: If Hypothesis 1 holds, we are further interested in whether the CEFR level still carries independent predictive power as soon as the effects of other lexical variables (e.g., word frequency) are controlled for.

Hypothesis 2: The CEFR level of an English word carries independent predictive power over other lexical variables. We do not have a strong hypothesis as to where the CEFR level fits into the hierarchy of predictors. However, we assume that at least the established effect of corpus frequency is stronger than that of the CEFR level.

Method

Data sources

Data on English Wiktionary views were derived from daily server logs covering a period of four years from 2019-01-01 to 2022-12-31. These statistics are provided by the Wikimedia Foundation and are freely downloadable, but the volume of the original data is huge, given that the data include page visits from all Wikimedia projects (Wikipedia, Wiktionary, Wikidata, Wikimedia, Wikinews, etc.) and needed to be filtered to select the data specific to the English Wiktionary.

We sourced the CEFR levels for English lexis from the English Vocabulary Profile (Capel, 2015). Since there was no way of knowing which exact sense users sought in the Wiktionary, we always used the lowest CEFR level if an item happened to have different CEFR levels for different senses. This seemed to make the most sense, given that these were the most frequent and essential senses and thus likely to drive the larger part of dictionary consultations. In any case, the majority of items only had single CEFR levels, so this was not a major issue.

Further, lexical frequency information was drawn from the SUBTLEX-US corpus (as described in Brysbaert and New, 2009). To obtain information on polysemy, we extracted the number of senses for each entry word in the English Wiktionary. The custom extraction function written in R (R Core Team, 2023) accesses the edit page of each article, and is available upon request. Age-of-acquisition (AoA) ratings were extracted from the supplementary material attached to Kuperman, Stadthagen-Gonzalez, and Brysbaert (2012). Finally, word prevalence values derive from the supplementary material published alongside Brysbaert et al. (2019). Both AoA and prevalence were assessed by human participants in very large language surveys. The methodology is described in detail in the respective publications.

Modeling

To verify if, and to what extent, the CEFR levels of lexical items predicted the number of views of their entries in the Wiktionary, we ran a regression model (after checking model assumptions) predicting logged views on the CEFR level of the same lexical item. Because the CEFR scale is meant to progress in stepwise fashion from A1 all the way to C2, we used the contrast coding that gave justice to this idea, namely backward-difference coded contrasts, using the R package codingMatrices (Venables, 2023). Due to the poor representation of the highest C1 and C2 levels in the CEFR resource—each one holding fewer vocabulary items than either of the B levels—we conflated these two levels into a single combined level C.

As the distribution of Wiktionary look-up data is skewed towards low values (a large proportion of entries are looked up only occasionally), we log-transformed the views variable, achieving a near-normal distribution for the purpose of model computation. Similarly, we used log-transformation on the standardized corpus frequency variable. This ensured that the distribution of the residuals of the linear model was near-normal. Further, we also standardized all continuous predictors (i.e.,: age of acquisition, log frequency, and prevalence) to z-scores by subtracting the mean from each value and then dividing by the corresponding standard deviation. This operation brings the values of the predictors into the same range with a mean value of 0 and a standard deviation of 1, making linear regression model estimates comparable.

Results

Words with CEFR levels versus words in the Wiktionary

To see whether CEFR-level information was related to user interest in the associated dictionary entries, we crossed it with information on the identity and popularity of all existing English Wiktionary entries drawn from Wikimedia logs.

The CEFR resource yielded nearly ten thousand items, both single- and multi-words, each with an assigned CEFR level. Of these items in the CEFR dataset, 67 per cent had an equivalent English Wiktionary entry, but this varied by CEFR level, with details as given in Table 1. It will be seen that the coverage declines with increasing CEFR level, from nearly 90 per cent for the most basic A1 level, down to only about half at the highest CEFR level of C2.

Table 1 Coverage of items from the CEFR dataset in the Wiktionary, broken down by CEFR Level.
Full size table

CEFR level and views

In order to see to what extent the CEFR levels of lexical items predicted the number of views of their entries in the Wiktionary, we regressed the logged views on the CEFR level of the corresponding lexical item. The model’s R2 was 0.235 (adjusted R2: 0.234), and the coefficient estimates are given in Table 2. It will be seen that all two-way differences between consecutive CEFR levels are significant, meaning that the step from one to the other makes a significant contribution to predicting user views. The direction of the relationship is such that words assigned a higher (i.e., more advanced) CEFR levels tend to attract fewer views. Conversely, more basic vocabulary items tend to get more views.

Table 2 Parameter estimates of a regression model predicting logged views from CEFR levels.
Full size table

The outcome of this model is summarized graphically in Fig. 1, which also includes a beeswarm-plot representation of the complete underlying data.

Fig. 1: Views (log scale) by CEFR level.
Fig. 1: Views (log scale) by CEFR level.
Full size image

Levels C1 and C2 have been conflated as C. Each colored dot marks one word. Predicted values of Wiktionary views from a linear model with CEFR level as predictor, with 99.9% Confidence Intervals in red.

Thus far, we have explored the relationship between CEFR level and the number of views of the corresponding Wiktionary entry. We will now expand our model to include more information for the English items: specifically, the four lexical factors outlined earlier in this paper: (1) lexical frequency; (2) polysemy; (3) prevalence; and (4) age-of-acquisition. In this part of the analysis, we only kept items that had all these pieces of information.

Refining the model: CEFR + lexical variables

Having combined the CEFR information with the other four lexical predictors, we fitted a multiple regression model with Wiktionary views (logged) as the outcome variable and standardized predictors. The model’s R2 was 0.503 (adjusted R2: 0.502) and parameter estimates are given in Table 3.

Table 3 Parameter estimates of a regression model predicting logged views from CEFR level, frequency, polysemy, prevalence, and age of acquisition.
Full size table

The CEFR-level effect in this more complex multiple regression model was significant, except in the pairwise difference B2-B1, which was marginally significant, while C-B2 was just shy of marginal significance (at p = 0.105). This suggests that CEFR is a useful predictor of user interest even once the other predictors are controlled for. Frequency, polysemy, prevalence, and age-of-acquisition were all highly significant and effects pointed in the previously suggested directions (note the positive versus negative signs in the Estimate column of Table 3). Because one would expect the predictors to be correlated, we checked for collinearity using the VIF and the GVIF1/(2⋅df). The magnitude of variance inflation appeared to be in safe territory for all predictors (with the highest VIF values being 2.33 for the VIF and 1.53 for the GVIF1/(2⋅df)). The CEFR effects estimated by this model are summarized graphically in Fig. 2, showing predicted mean views with 95% Confidence Intervals.

Fig. 2: Effect of CEFR Level on Views in the English Wiktionary after correcting for effects of frequency, polysemy, prevalence, and age of acquisition.
Fig. 2: Effect of CEFR Level on Views in the English Wiktionary after correcting for effects of frequency, polysemy, prevalence, and age of acquisition.
Full size image

Shown are predicted mean views with 95% Confidence Intervals. Levels C1 and C2 have been conflated as C. Lower CEFR levels are viewed more often.

Relative importance of CEFR versus other predictors

To estimate the relative importance of the predictors, we computed the change in adjusted R2 after dropping each predictor from the model, as given in Table 4. The adjusted R2 of the complete model was 0.502. A reduction in adjusted R2 when a given predictor is dropped from the model represents the loss of explanatory power of the model, and thus the relative importance of that predictor. It will be seen from Table 4 that CEFR level comes out as the third most important predictor, following lexical frequency and polysemy, but ahead of prevalence and age-of-acquisition.

Table 4 Adjusted R2 and reduction in adjusted R2 after dropping individual predictors, ordered by decreasing importance.
Full size table

Discussion and implications

The English Vocabulary Profile CEFR-graded vocabulary exhibits marked variation in terms of its coverage in the English Wiktionary. The coverage is nearly complete (90 percent) for the most basic A1 level, but declines with increasing CEFR level, so that at the highest CEFR level of C2, only about every second vocabulary items is covered. This is to be expected, given that the lower levels of the CEFR scale are designed to cover the most basic vocabulary items, while the vocabulary range at CEFR level C2 is in principle open-ended. In addition, items at the higher CEFR levels in the English Vocabulary Profile often contain lexicographic placeholders (such as sb, sth) and special symbols such as slashes. For the current contribution, we did not try to resolve these but went with exact matches between CEFR item and Wiktionary page/entry title.

Our study shows that CEFR level is a useful predictor of user interest in English lexical items. The relationship is such that words from the lower (more basic) levels are looked up more in the English Wiktionary, and that in the simple regression model this relationship holds for each individual increment between CEFR levels (except between C1 and C2, for which the data are very incomplete, so these have been conflated as C). Adding the four further lexical predictors—that is corpus frequency, polysemic status, age-of-acquisition, and prevalence—improves the model in terms of its predictive power (the coefficient of determination R2 doubles), which suggests that these lexical factors also hold useful information that translates into increments in user interest. Nevertheless, CEFR remains a significant factor also in the multiple regression model. Specifically, in this model, the increments from A1 to A2 and from A2 to B1 were significant, B1 to B2 marginally significant, and B2 to C came very close to marginally significant. The inclusion of these other lexical factors ensures that CEFR level is not merely a proxy for another factor, notably lexical frequency. Thus, CEFR level contributes additional information on how useful a lexical item is to Wiktionary users, over and above that carried by the frequency of the item, its age-of-acquisition, prevalence, and polysemy status. Comparing the relative importance of the five variables in terms of how they affect user views, it will be seen from Table 4 that CEFR level ranks as the third most important predictor, following lexical frequency and polysemy, but ahead of prevalence and age-of-acquisition.

In conclusion, the two analyses presented here suggest the presence of the effect of CEFR when considered as a sole predictor, but also in combination with four other lexical factors in a multiple regression model. All in all, the idea to include CEFR information in dictionaries, particularly at the sense level, is a promising avenue of research which could benefit both lexicography and the field of second-language learning.

Limitations and future work

CEFR levels are primarily designed with language learners in mind, but the Wikimedia page view data do not (and cannot) include any information on the personal characteristics of page visitors. Therefore, it is impossible to assess from this data what proportion of the English Wiktionary views come from language learners. Having said that, CEFR may also have some relevance for someone who is not actively learning a language (e.g., language teachers or writers who want to draft a text for a specific audience).

Another, more general problem, concerns the completeness and accuracy of CEFR labeling as such. As already mentioned, CEFR information for the highest C1 and C2 levels is rather fragmentary and likely not representative. In principle, these levels do not even have a cap on vocabulary coverage, and C2 in particular is generally thought to be an open-ended set. These reservations might also to a degree apply to level B2. This issue could be addressed in the future if more complete data on upper-level vocabulary become available. More generally, CEFR labeling as such is not necessarily consistent, fully principled, and remains necessarily arbitrary up to a point. There are also concerns about CEFR lacking specificity and nuance in its one-size-fits-all prescriptions (Weir, 2005, Foley, 2019).

The CEFR vocabulary set includes some specific and even arbitrary formulations that combine variants of multi-word expressions and lexicographic meta-words (sth, sb, etc.). The following list is just a small sample of such problematic items.

 not be cut out to be sth/not be cut out for sth

 be slow to do sth; be slow in doing sth

 day by day/little by little/one by one, etc.

 so did we/so have I/so is mine, etc.

 from the 1870s/March/6.30 pm, etc. onwards

It is no surprise that such phrases find no exact matches amongst the English Wiktionary entries. Further work might explore whether CEFR entries such as these can reliably be mapped automatically to entry titles in the English Wiktionary.

Another avenue for future research might leverage the relationship between CEFR level and other known lexical variables, as discovered in studies such as the present attempt, to try to produce, adjust, or supplement CEFR-grading.