en>ChrisGualtieri: Remove stub template(s). Page is start class or higher. Also check for and do General Fixes + Checkwiki fixes using AWB

2013-12-27T01:25:08Z

Remove stub template(s). Page is start class or higher. Also check for and do General Fixes + Checkwiki fixes using AWB

New page

'''Word lists by frequency''' are lists of a language's words grouped by frequency of occurrence within some given [[text corpus]], either by levels or as a ranked list, serving the purpose of [[vocabulary acquisition]]. A word list by frequency "provides a rational basis for making sure that learners get the best return for their vocabulary learning effort", ({{Harvnb|Nation|1997|p=}}) but is mainly intended for course writers, not directly for learners. Some major pitfalls are the corpus content, the corpus [[register (sociolinguistics)|register]], and the definition of "[[word]]". While word counting is a thousand years old, with still gigantic analysis done by hand in the mid-20th century, [[natural language processing|natural language electronic processing]] of large corpora such as movie subtitles (SUBTLEX megastudy) has accelerated the research field.

In [[computational linguistics]], a '''frequency list''' is a sorted list of [[word]]s (word types) together with their [[frequency]], where frequency here usually means the number of occurrences in a given [[text corpus|corpus]], from which the rank, less meaningful, can be derivared
{| class="wikitable right"
! Type || Occurences || Rank
|-
| the || 3789654 || 1st
|-
| he || 2098762 || 2nd
|-
| [...] || ||
|-
| king || 57897 || 1,356th
|-
| boy ||56975 || 1,357th
|-
| [...] || ||
|-
| stringyfy || 5 || 34,589th
|-
| [...] || ||
|-
| transducionalify || 1 || 123,567th
|}
== Methodology ==
=== Factors ===
Nation ({{harvnb|Nation|1997}}) noted the incredible help provided by computing capabilities, making corpus analysis much easier. He cited several key issues which influence the construction of frequency lists:
* corpus representativeness
* word frequency and range
* treatment of word families
* treatment of idioms and fixed expressions
* range of information
* various other criteria

=== Corpuses ===
;Traditional written corpus
{{main|Text corpus}}
Most of currently available studies are based on written texts.

;SUBTLEX movement
However, {{harvnb|New|Brysbaert|Veronis|Pallier|2007}} proposed to tap into the large number of subtitles available online to analyse large numbers of speeches. {{harvnb|Brysbaert|New|2009}} made a long critical evaluation of this traditional textual analysis approach, and support a move toward speech analysis and analysis of film subtitles available online. This has recently been followed by a handful of copy-cat studies, providing valuable frequency count analysis for various languages. Indeed, the SUBTLEX movement completed in five years full studies for French ({{harvnb|New|Brysbaert|Veronis|Pallier|2007}}), American English ({{harvnb|Brysbaert|New|2009}}; {{harvnb|Brysbaert|New|Keuleers|2012|p= }}), Dutch ({{harvnb|Keuleers|New|2010|p= }}), Chinese ({{harvnb|Cai|Brysbaert|2010|p= }}), Spanish ({{harvnb|Cuetos|Glez-nosti|Barbón|Brysbaert|2011|p= }}), Greek ({{harvnb|Dimitropoulou|Duñabeitia|Avilés|Corral|Carreiras|2010|p= }}), and Vietnamese ({{harvnb|Pham|Bolger|Baayen|2011|p= }}).

=== Lexical unit ===
In any case, the basic "word" unit should be defined. For Latin scripts, words are usually one or several characters separated either by spaces or punctuation. But exceptions can arise, such as English "can't", French "aujourd'hui", or idioms. It may also be preferable to group words of a [[word family]] under the representation of its [[base word]]. Thus, ''possible, impossible, possibility'' are words of the same word family, represented by the base word ''*possib*''. For statistical purpose, all these words are summed up under the base word form *possib*, allowing the ranking of a concept and form occurrence. Moreover, other languages may present specific difficulties. Such is the case of Chinese, which does not use spaces between words, and where a specified chain of several characters can be interpreted as either a phrase of unique-character words, or as a multi-character unique word.

=== Statistics ===
It seems that [[Zipf's law]] holds for frequency lists drawn from longer texts of any natural language. Frequency lists are a useful tool when building an electronic dictionary, which is a prerequisite for a wide range of applications in [[computational linguistics]].

German linguists define the ''Häufigkeitsklasse'' (frequency class) <math>N</math> of an item in the list using the [[binary logarithm|base 2 logarithm]] of the ratio between its frequency and the frequency of the most frequent item. The most common item belongs to frequency class 0 (zero) and any item that is approximately half as frequent belongs in class 1. In the example list above, the misspelled word ''outragious'' has a ratio of 76/3789654 and belongs in class 16.
:<math>N=\left\lfloor0.5-\log_2\left(\frac{\text{Frequency of this item}}{\text{Frequency of most common item}}\right)\right\rfloor</math>
where <math>\lfloor\ldots\rfloor</math> is the [[floor function]].

Frequency lists, together with [[semantic network]]s, are used to identify the least common, specialized terms to be replaced by their [[hypernym]]s in a process of [[semantic compression]].
=== Pedagogy ===
Those lists are not intended to be given directly to students, but rather to serve as a guideline for teachers and book makers ({{Harvnb|Nation|1997|p=}}). [[Paul Nation]]'s modern language teaching summary encourages first to "move from high frequency vocabulary and special purposes [thematic] vocabulary to low frequency vocabulary, then to teach learners strategies to sustain autonomous vocabulary expansion" ({{harvnb|Nation|2006la}}).

== Effects of words frequency ==
Word frequency is known to have various effects ({{harvnb|Brysbaert|Buchmeier|Conrad|Jacobs|Bölte|Böhl|2011|p= }}; {{harvnb|Rudell|1993|p= }}). Memorization is positively affected by higher word frequency, likely because the learner is subject to more exposures ({{harvnb|Laufer|1997}}). Lexical access is positively influenced by high word frequency ({{harvnb|Segui|Mehler|Frauenfelder|Morton1982|p= }}).

== Languages ==
Below is a review of available resources.

=== English ===
Word counting dates back to [[Hellenistic]] time. Thorndike & Lorge, assisted by their colleagues, counted 18,000,000 running words to provide the first large scale frequency list in 1944, before modern computers made such projects far easier ({{harvnb|Nation|1997}}).

==== Traditional lists ====
;The Teachers Word Book of 30,000 words (Thorndike and Lorge, 1944)
The TWB contains 30,000 lemmas or ~13,000 word families (Goulden, Nation and Read, 1990). A corpus of 18,000,000 written words was hand analysed. The size of its inputted corpus increased its usefulness, but its age and language change reduced its applicability ({{harvnb|Nation|1997}}).

;[[General Service List|The General Service List]] (West, 1953)
The GSL contains 2,000 headwords divided into two sets of 1,000 words. A corpus of 5,000,000 written words was analysed in the 1940s. Rate of occurrence (%) for different meanings and parts of speech of the headword are provided, while it was also a careful application of the various criteria other than frequency and range. Thus, despite its age, some errors, and its solely written base, it is still an excellent database (word frequency, frequency of meanings, reduction of noise) ({{harvnb|Nation|1997}}).

;The American Heritage Word Frequency Book (Carroll, Davies and Richman, 1971)
A corpus of 5,000,000 running words, from written texts used in United States schools (various grades, various subject areas). Its value is in its focus on school teaching materials, and its tagging of words, namely the frequency of each word in each of the school grade levels and in each of the subject areas ({{harvnb|Nation|1997}}).

;The Brown (Francis and Kucera, 1982) LOB and related corpora
These now contain 1,000,000 words from a written corpora representing different dialects of English. These sources are used to produce frequency lists ({{harvnb|Nation|1997}}).

=== French ===
;Traditional datasets
A review has been made by {{harvnb|New|Pallier|3.01}}.
An attempt was made in the 1950s–60s with the [[:fr:Français fondamental|Français fondamental]]. It includes the F.F.1 list with 1,500 high-frequency words, completed by a later F.F.2 list with 1,700 mid-frequency words, and the most used syntax rules.<ref>{{cite web|url=http://pedagogie.ac-amiens.fr/casnav/accueillir/fle_fls/didactique/francais_fondamental.htm|title=Le français fondamental}}{{dead link|date=June 2012}}</ref> It is claimed that 70 grammatical words constitute 50% of the communicatives sentence,<ref>{{citation|title=Comprendre et aider les enfants en difficulté scolaire: Le Vocabulaire fondamental, 70 mots essentiels|last1=Ouzoulias|first1=André|publisher=Retz|year=2004|url=http://www.ac-caen.fr/ia50/circo/cou/IMG/pdf/70_mots_essentiels.pdf}} - Citing V.A.C Henmon</ref> while 3,680 words make about 95~98% of coverage.<ref>{{cite web|title=Generalities|url=http://www.riviere.info/vbr/livre/ch1generalites.htm}}</ref> A list of 3,000 frequent words is available.<ref>{{cite web|title=PDF 3000 French words|url=http://robert.riviere.info/download.php?op=mydown&did=4}}</ref>

The French Ministry of the Education also provide a ranked list of the 1,500 most frequent [[word family|word families]], provided by the lexicologue [[Étienne Brunet]].<ref>{{cite web|title=Maitrise de la langue à l'école: Vocabulaire|url=http://eduscol.education.fr/pid23250-cid50486/vocabulaire.html|publisher=[http://eduscol.education.fr Ministère de l'éducation nationale]}}</ref> Jean Baudot made a study on the model of the American Brown study, entitled "Fréquences d'utilisation des mots en français écrit contemporain".<ref>{{citation|last1=Baudot|first1=J.|year=1992 |title=Fréquences d'utilisation des mots en français écrit contemporain|edition=|editor=|publisher=Presses de L'Université|place=|isbn=2-7606-1563-4|url=}}</ref>

More recently, the project [[Lexique 3]] provided a list of 135,000 French words, with [[orthography]], [[phonetic]], [[syllabation]], [[part of speech]], [[gender (linguistic)|gender]], number, frequency, associated [[lexeme]]s, etc., available under an open-source license<ref>http://www.lexique.org/</ref>

;Subtlex
{{harvnb|New|2007|p= }} made a completely new counting based on online film subtitles.

=== Spanish ===
There have been several studies of Spanish word frequency ({{harvnb|Cuetos|Glez-nosti|Barbón|Brysbaert|2011|p= }}).<ref>{{cite web|title=Spanish word frequency lists|publisher=Vocabularywiki.pbworks.com|url=http://vocabularywiki.pbworks.com/w/page/5254401/Spanish-word-frequency-lists}}</ref>

=== Chinese ===
As a frequency toolkit, Da ({{harvnb|Da|1998}}) and the Taiwanese Ministry of Education ({{harvnb|TME|1997}}) provided large databases with frequency ranks for characters and words. The [[Hanyu Shuiping Kaoshi|HSK]] list of 8,848 high and medium frequency words in the [[People's Republic of China]], and the [[Republic of China|Republic of China (Taiwan)]]'s [[Test of Proficiency-Huayu|TOP]] list of about 8,600 common traditional Chinese words are two other lists displaying common Chinese words and characters. Following the SUBTLEX movement, {{harvnb|Cai|Brysbaert|2010|p= }} recently made a rich study of Chinese word and character frequencies.

== References ==
<references/>

== See also==
* [[Letter frequency]]
* [[Most common words in English]]
* [[Long tail]]

== Sources ==
{{Wiktionary|Wiktionary:Frequency lists}}
; Theoretical concepts
* {{citation|last=Nation |first=I.S.P.|ref={{harvid|Nation|1997}}|editor1-last=Schmitt |editor2-last= McCarthy |title = Vocabulary: Description, Acquisition and Pedagogy |chapter= Vocabulary size, text coverage, and word lists |place= Cambridge |publisher= Cambridge University Press |year=1997 |pages=6–19 |isbn=978-0-521-58551-4 |url=http://www.lextutor.ca/research/nation_waring_97.html}}
* {{citation|last1=Laufer,, B. |first1=|year=1997 |chapter=What’s in a word that makes it hard or easy? Some intralexical factors that affect the learning of words. |editor=|title=Vocabulary: Description, Acquisition and Pedagogy |series=|isbn=9780521585514|publisher=Cambridge University Press|place=Cambridge|pages=140–155|url=}}
* {{citation|last=Nation |first=I.S.P. |authorlink1= Paul Nation |year=2006la |title=Language Education - Vocabulary|publisher=Encyclopedia of Language & Linguistics|place= Oxford|pages=494–499 |url= http://www.sciencedirect.com/science/article/B7T84-4M3C3K0-137/2/769ef6f2d69bd72cac7071fb7ba3b1f9}}.
* {{citation|last1=Brysbaert|first1=Marc|first2=Matthias|last2=Buchmeier|first3=Markus|last3=Conrad|first4=Arthur M|last4=Jacobs|first5=Jens|last5=Bölte|first6=Andrea|last6=Böhl|year=2011 |chapter=The word frequency effect: a review of recent developments and implications for the choice of frequency estimates in German.|editor=|issn=|title=Experimental Psychology|volume=58|issue=5|pages=412–424 |url=|doi=10.1027/1618-3169/a000123}}
* {{citation|last1=Rudell|first1=A.P. |year=1993 |chapter=Frequency of word usage and perceived word difficulty : Ratings of Kucera and Francis words|editor=|issn=|title=Most|volume=25|issue=4|pages=455–463 |url=}}
* {{citation|last1=Segui|first1=J. |first2=Jacques|last2=Mehler|first3=Uli|last3=Frauenfelder|first4=John|last4=Morton|year=1982 |chapter=The word frequency effect and lexical access|editor=|issn=|title=Neuropsychologia|volume=20|issue=6|pages=615–627 |url=http://www.sciencedirect.com/science/article/pii/0028393282900616}}

*[[Helmut Meier]]: ''Deutsche Sprachstatistik''. Hildesheim: Olms 1967. (frequency list of German words)

;Written texts-based databases
* {{citation|last1=Da|first1=Jun|year=1998|title=Jun Da: Chinese text computing |url= http://lingua.mtsu.edu/chinese-computing/statistics/}} [Accessed August 21, 2010].
* {{citation|author=Taiwan Ministry of Education|ref={{harvid|TME|1997}}|year=1997|title=八十六年常用語詞調查報告書|url=http://www.edu.tw/files/site_content/M0001/86news/ch2.html?open}} [Accessed August 21, 2010].
* {{citation|last1=New|last2=Pallier|year=3.01|title=Manuel de Lexique 3|url=http://www.lexique.org/outils/Manuel_Lexique.htm#_Toc152122347}}

;SUBTLEX movement
* {{citation|last1=New|first1=B.|last2=Brysbaert|first2=M.|last3=Veronis|first3=J.|last4=Pallier|first4=C.|year=2007 |chapter=SUBTLEX-FR: The use of film subtitles to estimate word frequencies|editor=|issn=|title=Applied Psycholinguistics|volume=28|issue=4|pages=661 |url=http://sites.univ-provence.fr/veronis/pdf/2007-AppliedPsy.pdf}}
* {{citation|last1=Brysbaert|first1=Marc|first2=Boris|last2=New|year=2009 |chapter=Moving beyond Kucera and Francis: a critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English|editor=|issn=|title=Behavior Research Methods|volume=41|issue=4|pages=977–990 |url=http://expsy.ugent.be/subtlexus/Brysbaert%26NewBehaviorResearchMethods.pdf}}
* {{citation|last1=Keuleers|first1=E, M, B. |last2=New|first2=B. |year=2010 |chapter=SUBTLEX--NL: A new measure for Dutch word frequency based on film subtitles|editor=|issn=|title=Behavior Research Methods|volume=42|issue=|pages=643–650 |url=http://crr.ugent.be/programs-data/subtitle-frequencies/subtlex-nl}}
* {{citation|last1=Cai|first1=Q. |last2=Brysbaert|first2=M.|year=2010 |chapter=SUBTLEX-CH: Chinese Word and Character Frequencies Based on Film Subtitles|editor=|issn=|title=PLoS ONE|volume=5|issue=6|pages=8 |url=http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2880003/}}
* {{citation |last1=Cuetos |first1=F. |first2=Maria |last2=Glez-nosti |first3=Analía |last3=Barbón |first4=Marc |last4=Brysbaert |year=2011 |chapter=SUBTLEX-ESP : Spanish word frequencies based on film subtitles |title=Psicológica |volume=32 |pages=133–143 |url=http://www.uv.es/psicologica/articulos2.11/1CUETOS.pdf}}
* {{citation|last1=Dimitropoulou|last1=M.|first2=Jon Andoni |last2=Duñabeitia|first3=Alberto|last3=Avilés|first4=José|last4=Corral|first5=Manuel|last5=Carreiras |year=2010 |chapter=SUBTLEX-GR: Subtitle-Based Word Frequencies as the Best Estimate of Reading Behavior: The Case of Greek|editor=|issn=|title=Frontiers in Psychology|volume=1|issue=December|pages=12 |url=}}
* {{citation|last1=Pham|first1=H.|last2=Bolger|first2=P.|last3=Baayen|first3=R.H. |year=2011 |chapter=SUBTLEX-VIE : A Measure for Vietnamese Word and Character Frequencies on Film Subtitles|editor=|issn=|title=ACOL|volume=|issue=|pages= |url=}}
* {{citation|last1=Brysbaert|first1=M.|last2=New|first2=Boris|last3=Keuleers|first3=E.|year=2012 |chapter=SUBTLEX-US : Adding Part of Speech Information to the SUBTLEXus Word Frequencies|editor=|issn=|title=Behavior Research Methods|volume=|issue=|pages=1–22 |url=http://crr.ugent.be/papers/Adding%20Part%20of%20Speech%20information%20to%20the%20SUBTLEX-US%20word%20frequencies.pdf}} ([http://expsy.ugent.be/subtlexus/ databases])
* SUBTLEX-DE: [Not yet puclished: {{harvnb|Buchmeier|2012}}:] {{citation|last=Buchmeier|year=2012|title=The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German|url=http://scholar.googleusercontent.com/scholar?q=cache:WrKoBjtf0zEJ:scholar.google.com}} [http://crr.ugent.be/subtlex-de/ database]

{{Frequency list}}

[[Category:Language acquisition]]
[[Category:Quantitative linguistics]]
[[Category:Computational linguistics]]
[[de:Häufigkeitsklasse]]
[[hy:Հաճախականության բառարաններ]]
[[de:Frequenzwörterbuch]]
[[ru:Частотный словарь]]
[[uk:Частотний словник]]

Complexity function - Revision history

en>ChrisGualtieri: Remove stub template(s). Page is start class or higher. Also check for and do General Fixes + Checkwiki fixes using AWB