Readability of Encyclopedias

Much research has already been carried out on the quality of information in Wikipedia as well as the perception of quality or credibility by its users. However, a different concern has recently been raised, namely about its readability.

In a study on the accuracy of cancer information (Rajagopalan, et al., 2010) on Wikipedia, it was found that while information on this topic was of comparable accuracy with information from a professionally maintained database, readability of Wikipedia was significantly worse. This was investigated by applying Flesch–Kincaid readability tests (Kincaid, et al., 1975) to articles from both Web sites and reported by means of the Flesch–Kincaid grade level. This enables a comparison with reading levels expected at the various grades in the U.S. educational system. While poor readability was only shown for Wikipedia articles on cancer, we have no reason to expect that articles on different topics will score better.

Wikipedia does not state a particular target audience for whom the encyclopedia was built, except for Jimmy Wales’ quote at the beginning of this paper. He stated that the sum of all human knowledge is made accessible for every single person. However, we hypothesize that information on Wikipedia is often not accessible to many, because the articles are too difficult to read. Walraven, et al. (2009) found frequent use of Wikipedia among 14–year–old high school students. These adolescents are not expected to have fully developed reading abilities and may thus have limited access to information on Wikipedia.

Readability can be described as the ease with which a reader can understand the message conveyed by a writer. Several attempts have been made to evaluate the readability of texts automatically (DuBay, 2004). One of the earlier attempts is the Flesch reading ease test (Flesch, 1948), which incorporates two easily measurable concepts deemed important for readability in a simple algorithm.

The article “Readability of Wikipedia”, by T Lucassen, R Dijkstra, JM Schraagen and published in First Monday, 2012, investigates the readability of both the English and Simple English Wikipedia by means of the Flesch reading ease test.

The results of this study show that the readability of the English Wikipedia is overall well below a desired standard. Although the average score of 51.18 does not seem far from the desired goal, nearly 75 percent of all articles scored below 60 in the Flesch reading ease test. Moreover, half of the articles can be classified as difficult or worse. This finding confirms our hypothesis that numerous articles on Wikipedia are too difficult to read for many people.

This readability problem was already demonstrated for articles on cancer (Rajagopalan, et al., 2010). For articles on this topic, a Flesch–Kincaid grade level of 14 was found, which reflects a reading ease score of about 30. Fortunately, we showed that the average readability of the English Wikipedia is considerably higher than this score, although numerous articles (about seven percent, which amounts to about 12,500 articles) scored 30 or below. This indicates that readability may vary heavily between different topics on Wikipedia. A topical analysis (like Halavais and Lackaff, 2008) can now reveal which articles most urgently need to improve readability. Entries on technical topics may be less readable than articles on other subjects, such as popular culture.

The urgency of improving different articles also depends on usage statistics. Articles with many page views need to be improved more quickly than those on arcane subjects seeing few readers. This can also be related to users with a limited proficiency in English, as they might be more interested in certain topics over others. The use of openly available data on page views for each Wikipedia article could help in prioritization topics for improvement.
Wikimedia Foundation proposed a solution to this problem in the form of the Simple English Wikipedia. This version of Wikipedia performed much better on readability tests than the English Wikipedia. However, the results show that readability dropped 10 points in the last four years (Besten and Dalle, 2008), which means that it now falls in the Standard category. This is a worrying result, as it seems that the Simple English Wikipedia has lost its focus. Instead, this version now seems suitable for the average reader, instead of aiming at those with limited language abilities. One way to check the success of the Simple English Wikipedia is to test how many of words in it appear on the 850– and 1,500–word Basic English lists (Ogden, 1968).

A possible explanation for the poor readability of both versions of Wikipedia rests with the contributors to Wikipedia. It can be expected that these are predominantly well–educated individuals, as it requires a substantial level of domain knowledge to be able to add content to Wikipedia which is not already available. It may be possible that these contributors write entries for their peers, instead of considering the broad and generic audience of Wikipedia. Even if they pay special consideration to their audience, as in the case of the Simple English Wikipedia, they seem to overestimate their audience’s reading skills.

It would be interesting to see how contributors of Wikipedia could be encouraged and supported in improving the readability of their entries. Perhaps it could be achieved by implementing a tool in the edit environment of Wikipedia, which alerts contributors to long sentences or difficult words. Suggestions to break up sentences or to use easier wording could be given during editing. The nature of Flesch readability tests makes it possible to have an online readability measure available for contributors to Wikipedia.

It should be noted that the Simple English Wikipedia is still rather underdeveloped. This is not only reflected by the relatively low number of entries but also by their average length. During our analysis, we noticed that about half of the articles consisted of only three sentences or less. This observation gives extra motivation to shift the focus of contributors towards enhanced readability, especially at this stage. We expect that it is more effective to write articles in an easy way initially, rather than to improve readability in later versions.

References and Further Reading

Matthijs L. den Besten and Jean–Michel Dalle, 2008. “Keep it simple: A companion for Simple Wikipedia?” Industry and Innovation, volume 15, number 2, pp. 169–178.https://dx.doi.org/10.1080/13662710801970126

William H. DuBay, 2004. “The principles of readability,” at https://www.nald.ca/library/research/readab/readab.pdf, accessed 20 August 2012.

Rudolph Flesch, 1948. “A new readability yardstick,” Journal of Applied Psychology, volume 32, number 3, pp. 221–233.https://dx.doi.org/10.1037/h0057532

Jim Giles, 2005. “Internet encyclopaedias go head to head,” Nature, volume 438, number 7070 (15 December), pp. 900–901, and at https://www.nature.com/nature/journal/v438/n7070/full/438900a.html, accessed 20 August 2012.

Alexander Halavais and Derek Lackaff, 2008. “An analysis of topical coverage of Wikipedia,” Journal of Computer–Mediated Communication, volume 13, number 2, pp. 429–440.https://dx.doi.org/10.1111/j.1083-6101.2008.00403.x

J. Peter Kincaid, Robert P. Fishburne, Richard L. Rogers, Brad S. Chissom, 1975. “Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy enlisted personnel,” Millington, Tenn.: Chief of Naval Technical Training, Naval Air Station Memphis; Springfield, Va.: distributed by National Technical Information Service (NTIS).

Ida Kubiszewski, Thomas Noordewier, and Robert Costanza, 2011. “Perceived credibility of Internet encyclopedias,” Computers & Education, volume 56, number 3, pp. 659–667.https://dx.doi.org/10.1016/j.compedu.2010.10.008

Teun Lucassen and Jan Maarten Schraagen, 2011. “Factual accuracy and trust in information: The role of expertise,” Journal of the American Society for Information Science and Technology, volume 62, number 7, pp. 1,232–1,242.

Charles K. Ogden, 1968. Basic English: International second language. New York: Harcourt, Brace & World.

Felipe Ortega, Jesus M. Gonzalez–Barahona, and Gregorio Robles, 2009. “Quantitative analysis of the top ten Wikipedias,” In: Joaquim Filipe, Boris Shishkov, Markus Helfert, and Leszek Maciaszek (editors). Software and data technologies: Second International Conference, ICSOFT/ENASE 2007, Barcelona, Spain, July 22–25, 2007, revised selected papers. Communications in Computer and Information Science, volume 22. Berlin: Springer–Verlag, pp. 257–268.

M.S. Rajagopalan, V. Khanna, M. Stott, Y. Leiter, T.N. Showalter, A. Dicker, and Y.R. Lawrence, 2010. “Accuracy of cancer information on the Internet: A comparison of a Wiki with a professionally maintained database,” Bodine Journal, volume 3, number 1, article 8, at https://jdc.jefferson.edu/bodinejournal/, accessed 20 August 2012.

Amber Walraven, Saskia Brand–Gruwel, and Henny P.A. Boshuizen, 2009. “How students evaluate information and sources when searching the World Wide Web for information,” Computers & Education, volume 52, number 1, pp. 234–246.https://dx.doi.org/10.1016/j.compedu.2008.08.003

Dennis M. Wilkinson and Bernardo A. Huberman, 2007. “Cooperation and quality in Wikipedia,” WikiSym ’07: Proceedings of the 2007 International Symposium on Wikis, pp. 157–164, and at https://www.hpl.hp.com/research/scl/papers/wikipedia/wikipedia07.pdf, accessed 20 August 2012.