I returned, and saw under the sun, that the race is not to the swift, nor the battle to the strong, neither yet bread to the wise, nor yet riches to men of understanding, nor yet favour to men of skill; but time and chance happeneth to them all.
Ecclesiastes
9:11
Scientific evaluation of Charles Dickens
Eye-opening results of the
Great prose or not? quiz
By Mikhail Simkin
This article appeared in February 2013 issue of the Journal of Quantitative Linguistics. Soon after Daily Mail published a positive review of this research. This triggered a flood of rash criticism most of which I refuted in my article Statistics against irritations. The remainig arguments of my critics I destroyed in the article Scientific inquiry into poetry. Before publication by the Journal of Quantitative Linguistics the article was accepted by Skeptic Magazine and I even got the page proof. However, at the end the editor caved in to pressure and did not publish the article.
Let me tell you about the very rich. They are different from you and me.
-- F. Scott Fitzgerald
Yes, they have more money.
-- Ernest Hemingway
Are the very famous writers different from the obscure ones?
Edward Bulwer-Lytton is the worst writer in history of letters. An annual wretched writing contest was established in his honor. In contrast, Charles Dickens is one of the best writers ever. Can one tell the difference between their prose? To check this I wrote the Great prose or not? quiz. It consists of a dozen representative literary passages, written either by Bulwer-Lytton or by Dickens. The takers are to choose the author of each quote. They face a formidable task and are often surprised to learn the correct answers. “What mindless boob would write such tripe? Dickens, one would know now.” – wrote me one of respondents. And the distribution of the scores received by over nine thousands quiz-takers is shown in Figure 1. The average score is 5.78 or 48.2% correct.
Figure 1. The histogram of the scores earned by 9,461 people on the “Great prose or not?” quiz. The average score is 5.78 or 48% correct. The standard error of this average is 0.022 or 0.19%.
There are two possible answers to each test question. If one is completely clueless and resorts to random guessing, he will on average get 50% of the questions right. With the average score of 48% our quiz-takers lost to a monkey. On average, a quote from Bulwer-Litton was selected as Dickens (or great prose) by 52% of quiz-takers, while a quote from Dickens was selected as Dickens by only 48%. Does this mean that Bulwer-Lytton is a better writer than Dickens? Probably not. Table 1 shows for every quote the fraction of people who attributed it to Dickens. This fraction varies between the quotes with the lowest being 36% (No. 9) and the highest 74% (No. 12). This suggests that a different selection of quotes could lead to a different average score. For example, if we remove the most Dickensian Bulwer (No. 12) and the most Bulwerian Dickens (No. 10), and recalculate the scores based on 10 remaining questions, - the average score becomes 51%.
An interesting thing is that out of 9,461 people 38 got every question wrong and 37 got everything right. The approximate equality of these numbers is consistent with random guessing, but their magnitude is not. It is more than fifteen times bigger than what random guessing would give. The explanation is that some of quiz takers can sniff stylistic similarities between certain literary passages and attribute them to the same writer. This would help them to get a higher score, provided that they can determine which writer is good and which is bad, otherwise they are equally likely to get a very high or a very low score.
Table 1. Fraction of people who attributed each quote to Dickens and Bulwer-Lytton, along with the true author.
| Question number | The real author, and the book the excerpt is taken from | Selected as Dickens | Selected as Bulwer-Lytton |
| 1 | Charles Dickens, "Great Expectations" | 42.5% | 57.5% |
| 2 | Edward Bulwer-Lytton, "Eugene Aram" | 50.5% | 49.5% |
| 3 | Charles Dickens, "Great Expectations" | 54.6% | 45.4% |
| 4 | Edward Bulwer-Lytton, "Eugene Aram" | 49.9% | 50.1% |
| 5 | Charles Dickens, "David Copperfield" | 50.7% | 49.3% |
| 6 | Edward Bulwer-Lytton, "Eugene Aram" | 50.1% | 49.9% |
| 7 | Charles Dickens, "Great Expectations" | 59.9% | 40.1% |
| 8 | Edward Bulwer-Lytton, "Paul Clifford" | 49.6% | 50.4% |
| 9 | Edward Bulwer-Lytton, "Eugene Aram" | 36.5% | 63.5% |
| 10 | Charles Dickens, "Great Expectations" | 40.6% | 59.4% |
| 11 | Charles Dickens, "David Copperfield" | 40.8% | 59.2% |
| 12 | Edward Bulwer-Lytton, "Paul Clifford" | 74.3% | 25.7% |
The performance of our quiz-takers is bad. But could this be because they don’t know English? The feedback demonstrates, however, that even educated people can't tell Dickens from Bulwer. One of quiz-takers wrote me "I got a 50%. My cat could do that well. The wine experts say a peek at the label is worth a thousand sips, and that seems to hold here. As a classicist I'm frequently called on to teach stuff I think is wretched, just because it's old." Some experts don't even dare to take the quiz. Prof. Scott Rice, the founder of the Bulwer-Lytton fiction contest, wrote to me "I haven't really taken it yet myself. Perhaps I am afraid to." We can even address the education issue in a scientific way. Fortunately, the quizzing script records taker’s IP address. From it one can infer where their computers were located. I selected a subset of scores, which were received by people coming from English-speaking (American, British, Australian, and New Zealandian) universities. The histogram of the scores received by 602 such people is shown in Figure 2. The average score is 5.76 or 48.0% correct. The standard error of this average is 0.095 or 0.8%. Educated English-speaking folks lost to the general public, whose average score is 48.2%. The difference between the scores is, however, statistically insignificant, because it is less than the standard error.
Figure 2. The histogram of the test scores earned by 602 people, coming from American, British, Australian, and New Zealandian universities. The average score is 5.76 or 48.0% correct. The standard error of this average is 0.095 or 0.8%.
But, perhaps, just knowing English is not enough? May be the beauty of Dickens’ prose is so far beyond the apprehensions of the vulgar that only the most cultured people can appreciate it? To check this I selected a subset of scores, earned by people coming from elite universities (Ivy League and Oxbridge). Table 2 contains the statistics of scores received by 76 of the chosen. The average score is 6 or 50% correct. The elite won over crowd by the whole 2%. The difference between the elite and general scores is, however, statistically insignificant. Due to the small size of the elite sample the standard error of the average elite score is 2.6%.
Table 2. Statistics of the elite (Ivy League and Oxbridge) scores on “Dickens or Bulwer-Lytton?” quiz. The average elite score is 6 or 50% correct. The standard error of this average is 0.3 or 2.6%.
| Elite School | number of respondents | minimum score | maximum score | average score |
| Brown University | 2 | 3 | 6 | 4.50 |
| Columbia University | 13 | 2 | 9 | 5.08 |
| Cornell University | 3 | 4 | 7 | 6.00 |
| Harvard University | 14 | 1 | 10 | 5.71 |
| Princeton University | 2 | 3 | 9 | 6.00 |
| University of Cambridge | 16 | 2 | 9 | 6.13 |
| University of Oxford | 10 | 3 | 10 | 6.30 |
| University of Pennsylvania | 7 | 2 | 11 | 7.71 |
| Yale University | 9 | 3 | 11 | 6.56 |
| Total | 76 | 1 | 11 | 6.00 |
The results of the quiz show that people can’t appreciate great prose when the name of a great writer is detached from it. The answer to the question, we started with, is: Yes, they have more readers.
January 16, 2006, 2006
Updated September 12, 2009
Technical issues
- When I looked at the quiz results I noticed hundreds of cases when two or more scores came from the same IP address within few minutes. In many of such cases the later score was 100%. This suggests that many people took several shots at the quiz. To eliminate this cheating I cleaned the data by selecting only the first score from each IP address. Afterwards I cleaned the data from the results, where one or more questions were skipped.
- I identified American, British, Australian, and New Zealandian universities by Internet domains: edu, ac.uk, edu.au, and ac.nz.
