How to Lie with Statistics

A recent email from a reader of this blog pointed me to a website I'd seen but forgotten about, in which the author attempted to prove that Patsy and only Patsy could have written the Ramsey "ransom" note by means of a statistical analysis. If you're curious, here's the URL: http://gemart.8m.com/ramsey/note

I had to laugh, because this person's use of statistics was clearly a travesty. It reminded me of the title of a book I'd heard about, published back in the 50's by Darrell Huff, entitled How to Lie with Statistics. Here's how his little game worked:


Our "expert statistician" based his analysis on only two examples of Patsy's writing, drawn from a comparison done by journalist Frank Coffman (who later became a staunch Ramsey defender). The first, is this one:

Note that the above contains all of eleven words, not counting Patsy's signature, in cursive and the block letters at the bottom, which could have been penned by someone else.

Here's the other:

 

Anyone directly comparing either of these documents to the ransom note would immediately notice the complete difference in style between the hand that wrote them and the hand that wrote the "ransom" note. But our author could care less about the look of Patsy's hand -- he's interested only in making a statistical analysis of how likely it is that both Patsy and the writer of the note would form their letters in certain ways.

His Exhibit One is intended to assess the likelihood that both Patsy and the note writer  would switch "back and forth between printed block letters and cursive script letters at random when printing." Well, this is odd, because the note contains very few examples of either block lettering or cursive, and the only example of cursive I can find in Patsy's samples is in her signature. And since most people write their signature in cursive, that would be expected, no? And I see no examples of block lettering by Patsy, aside from capital letters, which are usually written that way, and the bottom of the first example, which is clearly separate from the rest, so sorry but I see no examples of Patsy going "back and forth" between block and cursive, at least not in these documents. Nevertheless our statistician sees three different possible variants:

• Note written completely in cursive script style text.
• Note written in completely in printed block style text.
• Note written as shown in a combination of both styles.

And since, according to what he claims to have seen, he (falsely) concludes that both the note and Patsy's exemplars are in a combination of both styles. And since there are three possibilities, that means there is a 3 to 1 chance that the same person wrote all three documents:

There are only three possible variations the author of the ransom note could have used which are listed in the box above.  That gives us a one-in-three possibility or 3:1 odds if we were to guess which characteristic any person would use when writing an identical exemplar.

Yikes! Even if the similarities he sees were actually there, the notion that this sort of analysis gives us any sort of meaningful odds regarding the author of the note is completely unwarranted. For one thing, there is absolutely no control over the possibility of cherry picking, i.e., concentrating on the sorts of comparison that give you what you want to see and ignoring others that might give you the opposite.  For another, we have no way of knowing what the odds would be for anyone other than Patsy, because no other exemplars are compared, only hers. If we include the one example we have from John in our little game, we see that he combines block letters with manuscript letters, notably using a block letter "F" as the second letter in the word "oF". And we see something very similar in the ransom note, in the word "iF." And that's not the only place in the note where we find block "F" in the middle or end of a word. So by this reasoning, the same 1 in 3 odds also apply to John. But John is not considered in this comparison, only Patsy.

Moving on to Exhibit 2, our statistician writes: "Patsy Ramsey connects the cross stroke on the letter "t" to the following letter "e" she prints." And we are provided with examples from the ransom note in which the letter "t" is also followed by the letter "e" in a supposedly similar manner. The problem here is that the horizontal in Patsy's "t" continues smoothly to connect with the following "e" in a single stroke. Whereas if we look carefully at the note, we see that in almost every case, the cross stroke on the "t" simply abuts the cross stroke in the letter "e" -- there is no continuity between the two strokes, they just touch one another.

Now if you look really carefully at the note you'll be able to find a few examples where the two letters are connected by a single stroke, but what can that mean, statistically speaking? And here we must consider a problem completely ignored by our statistician. What can it mean if, say, Patsy connects the letters 90% of the time and the note writer connects them only 5% of the time? Does that count as a similarity or a difference? Statistically speaking, it would certainly count as a difference, and a significant one at that. But our analyst has made no effort to assess the relative frequency of any given trait, he is simply cherry picking what suits him. If he can find one single example of something that appears to be similar in some way, that's good enough for him.

Exhibit 3 is concerned with the difference between manuscript "a" and cursive "a". And here again the claim is that both the note writer and Patsy switch back and forth between the two. Sorry, Charley, but statistically speaking you have the same problem here as above. Patsy does indeed have a tendency to switch back and forth between these two forms of the letter "a," though that's more apparent in her London Letter than the examples seen above. However, the great majority of "a"s in the note are manuscript, with only a very few written in cursive. So again, statistically speaking, there is a huge difference between the two, NOT a similarity. Seeing that Patsy tends to use both forms and then searching to see if you can find both in the ransom note, even if one type is relatively rare, sorry but that's cherry picking, NOT statistics. And if in fact both forms were not found in the note, then that difference would be ignored, wouldn't it? Well, it wouldn't be ignored in a truly scientific comparison, where differences need to be given exactly the same weight as similarities. But clearly our "statistician" has no interest in spotting differences.

I don't really need to go on, but I will, because this is kinda fun. Exhibit 4 concerns the presence of the "straight letter "y" descender." "Patsy Ramsey forms the letter "y" with a u-shaped arc to the right followed by a straight line downward." And so, apparently, does the writer of the ransom note. Since he finds 6 different possible ways of forming that letter, that gives him 6 to 1 odds in favor of the note being written by Patsy.

Well, if you look carefully at the note you'll find that it contains at least two types of letter "y," one with a u-shaped arc at the top and one with a triangular shaped "v" at the top. We also see both straight line vertical descenders and descenders slanting down to the left. And lo and behold, we find both types in John's sample, the triangular, left-slanting type in the name "Ramsey" and the u-shaped, vertical descender type in the word "satisfactorily." And if you don't believe me, take a look for yourself:

So, according to the method pioneered by our intrepid statistician, the odds in this case are far more with John than Patsy, because, after all, how many times would you find BOTH ways of forming both elements in the letter "y" in the writings of any two people? Unfortunately, however, our "expert" is focused only on Patsy, and only on whatever he can find in Patsy's exemplars that might have some similarity with what he can find in the note.

In the next exhibit, we are informed that both the note writer and Patsy like to write the letters R and D with an overhang at the upper left. But once again we find the same trait in John's writing.

I could go on, but at this point I'm getting bored. As should be clear, this is not really a statistical analysis at all, but in fact a travesty of both statistics and scientific method. I'm not saying this person was deliberately "lying with statistics," because that's probably not the case. He simply has no understanding of how to actually use statistics scientifically, with some measure of control, self criticism and humility. Lacking any such skill, he comically comes up with a "One in 55 million" chance that Patsy didn't pen that note. Yeah, right.