Selection Bias: before & after Photos
Full disclosure: I stole this collage from the internet, I don’t know any of these people.
I’ve been asked often why I don’t do many “before and after” profiles of clients. I suppose it is an odd choice because that type of marketing can be a powerful advertisement for potential clients. I mean, who doesn’t love looking at dramatic transformations? I personally have spent hours reading various forums profiling the pictures and stories of folks who have made drastic body composition changes in all phases of life - old, young, post-partum, post-athletic career, whatever - it’s always inspirational.
So then why eschew it with my clients? Well, it’s actually rather simple, and it comes from trying to eliminate bias. In this case, it’s an example of selection bias, or, more accurately, attrition or survivorship bias.
A classic example of survivorship bias is the stock-picking letter scam. You send 10,000 people a letter, half of which predict Stock ABC will go up, half of which say Stock ABC will go down. After you see the results, you discard the folks who got the erroneous letter. You then divide the “correct” 5,000 folks into two groups of 2,500, and repeat. After 10 iterations of this, and ignoring rounding, you are left with 20 people who have seen you accurately assess Stock ABC correctly 10 months in a row! Take my money!
Another example, from Nassim Taleb in The Black Swan, describes a story originally told by Marcus Tullius Cicero:
[…a nonbeliever in the gods was shown painted tablets bearing the portraits of some worshippers who prayed, then survived a subsequent shipwreck. The implication was that praying protects you from drowning. [He] asked, ‘Where were the pictures of the people who prayed, then drowned?’
The drowned worshippers, being dead, would have a lot of trouble advertising their experiences from the bottom of the sea. This can fool the casual observer into believing in miracles.
And no description of survivorship bias is complete without the most famous example: Arthur Wald, bullet holes, and WWII.
If you are reading a text and you see this photo (esp in the prob & stats world), be sure the author is about to invoke survivorship bias to explain a phenomenon. It’s ubiquitous.
Simply put, survivorship bias is the tendency to view existing data as a comprehensive sample without considering the data that failed, did not survive, or was ignored.
So, as the case is sometimes when I write, this was a long road to a short path.
My point is, traditional “before and after” photos are always going to be dramatic, because if they were not, no gym would use them. As if, “Here is my client, Ted, who lost a ¼ inch off his left forearm, and shaved 2.5 seconds off his 10K time.”
I mean, I’ve had plenty of clients who have made dramatic progress. On balance, I think clients get some pretty great results at ATC, which is reflected in, if I might brag, a very high retention rate. Of course, I’ve also had clients who were a terrible fit for our model. I’ve had clients who have gotten plenty strong, but made no dietary changes. And of course I’ve had some clients who seemingly go out of their way to do literally nothing I ask of them (fortunately very few). Those clients don’t get amazing results.
So, let’s be honest. If you are going to use before and after photos, use them for every person who has ever signed up. Sure, clients who have had wildly successful transformations. But also clients who quit after a few weeks. Clients who sustained major injuries. Clients who dutifully show up everyday, but never seem to get better. Let’s see the whole population, not just the survivors.
(Author’s Note: While all of this is true, there is also just a notable amount of abject administrative laziness that prevents this, too.)
