Step 1: Create an iron-clad researcher contract
Nobody’s 100% unbiased, because we’re all human and come at things with our own experiences and knowledge. But at Examine, we try our darndest to get close to the “no bias” asymptote.
The first step is contractual. Examine contractually mandates that all staff avoid any and all connections to supplement, food, or health companies.
Here’s just one portion of the legalese directly from our researcher contract, condensed down for readability:
The Contractor hereby confirms that he or she is not presently engaged, either directly or indirectly, with a person or company involved in the business of health supplements or health-intervention programs … The Contractor agrees to notify the Company immediately if, at any time, and for any reason, Contractor becomes engaged, either directly or indirectly, with a person or company involved in the business of health supplements or health-intervention programs …
We take this very seriously, and have no researchers with any investments, partnerships, contracts, or any other relationship with health-related companies that might bias their research for us. Examine staff are also prohibited from receiving gifts or even free samples from health-related companies.
Note that Examine doesn’t hire freelance writers like other big health websites do. Most of our staff are full-time researchers working exclusively for Examine. We pay everyone fairly and create a low stress working environment to encourage researchers to stay for many, many years.
By having a close-knit, long-tenured team, we avoid having to worry about temporary writers who may have hidden biases which otherwise take months to emerge.
Step 2: Encourage “Well, actually …” discussions
Ever since childhood, I’ve been that slightly annoying guy who won’t let inaccurate statements go. But at Examine, I feel comfortable knowing that I’m surrounded with many other “Well, actually” people!
We point out each other’s inaccuracies whenever they pop up, without hesitation. Here are just a couple real-life examples from our internal conversations on Slack:
Person 1: Posts an article on drug composition (racemic mixtures) they thought was cool and interesting.
Person 2: “Well, actually ... is the article you posted accurate or misleading? This meta-analysis says that 20 mg of nexium outperformed 40 mg of prilosec for symptom relief.”
Person 1: Lists a protein intake number supported by a famous study.
Person 2: “I've read some compelling explanations for why that meta-regression might have inadequate data points and could change with further research.”
Now, this doesn’t always go smoothly. Sometimes we have to decide how far to take an internal debate, because they can chew up hours and hours without having much real-world payoff.
And because we’re humans, very, very occasionally feelings can be hurt. Even more rarely, when one researcher “well, actuallys” another researcher but ends up being wrong themselves, that can get awkward. Yet Examine remains more open to internal debate and correction than literally any group of people I’ve ever been around.
Step 3: Assign researchers a broad variety of topics
Examine researchers don’t just research topics they’re interested in. In fact, sometimes they’re asked to research topics that don’t jive with their personal diet and supplement regimens.
For example, if a researcher eats a ketogenic diet, they have to be equally adept at analyzing vegan diet studies without exhibiting bias. And if a researcher eats a vegan diet, they have to be equally adept at analyzing ketogenic diet studies.
We have researchers who eat all kinds of diets and have all kinds of viewpoints on supplements. Their common denominator is open-mindedness and curiosity, avoiding at all costs the assumption that their personal experience (or their patients’ experience) is the universal truth.
As individual researchers, we’re fallible and limited by virtue of having just one single brain. But as a team, we check each other, teach each other, and assemble as a group that’s much stronger than its parts.
Step 4: Maintain an (overly?) extensive review process
Most big health websites list reviewers in their articles, but often the review process takes all of half an hour.
Quick reviews are financially prudent, because readers don’t usually have access to the full text of studies, and can’t easily fact check. Plus, the vast majority of health articles on the web are rarely updated, so it’s a one-time review rather than an ongoing one.
We review our writing and research way more than is financially prudent. We have four highly-skilled copyeditors, including a specialized medical editor, to make sure our writing clearly communicates the key concepts without being unintentionally misleading. Then, an internal reviewer (and sometimes external reviewer) go line by line and comment on analysis that may not be interpreted correctly or topics that need more explanation.
The end result is that researchers simply can’t get away with being biased. We hire researchers and reviewers from a wide variety of backgrounds (dietitians, PhD researchers, pharmacists, medical doctors, etc.), which makes it a quick process to root out any bias that can affect the veracity of our writing.
I hope this peek behind the curtain was at least a little bit interesting. There’s a lot more we’re working on behind the scenes to improve bias minimization, such as revamping our backend system to allow easier collection of study funding information and cleaner data extraction from meta-analyses.