What most people get wrong about observational studies
Whenever a giant observational study is published, members of both the media and the public react predictably. Some will automatically assume a cause-and-effect relationship between the variables being studied, while others will offhandedly dismiss the findings on the basis that correlations don’t imply causation.
Yet correlations can imply causation when you’ve ruled out other correlations and figured out which event precedes the other. Take smoking: it would be both impractical and unethical to set up an intervention that would randomize healthy participants between a smoking group and a non-smoking group to see whether or not smoking causes cancer. Yet we are confident that it does, thanks to high quality epidemiological and mechanistic studies that show very strong correlations as well as biological plausibility.
This is why it’s never a good idea to immediately dismiss observational studies. Not only can they help generate hypotheses, they can also, under the right conditions, test them, especially when followed either by mechanistic studies or by short-term RCTs with surrogate markers (if ethical). In order to produce such causal inferences, however, the data have to be strong. The possibility that they are is why these studies are worth examining.
Correlations don’t always imply causation, but they can do so if the data are strong enough and if there’s additional evidence to suggest a causal relationship.
Who and what was studied?
Two weeks ago, a group of researchers published a series of papers on the relationship between macronutrient intake and certain health outcomes. The study that caused the most controversy investigated the associations between intake of macronutrients (carbohydrates, proteins, and fats) and two outcomes: cardiovascular disease and mortality. Titled the Prospective Urban Rural Epidemiology (PURE) study, it followed a cohort of 135,335 individuals aged between 35 and 70 living in 18 low-income, middle-income, and high-income countries on five continents. The cohort was weighted more toward middle-income and low-income countries — which is important because those areas are typically understudied. The great diversity of its cohort made this study one of the first of its kind.
Using standardized questionnaires, the authors collected demographic data, such as lifestyle data, socioeconomic status, and health and medication history. Using a food frequency questionnaire, they also collected dietary data once at baseline.
The authors looked at two primary outcomes (total mortality and major cardiovascular events) and four secondary outcomes (heart attacks, strokes, cardiovascular disease mortality, and noncardiovascular disease mortality). They collected those data by tracking the participants’ physician diagnoses, abnormal biomarkers, hospitalizations, and autopsies. The participants’ median follow-up time was 7.4 years.
The PURE study is an epidemiological cohort study. For about seven years, it followed 135,335 individuals aged between 35 and 70 living in 18 low-income, middle-income, and high-income countries. The cohort was weighted toward middle-income and low-income countries. The purpose of the study was to investigate the associations between macronutrient intake and two outcomes: cardiovascular disease and mortality.
What were the results?
The authors organized macronutrient intake into quintiles. The main sources of carbohydrates were white bread, white rice, cakes, fruits, fruit juices, and sugary drinks — i.e., mostly refined foods poor in nutrients. After adjusting for ten confounders (energy intake, physical activity, waist-to-hip ratio, smoking, diabetes, age, sex, education, socioeconomic status, and urban or rural location), the researchers found a statistically significant positive association between carbohydrate intake and total mortality. In general, the higher the carbohydrate intake, the higher the rates of all-cause mortality and non-cardiovascular disease mortality, at any point in time. Those trends were found in both Asian and non-Asian regions, but were significant only in non-Asian regions.
However, positive associations were not observed between carbohydrate intake and four outcomes: strokes, heart attacks, major cardiovascular diseases, and cardiovascular disease mortality.
Total fat intake was also divided into quintiles, and the researchers found a statistically significant inverse association between fat intake and total mortality. In other words, the higher the fat intake, the lower the risk of dying at any point in time. This trend held up for noncardiovascular disease mortality and for strokes, but not for heart attacks, major cardiovascular diseases, and cardiovascular disease mortality.
In both Asian and non-Asian regions, higher intakes of monounsaturated fatty acids were inversely associated with total mortality, whereas higher intakes of polyunsaturated fatty acids were inversely associated with total mortality only in Asian regions.
Those fat intake associations were all significant.
The authors also assessed the effect of replacing carbohydrates with certain classes of fat. Replacing carbohydrates with polyunsaturated fats was associated with a lower risk of total mortality and noncardiovascular disease mortality, whereas replacing carbohydrates with saturated fat was associated with a lower risk of stroke.
After adjusting for several covariates, the authors found a clear association between carbohydrate intake and two outcomes (total mortality and noncardiovascular disease mortality) but not between carbohydrate intake and four other outcomes (strokes, heart attacks, major cardiovascular disease, and cardiovascular disease mortality). Total fat intake was inversely associated with total mortality, noncardiovascular disease mortality, and strokes, but not with heart attacks, major cardiovascular diseases, and cardiovascular disease mortality.
What does this study really tell us?
These results seem quite scary at first — increased risk of death with higher intakes of carbohydrates? That doesn’t sound too good. But it’s important to note some of the limitations of the study before drawing any conclusions.
After adjusting for many confounders in their models, the researchers found a clear positive association between higher carbohydrate intake and, at any point in time, both total mortality risk and noncardiovascular disease risk.
Confounders are something to always keep in mind, even when corrections are made, because several unmeasured or unknown variables may affect the outcomes. The best way to deal with confounders is through randomization, allowing inference of a cause-and-effect relationship. You can’t do this in observational studies like the PURE study, but you can somewhat account for confounders by making adjustments. This method has its drawbacks, however: you can’t adjust for unknown confounders; overadjustment can sometimes produce confounding effects; and the more analyses you run, the more likely you are to obtain a false positive. So it is possible that some of the results could be statistical flukes after adjustment.
No positive association was found, however, between higher intakes of carbohydrates and either strokes, heart attacks, major cardiovascular disease, or cardiovascular disease mortality — which is important because outcomes like those are a lot more specific than total mortality risk or noncardiovascular disease risk. We must therefore consider two possibilities: First, it is possible there really was no dose-response relationship, so no causal relationship between carbohydrate intake and any of those specific outcomes, since dose-response is usually a hallmark of causality. Second, it is possible there was a dose-response relationship, but one the small sample size made difficult to detect, since much fewer people experienced those outcomes than people died (total mortality and noncardiovascular disease mortality).
However, there were clear associations between carbohydrate intake and both total mortality and noncardiovascular disease risk. Is there some merit to these associations? Maybe. Or maybe these outcomes have another cause. Many of the countries included in the analyses consumed most of their carbohydrates from highly refined foods poor in nutrients. For example, in Bangladesh, most folks consume the majority of their calories from white rice. Poor nutritional status could be contributing to mortality along with the lack of access to certain resources. Even though the authors used four different data (education, household wealth, household income, and income level of the country) to adjust their models for socioeconomic status (SES), their global variables may not have captured all the risks of increased mortality that typically go along with SES — which is something they acknowledge in the paper.
But what about fat? Does this study suggest that fat is protective? Unlike carbohydrates, fats weren’t all lumped together by the authors, who found that replacing a small amount of carbohydrates with polyunsaturated fats reduced the risk of total mortality and noncardiovascular disease mortality. Typical sources of these fats include fish, nuts, and vegetable oils. These foods are far richer in nutrients than refined sources of carbohydrates. They’re also far less available in poor areas. Which raises the question: Is a higher intake of polyunsaturated fats protective? Or is it simply associated with better nutritional status and/or unmeasured markers of SES?
The analyses performed by the authors of this study also make it a bit difficult to draw any strong conclusions at the individual or nation-wide level: data from 18 countries were combined to produce these results, and such a vast sample size is a double-edged sword. On the one hand, it facilitates the detection of real effects, and the varied population allows for generalizations. On the other hand, by combining data from people who live very different lifestyles, have access to different sets of resources, and vary in genetic characteristics, the researchers made it harder to determine if those general results apply to specific subpopulations. In other words, any conclusions drawn from their analyses will hold up better for the general world population than for a specific population or an individual.
Now, one must remember that this is all speculation based on the limitations of this study and of epidemiological research in general. A lot of other potential factors could be contributing to the trends we see in the data, but still, the data do show an association between high carbohydrate intake and both total mortality and noncardiovascular mortality. And it’s actually quite impressive that the trends remain after adjusting for so many covariates. So, despite the effect sizes from this study not being of the same magnitude as those from epidemiological studies that help infer causal relationships, they’re still noteworthy.
The study’s results are limited by the very design of the study. The results may be affected by unaccounted-for confounders, such as unmeasured markers of socioeconomic status. In other words, poor nutritional status and a lack of access to medical resources may account for the high mortality rates. However, the results of this study shouldn’t be discounted offhand: despite many adjustments, the associations between carbohydrate intake and mortality remained, which makes them noteworthy. They may just not be as useful for personalized medicine as they are for global policy.
The Big Picture
Despite this study’s methodological limitations, its results are worth following up on. A lot of possible factors could be examined. Poor nutritional status could be a reason why high carbohydrate intake was associated with mortality, and since the PURE study actually collected blood samples for a separate paper, those may be worth examining for micronutrient deficiencies. As mentioned before, conducting a randomized controlled trial (RCT) of this scale or length would be fairly difficult. However, a short-term, well-designed, statistically powered RCT with surrogate markers could corroborate the results of this epidemiological study.
The authors claimed that the results of their study do not support the World Health Organization’s current dietary guidelines, which limit total fat intake to 30% and saturated fat intake to 10%. Are the results of this particular study strong enough to reform the dietary guidelines as suggested by the authors? Probably not by themselves. However, they do add to a plethora of studies published in the last few decades suggesting that saturated fat may not be as harmful as originally thought, and that refined carbohydrates are not healthy for a number of reasons. So, between the PURE study and other emerging evidence, lifting the cap on fats and placing one on carbohydrates may be something worth considering when updating dietary guidelines.
The results of this study are worth following up on. They could be coroborated by randomized controlled trials or by examining different factors, such as micronutrient deficiencies. By themselves, they may not be strong enough to reform the prevalent dietary guidelines, but combined with other evidence, they may contribute to future guidelines raising their caps on fats and lowering their caps on carbohydrates.