An overhead shot of a plate with a neatly arranged meal of beef steak, asparagus, potatoes and a drizzle of balsamic vinegar.

The researchers created a star-based scale that ranks the quality of evidence for a link between a particular behavior — such as eating red meat — and certain health outcomes.Image Credit: Education Images / Universal Images Group / Getty

Does eating red meat reduce life? Some researchers certainly think so. Work such as the Global Burden of Disease, Injury and Risk Factors Study1 He led the World Health Organization and the US Department of Agriculture to advise that people limit consumption of unprocessed red meat, to protect themselves from diseases such as type 2 diabetes and various types of cancer.

Other researchers are less confident. Red meat consumption goals, set by public health officials and expert committees, vary widely, with some advising people not to eat more than 14 grams per day and others not stating the recommended limit. This sends a confusing message, and it is in and of itself not good for overall health.

And it’s not just red meat: The evidence base surrounding much of the nutritional and health advice is similarly disputed. Now, a new approach could help health policy makers better assess the quality of studies that assess potential health risks. A team at the Institute for Health Metrics and Evaluation (IHME) at the University of Washington in Seattle has created a star-based scale that ranks the quality of evidence for a link between specific behavior — such as eating red meat or smoking — and certain health outcomes.2. A five-star score means that the link is clearly defined; One asterisk means that either there is no association between the two factors or the evidence is too weak for a definite conclusion.

What researchers call the “burden of proof” analysis does not, in and of itself, remove troubling issues such as the risks of red meat or the benefits of vegetables. But as a judge of the quality of the available research, it can help identify research funders, areas that require better evidence for more consistent conclusions.

How is the star rating built? What are its criteria – and can the methodology itself be considered rigorous research? The IHME team has done many things to try to determine the effects of different biases in the studies being evaluated. An epidemiological study, for example, may be biased in different ways to a study that tests the outcomes of health interventions. The researchers also eliminated what could be a common source of bias in the research, which is the assumption that health risks increase exponentially with the parameter under study, for example blood pressure or consumption of unprocessed red meat. They tried to explain the bias that can arise when sample sizes are small.

Applying this framework to studies that evaluated a total of 180 questions produced mostly unsurprising results. Studies that evaluate the relationship between smoking and different types of cancer, for example, receive a five-star rating3. Likewise, high systolic blood pressure–the force the heart exerts to pump blood–has a five-star association with narrowing of the blood vessels called ischemic heart disease.4.

Studies evaluating diet and its health outcomes receive significantly lower star ratings. For example, the IHME analysis found only weak evidence of an association between eating unprocessed red meat and outcomes such as colorectal cancer, type 2 diabetes, and ischemic heart disease.5. It found no relationship in studies exploring whether eating unprocessed red meat leads to two types of stroke. There is stronger, but not conclusive, evidence that eating vegetables reduces the risk of stroke and ischemic heart disease.6.

In some cases, the lower star rating may be due to the size of the effect: for example, any health risks from consuming red meat are likely to be small compared to the enormous toll that smoking inflicts on the body. Above all, the low-ranked results show that studies in these areas need improvement if they are to yield convincing results.

It is difficult to derive the effect of a single food component from the effect of a complex variety of exposures over a person’s lifetime. You’ll need larger studies, with a diverse group of participants and strict control of their daily diet. Such studies will necessitate collaboration between research groups with different expertise, and access to participants in different environmental settings – a move that funders should encourage. This is an undertaking that deserves priority. A small individual risk does not imply a small impact on public health: low-risk behavior can have a significant impact at the population level if it is very common.

The literature on responsible research and innovation highlights how metrics in science must always be questioned for robustness and rigor. There should be broad consultation and, as far as possible, the unintended consequences of using metrics, such as initiatives such as the San Francisco Declaration on Research Evaluation and the Leiden Statement, should be anticipated. This work has to come sooner rather than later.

We have evidence that weak clinical studies, which lack the controls needed to understand the data, are not helping. If funders do not target their efforts to produce high-quality data, the public will remain confused, stressed, distrustful and deprived of the information they need to make healthy, informed lifestyle choices.

Leave a Reply