Zombie meat beef jerkey
You can now purchase dried zombie meat at your local convenience store in Japan. The packaging claims it contains blue flesh aged to perfection in the graveyard.
via Pink Tentacle
I do want to that unless you have citations that show a RR of higher than 20, I'm not really interested in discussing the relative merits of diet change, weight loss, cholesterol control, heart disease control therapies, pharmacologies, etc. I just don't think findings that have a lower RR than 20 are really worth about, especially in a general and largely theoretical discussion like this. not epidemiologically sound and from my perspective are not worth considering. They're more like tilting at windmills or lashing out against the dark, and are tiny better than faith healing.
If using that BMJ editorial by Glasziou to justify this thinking, you've really misread his point — a point by the way, that many biostatisticians would disagree with. His point applies only to single cohort and case-series studies, not randomized trials. Moreover, he argues for a threshold of relative risks ranging from 5-10 to essentially take these studies as "definitive," not a threshold of 20.
Using an effect magnitude threshold to assess biomedical research is a very poor alternative for actually using good judgment in reviewing the methodology used, its limitations, and the potential for unaccounted for confounding that may exist. To justify doing this based on the notion of signal-to-noise makes no sense because the RR alone does tiny to capture "noise."
In fact, the conventional statistics (p-values, confidence intervals) are quite good metrics of precisely what might be described as signal-to-noise. Take the example of a one-sample t-test. The t-test Z is defined as the mean difference times the root of the sample size divided by the standard deviation of the difference. In other words, it is signal (mean difference or effect size) divided by noise (standard error). Most statistical approaches used are parallels of this. Even modern empirical approaches (bootstrapping, jackknife, permutation testing) provide statistics that are for all intents and purposes, a measure of signal-to-noise. If there is a demand for more critical vetting of study outcomes, a lower alpha level does a much better job of this than looking at the RR. RR is signal only — no accounting for sample size, no for variance. This makes no sense at all.
Consider the simple example of my case-series of two patients — a man and a woman. I followed them for a few weeks and the man had a heart attack while the woman didn't. Are you satisfied with the conclusion that men have a RR of infinity for heart-attacks relative to women? Of course not. So you state "well, I will look at the sample size. It sucks. I look at the sample size and the error statistics in making my judgment." Fine, but this is essentially captured in a rigorous quantitative way with the results of hypothesis tests typically presented in papers.
The magnitude of the relative risk is NOT a good reflection of a "real" effect despite or regardless of confounding. In other words effect size and confounding may be correlated on average, but they are poorly so. This is throwing out the baby with the bath water to the extreme. Consider the fact that an arbitrary high RR ignores the impact of baseline outcome incidence on potential results for a study. Not only is the example of a RR of > 10 or > 20 a bad idea, it's mathematically impossible in many circumstances. Take a disease in which 10% of patients consistently have spontaneous resolution and the other 90% die. If a drug for this condition yields a 90% cure rate, your true relative risk will be 9. Studies that sample populations treated and untreated with this agent will have a mathematical limit for the point estimate of RR that is always potential for confounding even after careful control of known confounders with appropriate statistical adjustments. Let me be clear though: establishing unequivocal causality in the absence of a randomization, sufficient sample size, and blinding is near impossible, regardless of the magnitude of effects reported in such studies. In this sense, I am more conservative than that BMJ article, because I don't assume truth or causality from any single study like that, regardless of the magnitude.
Lost in this is the fact that science is an empirical study — it may always be seeking the "truth" but it will never find it. Ideal evidence evolves over time. In the case of biomedical research, that evidence is an accumulation of the results from an aggregate of studies of varying quality, in the context of pre-clinical data that often includes animal models and in vitro data. A single study is never definitive — even RCTs. That doesn't mean the results are valueless and to be ignored, and it doesn't mean that authors' conclusions from a study are also taken on faith. It means you do the best you can with the information you have. To do so with biomedical research in good faith requires educating yourself about the methodology in the first place.
posted by drpynchon at 9:29 AM on May 13 [7 favorites]