Sunday, August 25, 2013

Research Chicanery and IBD

Scientific Cooking (Part 1)


This posting is not about the type of cooking that’s done on a stove, but the type that’s done in a laboratory (or more frequently, on a computer with lab results).  There is much cooking that can be done to make raw research look a particular way.  All scientists cook – papers don’t get published with just a table of statistics – but whether they cook their results in a straightforward manner that is easily reproduced and logical or they take dress their results to appear to be something different is the difference between legitimate research and misrepresentation.

Facts are misrepresented for many reasons – intent cannot be determined from a paper in isolation.  The reasons range from accident (setting up a measurement wrong or miscalculating) to ignorance (inadvertently biasing results) to willful ignorance (trying to slant the results) to outright fraud (falsifying results).  Even with the gold standard in evidence based IBD research – the double blind study – doesn’t completely protect against misrepresentation.  At one end, investigator bias can creep into both protocols and interpretations.  At the other, outright alteration of the aggregate results can occur.  This post deals with some of the inadvertent bias that can creep into the results of honest researchers, and provides some things to look for when evaluating a research paper.

Before getting started, an understanding of p values is necessary.  The p value is the probability of obtaining a result that is the same (or better) than what was observed, assuming the null hypothesis is true.  In layman’s terms, the p value is used to determine if a result is significant or not.  If the p value is lower than a particular threshold, it is assumed that the result is significant.  The most common thresholds used are .05 and .01, which correspond (kind of*) to a 1 in 20 or a 1 in 100 chance that the result is random.  The p value has been often criticized, citing the random choice of a “significance threshold” of .05 and the ability to manipulate the results.(1)

Disraeli is quoted as saying “There are three kinds of lies:  lies, damned lies, and statistics”**.  This is true in medical research as it is elsewhere.  Aside from a dishonest researcher, how can an honest researcher fall victim to statistical traps?

Framing


Framing involves running an experiment in different sets that are advantageous to the researcher.  Think of the difference between these two experimental designs:

1.       Flip 5 quarters 20 times each to determine if it always land on heads.
2.       Flip 20 quarters 5 times each to determine if any of the coins always land on heads.

At first glance, both look the same.  If my hypothesis is that a quarter will always land on “heads”, both experiments have 100 data points.  The second, however, is really a repetition of the same experiment ten times.  In the first experiment, the probability of getting all heads is on an individual set is 1/(2^20), or 1 in 1024.  The probability of getting all heads in any set is approximately The probability of getting all heads on any one set is approximately one in 200.  In the second, the probability of an individual set getting all heads is 1/(2^5), or 1 in 32.  Still a fairly unlikely occurrence.  However, the probability of any of the 10 sets landing all heads is slightly better than 1 in 2 – a much more likely occurrence.  Even though both experiments rely on 100 coin flips, but changing the framing of the flips a research can affect the likelihood of getting a set with all heads.

Discarding Data


While an honest researcher would not intentionally remove data that does not fit the expected results, they may choose to discard certain data points.  It is easy for a researcher, after the fact, to remove specific patients from the results and find a reason to do so.  One of the most common reasons is dropout – when an individual drops out of an experiment, the researcher must determine how to count that individual.  They have a few options:

1.       They can drop the individual completely as though they were never part of the experiment.
2.       They can use the last known datapoint for that individual an extrapolate a result.
3.       They can treat the dropouts as a separate category.
4.       The dropouts can be counted using their last datapoint without extrapolation.

Because there are multiple options, a researcher can choose the one that is most beneficial if the choice is made after the event.  A better practice is to define what to do with dropouts before running the experiment.
Similarly, a researcher may discard data that disagrees with a hypothesis by mentally convincing themselves there was a protocol error.  Consider this scenario:

1.       The first clinical trial doctor finds that there were adverse events that occurred in 1 out of their 20 patients.
2.       The second clinical trial doctor finds that there were adverse events that occurred in 2 out of their 20 patients.
3.       The third clinical trial doctor finds that there were adverse events that occurred in none of their 20 patients.
4.       The fourth clinical trial doctor finds that there were adverse events that occurred in 10 out of their 20 patients.

Many researchers, instead of concluding that adverse events occurred in 13 out of 80 patients, will try to look at protocol errors in the fourth dataset in attempt to throw it out and conclude that 3 of 60 patients had adverse events occur.  This is a bias in itself – the researcher should look for protocol errors in either all of the experiments or none of the experiments.  Even worse, the researcher may throw out the whole trial and run it again (and again and again) until they get the “expected” results.

High Dimensionality Experiments


When researchers are touting the benefits of a wonder cure that has little prior probability of working, you will frequently see them go on a fishing expedition by measuring dozens of different positive outcomes.  The researchers will then cherry pick the outcome that supports their hypothesis (that the wonder cure works).  A scenario may go something like this:

1.        The research hypothesizes that taking a particular herb will have a positive impact on Crohn’s disease.
2.       The herb will be given to half of 50 patients in each of two blinded groups.
3.       The researcher will conduct blood tests, physical tests, and self-reporting in 100 different areas.

In the above scenario, the researchers will frequently use the p value inappropriately.  If the control group shows that 21 of the 25 patients showed improved CRP levels, they may conclude that, based on p value, this is a significant finding.  Unfortunately, when multiple dimensions are measured, the p value needs to be adjusted for the number of measurement categories.  Choose enough categories and what looked like a significant effect is really just random noise.  If enough measurements are taken, any group of individuals will improve on average in half and decline on average in half.  Most of those will not be statistically significant, but with enough measurement buckets and no correction on p values, eventually one of them will appear to be significant, and you can bet the researcher will tout that as a success for their treatment.  The even more dishonest researchers will omit the fact that there were other measurements taken and discarded.

Determining Measurements After the Fact


Similar to the issues with high dimensionality experimentation, there can be ex post facto issues with selecting measurements.  In an ideal experiment, the specific things to be measured and the success criteria are defined beforehand to avoid bias.  Unfortunately, some researchers wait until after the experiment to determine the measurements.  Consider the following dataset looking at C Reactive Protein, a frequent measurement of inflammation (a CRP > 3 mg/L is considered High):

CONTROL
Patient
CRP Before
CRP After
A
5
6
B
4
2
C
2
2
D
3
4
E
5
5

TEST
Patient
CRP Before
CRP After
F
5
5
G
4
5
H
3
2
I
3
2
J
5
6

The average CRP for each group remained completely unchanged (3.8 for the first and 4.0 for the second).  Consider how the following true statements can be used, however, to slant the results and make it look like the TEST protocol was effective:

·         Twice as many individuals using TEST dropped from a High CRP level to a moderate CRP level.
·         CRP was lowered in twice as many patients in the TEST group
·         40% of the TEST group showed improvement, compared to 20% of the CONTROL group

While all of the above are true, they are misleading and don’t really show there was actual efficacy.  The numbers can work the other way as well.  Consider the following:

CONTROL
Patient
CRP Before
CRP After
A
5
5
B
4
2
C
2
1
D
3
5
E
5
5

TEST
Patient
CRP Before
CRP After
F
5
0
G
4
5
H
3
4
I
3
3
J
5
6

Like the first experiment, the results don’t really show anything.  The Test group fell from 4 to 3.6, and the CONTROL group fell from 3.8 to 3.6.  However, consider the following would be true statements:

·         The TEST group’s average CRP fell twice as much as the CONTROL group
·         The TEST group showed one patient completely cured of inflammation, compared to no one in the control group

Part 2 will cover other ways that researchers cook the books statistically speaking to make their research appear more substantial than it really is.

Bottom Line


·         There are many ways to make research look statistically sound, even though the underlying protocols are flawed
·         Research making wild claims with small a priori probability should be viewed through a skeptical lens

* Technically the p value is only a measure of the evidence against the null hypothesis.  In science based medicine, the prior probability of the event occurring needs to be taken into account, in a more Bayesian approach.  That said, the p value is a quick and dirty way to do a base check.
** It was actually Mark Twain quoting Disraeli – there is quite a bit of doubt as to whether or not Disraeli actually uttered the words.

1.       Sellke, Thomas; Bayarri, M. J.; Berger, James O. (2001). "Calibration of p Values for Testing Precise Null Hypotheses". The American Statistician 55 (1): 62–71. doi:10.1198/000313001300339950.


Sunday, August 18, 2013

Feel the Heat

Heatwaves and IBD


A recent study posted in the American Journal of Gastroenterology looked at the correlation between heat waves and the hospitalizations due to both inflammatory bowel disease (IBD) and infectious gastroenteritis (IG).  The study found:

The presence of a heat wave increased the risk of IBD flares by 4.6% (95% confidence interval (CI): 1.6–7.4%, P=0.0035) and of IG flares by 4.7% (95% CI: 1.8–7.4%, P=0.0020) for every additional day within a heat wave period. In the control group there was no significant effect (95% CI: −6.2–2.9%, P=0.53). Screening of alternative forms for the effect of heat waves suggested that for IG the effect is strongest when lagged by 7 days (risk increase per day: 7.2%, 95% CI: 4.6–9.7%, P<0.0001), whereas for IBD no such transformation was required. (1)

The study looked at Zurich-based hospital admission on 738 IBD and 768 IG patients and 506 other chronic, non-infectious intestinal inflammations as a control group.  In press releases, the authors were quoted as saying:

The evidence of patients with IBD having a significant increase risk of flare ups compared to the control group shows a cause and effect between the climate and the disease," said lead author Christine N. Manser, MD. "This study ties heat stress to digestive symptoms supporting the observed seasonal variation in the clinical course of inflammatory bowel disease and suggests that microbial infections of the gut might be additionally influenced by climate changes.(2)

Before jumping to wild conclusions about stress and bacterial growth due to heat, there are a few issues to consider.  First, the study is retrospective and not prospective (I’m hoping the PR piece about showing cause and effect is the reporter or press person misquoting the author – this was a retrospective study).  Second, the control group’s p value is not very encouraging for drawing the baseline against which the study group was measured.

The most obvious mechanism isn’t given much press in the reporting – simple dehydration.   Those with IBD are already at a higher risk for dehydration-related issues, owing to poor absorption and chronic diarrhea.  Additionally, other studies have shown that everything from renal disease admissions (3) to general admissions increase during a heat wave.(4) 

Keeping hydrated (and keeping the electrolyte balance in check) is very important for anyone, and especially so for those with IBD.  Additional studies quantifying the risk (especially the aggregate risk the paper seems to show of multiple days of heat) due to exposure and the related preventative measures and their impact can help activity planning for Ulcerative Colitis and Crohn’s patients.  This study was a good initial study and will hopefully encourage additional work, but linking increased IBD admissions to climate change (which is a long term effect, not due to a few days of heat) is too far reaching, and the correlational data in the paper above is interesting, but may have been overplayed in the press. 

Bottom Line


·         Dehydration and electrolyte imbalance can affect those with inflammatory bowel disease, especially with prolonged exposure to hot weather.
·         Keeping cool and ensuring the intake of a proper volume and type of fluid are important.

--------------------------------------------------------------------------------------------------------------------------------------

1.     Manser, Christine N., Michaela Paul, Gerhard Rogler, Leonhard Held, and Thomas Frei. "Heat Waves, Incidence of Infectious Gastroenteritis, and Relapse Rates of Inflammatory Bowel Disease: A Retrospective Controlled Observational Study." The American Journal of Gastroenterology (2013).
3.      Hansen, Alana L., Peng Bi, Philip Ryan, Monika Nitschke, Dino Pisaniello, and Graeme Tucker. "The effect of heat waves on hospital admissions for renal disease in a temperate city of Australia." International journal of epidemiology37, no. 6 (2008): 1359-1365.

4.      Knowlton, Kim, Miriam Rotkin-Ellman, Galatea King, Helene G. Margolis, Daniel Smith, Gina Solomon, Roger Trent, and Paul English. "The 2006 California heat wave: impacts on hospitalizations and emergency department visits." Environmental health perspectives 117, no. 1 (2009): 61.

Sunday, August 11, 2013

Homeopathic IBD Treatments

Homeopathy and Watered Down Treatment Options For IBD


Homeopathy was invented by Samuel Christian Hahnemann in the late 1700’s and became popular toward the end of the 18th and the beginning of the 19th centuries.  Homeopathy was based on Hahnemann’s Principle of Similars.  Hahnemann believed that treatments that caused symptoms similar to those of a disease would also cure that disease.  Second, homeopathic treatments are diluted with water or alcohol to extract their “vital essence”, with dilutions occurring on the “C” scale from 1C to 100C.  Each “C” represents a dilution stage where the essence is diluted with the alcohol or water in a 1:100 ratio.  Finally, homeopathy proposes to treat based on the presentation of symptoms instead of the root cause – Ulcerative Colitis with bleeding and abdominal pain would be treated completely differently than UC with excessive diarrhea and rectal irritation.(1)

Compared with other treatments of its time like purging (through bleeding, vomiting, sweating, or other means) to balance one’s humors (or fluids and fibers, depending on your medical “beliefs”)(1) or the use of the jugum for nocturnal incontinence (not for the feint of heart) (2), homeopathy was relatively safe and did meet the “first do no harm” test for the most part at the time, but did it do any good?

A concise treatment of the effectiveness of homeopathy in general can be found in an article by Dr. Stephen Barrett over on Quackwatch.  Basically, true homeopathic treatments run afoul of a basic law of chemistry.  Avogadro’s Constant (6.02x10e23, which represents the number of particles in one mole of a substance).  Any dilution beyond that number will result in no molecules of the original compound being present.  Thus, dilutions greater than 12C, which represents most homeopathic remedies, contain none of the substance that they claim will cure the underlying condition, essentially making them pure water or alcohol.  Hahnemann was aware of the dilution limits (Avogadro was a contemporary), but proposed that the “essence” of the substance was left behind.  There is no known scientific basis, chemical or physical, for substances to have an “essence”.

Homeopathy has been put forth as a treatment for both Crohn’s Disease and Ulcerative Colitis and its use appears to be on the rise.  Though current numbers are difficult to come by, a prior article in Gut in 1986 found that 1% of Crohn’s patients were treating their illness with homeopathy.(5)  That number rose to approximately 9% in 2011 in the United States, and may be up to three times that percentage in Germany (where homeopathy is most popular).(6)

The claims for homeopathic treatment of IBD are fairly significant.  One of the major online providers of homeopathic remedies notes the following:

Homoeopathic medicines have proven their efficacy in all sought of Inflammatory Bowel conditions and help by reducing the inflammation & ulcerations and helping in restoring intestinal functions back to normal. It also helps by enhancing ones immune response. If homoeopathic treatment is sought early it helps in preventing the progress of disease and preventing any complications (Skin, Liver, and Arthritis) from occurring, which are usually associated with the disease. It helps by reducing the ulcerations at first and over a period of time by healing them.[sic for the whole paragraph](7)

Others are more moderated in their approach, and don’t claim a cure but just symptom relief as an adjunct to real medical care by a physician:

Homeopathy will not be a cure for Crohn's, but rather is another way to try to deal with your symptoms.(8)

Unfortunately, none of the claims of homeopathy in IBD are backed up by clinical trials – there are no well-designed, double blind studies with homeopathic treatments that show efficacy beyond placebo for IBD in either reducing symptoms or curing the underlying conditions.  In fact, there is no evidence of the efficacy of homeopathy in treating any condition.  The most comprehensive meta-analysis of homeopathy, from the British Journal of Clinical Pharmacology, analyzed 17 prior reviews and concluded:

… there was no condition which responds convincingly better to homeopathic treatment than to placebo or other control interventions. Similarly, there was no homeopathic remedy that was demonstrated to yield clinical effects that are convincingly different from placebo. It is concluded that the best clinical evidence for homeopathy available to date does not warrant positive recommendations for its use in clinical practice. (9)

Bottom Line


-          There no evidence that treatment of any medical condition by homeopathy is effective, including IBD
-          Because homeopathic dilutions beyond 12C contain no active ingredient, those wanting to add homeopathic treatment to their existing regime can do so at a lower cost by drinking a glass of water a day

--------------------------------------------------------------------------------------------------------------------------------
1.       Jonas, Wayne B., and Jennifer Jacobs. Healing with homeopathy: The complete guide. Hachette Digital, Inc., 2009.
5.       Smart, H. L., J. F. Mayberry, and M. Atkinson. "Alternative medicine consultations and remedies in patients with the irritable bowel syndrome." Gut27, no. 7 (1986): 826-828.
6.       Hilsden, Robert J., Marja J. Verhoef, Heidi Rasmussen, Antony Porcino, and Jennifer CC DeBruyn. "Use of complementary and alternative medicine by patients with inflammatory bowel disease." Inflammatory bowel diseases 17, no. 2 (2011): 655-662.

9.       Ernst, Edzard. "A systematic review of systematic reviews of homeopathy."British journal of clinical pharmacology 54, no. 6 (2002): 577-582.

Sunday, August 4, 2013

IBD Research Update

Research Update


Time for the most recent research update.  There has been lots of great work in the IBD arena in the last quarter, and a few highlights are in order.  There are a mix of studies this quarter looking at both Crohn’s and Ulcerative Colitis (and in one case both).

Ulcerative Colitis and Anti-TNF alpha Drugs


The early Anti-tumor necrosis factor alpha drugs were primarily focused on Crohn’s disease for their initial indications.  Shortly after their demonstrated success in clinical trials with Crohn’s, trial began for the treatment of Ulcerative Colitis.  The UC treatment studies are less extensive than with Crohn’s, but a review showed that both infliximab and adalimumab were effective in promoting mucosal healing and improving quality of life.  The cross-trial success means that the Anti-TNF alpha drugs should be considered earlier in the treatment spectrum, and that there are further options available before colectomy. (1)

Intestinal Failure and Crohn’s Disease


The risk of intestinal failure is one of the greatest fears for those with Crohn’s disease.  A good predictive model of who will experience intestinal failure is not in place, but several predictive factors were identified in a small, retrospective review of case studies.  The review found that risk factors included an “earlier age at diagnosis, family history of inflammatory bowel disease, stricturing disease, younger age at first surgery, and operative complications”.  The higher risk factors may influence the use of more aggressive treatments or more frequent monitoring in specific patient populations.  Because new treatment modalities were not available when the early disease hit many in the study, these may not be predictive for more recently diagnosed patients .(2)

Stem Cells and Crohn’s Disease


The ASTIC trial (Autologous Stem Cell Transplantation International Crohn’s Disease) is one of the most ambitious clinical trials currently in progress in that it seeks to potentially cure Crohn’s disease and not simply induce and maintain remission (not that maintaining remission is anything most of us wouldn’t take seriously!)   The trial created a transplant group, which received a high risk bone marrow transplant of stem cells.  The control group was provided transplantation at 13 months.  45 patients were recruited, but 9 of the control group dropped out due to complications (all of the patients were moderate to severe with disease activity).  The initial results are excellent, as shown in the table below:

Parameter
Treatment Group, % (n = 22)
Control Group, % (n = 13)
Simple Endoscopic Score for Crohn's Disease of 0 (normal)
25
0
Simple Endoscopic Score for Crohn's Disease <5 (inactive or mild)
61
20
Complete mucosal healing
40
15
Segmental healing
58
30

There are still some words for caution.  First, the trial is small and the results are short term.  Second, and more importantly, the treatment is high risk – one patient died, likely from complications, in the treatment group, and all patients had failed with other immunosuppressants first.  The study definitely shows promise for further studies and refining the safety of the treatment regime if they pan out.(3)

Crohn’s, UC and BMI


Body Mass Index is a frequent (though controversial) measure of obesity.  In the large EPIC (European Prospective Investigation into Cancer and Nutrition) study, over 300,000 people were reviewed to determine if obesity as measured by BMI was a predictor for Crohn’s disease or Ulcerative Colitis.  Despite being a predictor for other conditions (such as Type II diabetes), the study found that there was no correlation between BMI and the incidence rates for either form of IBD.  This indicates that obesity is not associated strongly with developing IBD, or that BMI is a poor measure of obesity for this particular prediction.(4)

Bottom Line


·         The Anti-TNF alpha drugs have shown repeatedly to be a viable option for early, effective treatment of UC.
·         The likelihood of intestinal failure in Crohn’s is becoming better understood, and earlier, more severe symptoms tend to predict later failure.
·         Stem cell treatments are very high risk but potentially high reward, with small, initial results showing dramatic improvements in treating Crohn’s.
·         Obesity as measured by BMI is not associated with a higher risk of developing Crohn’s or UC.

-------------------------------------------------------------------------------------------------------------------------------------
1.       Danese S, Colombel JF, Peyrin-Biroulet L, Rutgeerts P, Reinisch W. Review article: the role of anti-TNF in the management of ulcerative colitis -- past, present and future.  Alimentary Pharmacology and Therapeutics.  May, 2013.
2.       Predictors for developing intestinal failure in patients with Crohn's disease. Gearry RB, Kamm   MA, Hart AL, Bassett P, Gabe SM, Nightingale JM.J Gastroenterol Hepatol. 2013 May;28(5):801-7. doi: 10.1111/jgh.12115.

4.       Chan SS, Luben R, Olsen A, Tjonneland A, Kaaks R, Teucher B, Lindgren S, Grip O, Key T, Crowe FL, Bergmann MM, Boeing H, Hallmans G, Karling P, Overvad K, Palli D, Masala G, Kennedy H, vanSchaik F, Bueno-de-Mesquita B, Oldenburg B, Khaw KT, Riboli E, Hart AR.  Body mass index and the risk for Crohn's disease and ulcerative colitis: data from a European Prospective Cohort Study (The IBD in EPIC Study).  Am J Gastroenterol. 2013 Apr;108(4):575-82. doi: 10.1038/ajg.2012.453. Epub 2013 Jan 15.