Contra Science-Based Medicine

Epistemic status: hand-wavy, but making a serious point

TW: diets.

I recently did some reading about ketogenic diets for cancer, and I’d like to compare and contrast my approach with the explanation on the blog Science-Based Medicine, which consistently presents the “skeptical” perspective on alt-med questions.

David Gorski is a cancer biologist himself, as I am not; his posts are always informative, and I have no quarrel with his facts. I read the studies mentioned in the post, so we’re using pretty much the same set of data points. And I agree with the broad outlines of his claims: ketogenic diets have some promising but by no means conclusive preclinical evidence for brain cancers; they’re definitely not a substitute for chemotherapy in general; and Dr. Seyfried has been overselling his research as a cure for cancer in disreputable alt-med venues.

But I want to pick apart some points of perspective and interpretation.

The first part of the post is all about painting Seyfried as disreputable because of his associations with alt-med institutions. Gorski says of the American College for Advancement in Medicine, “this is not an organization with which a scientist who wishes to be taken seriously by oncologists associates himself.”

Now, I’m not defending the cancer quacks mentioned; these are people who pitch chelation and coffee enemas, things that are pretty clearly scientifically disproven.  However, I’m suspicious of the rhetorical trick of guilt by association and argument from consensus.  Surely we care about whether Seyfried is correct, not whether he is “taken seriously”, “reputable”, or “legitimate.”  These are all social words, not scientific ones, and constitute an emotional appeal to social conformity and authority.

To his credit,  Gorski doesn’t stop there; he does make substantive criticisms of Seyfried’s work.  But I think it’s worth pointing out when, as happens so often in the biomedical world, a social argument is conflated with a scientific one.

Gorski goes on to criticize Seyfried for “exaggerating how hostile the cancer research community is towards metabolism as an important, possibly critical, driver of cancer” when cancer metabolism is, in fact, an active area of current cancer research.  He goes on to say, “Dr. Seyfried, in my readings, appears all too often to speak of “cancer” as if it were a monolithic single disease. As I’ve pointed out many times before, it’s not. Indeed, only approximately 60-90% of cancers demonstrate the Warburg effect.”

None of these facts are wrong, but the interpretation is misleading. Cancer metabolism and metabolic mechanisms for cancer treatments are, in fact, common topics of cancer research; but this ought to be evidence in favor of Seyfried’s hypothesis, that it’s within the range of mainstream science and is supported by many cancer biologists, rather than being pure invention like most alt-med “cancer cures.”

I’d also argue with the statement that “cancer isn’t one disease.”  It’s true that not all cancers demonstrate the Warburg effect, but 60-90% is a lot of cancers; a drug that was effective in 60-90% of cancers would be as revolutionary an advance as chemotherapy.  An antibiotic that killed 60-90% of bacteria could fairly be said to “kill bacteria.” When most (if not all) cancers have structural features in common, that indicates that talking generally about “cancer” is meaningful, and that it doesn’t make sense to treat every sub-sub-type as though it is a completely different disease.  Cancer has both unity and diversity.  Saying “cancer isn’t one disease” is a rhetorically loaded move that means “don’t generalize from one type of cancer to another.”  But it’s not correct to never generalize; that would utterly paralyze research.  How much it’s safe to generalize depends on how common the relevant feature is across cancers; in the case of the Warburg effect, that’s a matter of current debate, but it’s fair to call it pervasive.

I don’t have much criticism of the way Gorski handles the ketogenic diet studies. He’s on the skeptical side, but skepticism is warranted. Mouse studies very frequently don’t generalize to humans; they’re suggestive, but only weak evidence. And while there were two case studies of patients who did notably better than typical glioblastoma patients on ketogenic diets, we don’t have enough patients to be confident that the improvements were a result of the diet.

But then Gorski says, “Clearly, ketogenic diets are not ready for prime time as a treatment for cancer.”

Now, wait a minute. What does that even mean?

As a cancer patient, does it make sense for you to try a ketogenic diet?  Well, there’s a plausible mechanism for it to work (particularly in brain cancer), there’s some suggestive evidence in mice and a few humans with brain cancers, and — crucially — it’s just a diet.  People go on ketogenic diets all the time, for no other reason than wanting to lose weight. It’s even been shown medically safe (though apparently hard to comply with) in cancer patients. Trying a special diet is pretty low risk, and a reasonable person aware of the evidence might very well choose to try it.

It doesn’t make sense to use a ketogenic diet as a replacement for chemotherapy or radiotherapy in cancers where those treatments work. That would be very unsafe.  But for certain advanced brain cancers, chemo barely extends life if at all, and is very unpleasant. If there’s anyone who has a good reason to refuse chemotherapy, it’s someone who’s almost certain to die soon and doesn’t want their last few months to be agonizing.

Is a ketogenic diet for cancer something that every oncologist in the world should be prescribing for his patients? No way. Should it be the “standard of care”? No; there isn’t enough evidence that it helps.  But is it worth trying for an individual who wants to? Quite possibly.

The distinction here is about where you put the reader’s locus of identity. Is a reader supposed to imagine herself as a potential cancer patient, considering whether or not to try the diet? Or as a potential administrator, considering whether or not to make the diet a policy for everyone?  The rhetorical trick Gorski’s using here is in identifying the reader with a nebulous “we”, as in “should we put cancer patients on ketogenic diets?”  You are meant to imagine a consensus, or an authoritative body.  The medical profession, the government, something like that.  This imagined “we” is the mirror image of the nebulous “they” that conspiracy theorists believe in, the “they” who doesn’t want you to know about cancer cures.

The overall effect of believing in an imagined “we” or an imagined “they” is to make social reality the primary reality.  “We” or “they” represents a vague model of “society” — the “respectable” people, the “legitimate” and “reputable” people, the “consensus”. In other words, the tribal elders. If you have a positive association with “the consensus”, as Gorski clearly does, then you want to expel the “disreputable” from the consensus.  If you have a negative association with “the consensus”, then you mistrust anything that sounds official and look for fellow mavericks and outsiders.  In neither case are you primarily evaluating claims of fact; you are evaluating people.

For instance, the existence of Phase I/II trials of the ketogenic diet on glioblastoma ought to be good news for ketogenic diets.  More evidence will soon come in; and the fact that the studies exist at all is further evidence that ketogenic diets are taken seriously by mainstream cancer researchers.  However, Gorski treats this as an indictment of Seyfried, because he wanted to do an (uncontrolled) case series of ketogenic diets rather than the more thorough controlled studies.  The overall intent of the blog post is to communicate Seyfried is disreputable, cancer is complicated, people who believe in cancer cures are beyond the pale, when one could have used exactly the same facts to make the point ketogenic diets are an exciting possibility for glioblastoma and the preliminary evidence is encouraging.

My own perspective can perhaps be summarized as “a contrarian worldview from mainstream sources.”  Looking at ordinary sources like journal articles and historical primary sources, looking at uncontroversial claims of fact, often gives me a view of the world that is quite different from the “we”-based view where “society” is more or less getting things right.  My object-level beliefs are rarely that unusual; the connotation of those beliefs is where I differ from most people.  I don’t feel myself to be safely nestled in the lamplit circle of “we”; I feel like I’m outside, tumbling in the abyss, with only the frail spark of my mind to illuminate a small patch around me.  And I think that, ultimately, the abyss is real, and the lamplit circle is imaginary.

What Is To Be Done?

Epistemic status: loose and speculative

This is the last post in my cancer series. On reflection, there’s a lot I want to edit and expand here, and I think the right format for this  is a book rather than a blog. So, over the next several months I’ll be working on that.

In the meantime, I want to draw some conclusions given what I’ve found out about cancer so far. What are the next steps? Where do we go from here? The world I see is one where the “efficient market hypothesis” doesn’t hold in cancer research. Just because an idea is promising doesn’t mean it’ll be tried, particularly in human clinical trials.

So what can you, a reader, do to cure cancer?

1. Do cancer research

This one’s kind of a no-brainer, but I put it here because I see a lot of young EA’s wondering what to do with their lives, and the most frequent ideas are things like “make a lot of money to give to charity” or “work at an EA organization”, but I really believe a broader range of object-level skills could be useful for bettering the world. Doctors and biologists are important. Ultimately, the things that kill the most human beings today are the noninfectious diseases of aging.  If your heroes are the people who eradicated smallpox, maybe you should take up the cause of ending another disease.

2. Invest in or donate to organizations doing undervalued cancer work

In my blog series, I pointed out some researchers that I thought were doing unusually promising early-stage research.  Getting preclinical studies to clinical trials takes funding.

I haven’t investigated any of these organizations as organizations — I don’t claim to know that they’re likely to be profitable investments or efficient charities. I’m just looking at the drug candidates that I’ve found promising and seeing which existing organizations are involved in researching them.

3-bromopyruvate, the glycolysis inhibitor I’m bullish on, is being developed by PreScience Labs.

The Cancer Research Institute, a nonprofit that accepts donations, has funded a great deal of important immunotherapy research, including most of the recent work on mixed bacterial vaccine by the late renowned immunologist Lloyd J. Old.

The Fibrolamellar Cancer Foundation, which focuses on the rare liver cancer fibrolamellar hepatocellular carcinoma, has Sanford Simon on its advisory board and funded his research on anti-IgG antibodies to precisely detect and potentially destroy cancer cells.

I expect that there are other avenues to funding particular lines of scientific research, from creating novel grant-giving foundations to crowdfunding experiments.

I’m also interested in institutions like IndieBio, which try to bring a radical, Silicon Valley spirit to the biotech industry, and get funding for biomedical startups working on hard problems.

3. Political reform

It would be easier to innovate in cancer research if the regulatory challenges were less onerous.  Lobbying and activism, in the US or elsewhere, could probably be helpful.

This is an area where think tanks and patient advocacy groups are relevant.  I don’t have a clear idea of which precise policy goals are the most useful and attainable, but people with more of a policy bent can probably answer that question.

A different kind of “political” approach is regulatory arbitrage — trying to find or negotiate a favorable political climate to research somewhere outside the US.

4. Further meta-research

We clearly need more evaluative work done on the questions “What types of cancer research are most promising? Where is the low-hanging fruit, if any?”  I’ve been doing that, but I’m only one perspective.  This seems valuable in the context of something like the Open Philanthropy Project, which tries to evaluate the tractability of entire goals.

 

Regulatory Problems with Cancer Research

Epistemic status: more argumentative than the other posts in this sequence. This is obviously informed by my own political views, but my intent is to be convincing to a range of audiences.

In this sequence I’ve been arguing that, while most cancer drugs developed over the past several decades are not very effective, there are potentially exciting avenues of research that haven’t gotten much attention or funding yet.

Why is this the case?  Cancer research is a huge field full of intelligent people. Cancer is a very common disease and there’s a lot of money to be made in treating it. “Curing cancer” is a byword for a lofty goal.  Why should there be any 20-dollar bills lying on the sidewalk at all?

In particular, why would there be less progress since the War on Cancer, which allocated much more federal funding to cancer research than was available before?

The conventional story is that cancer is simply hard. We already gathered the low-hanging fruit of radiotherapy and cytotoxic chemotherapy; now we’re trying to cure the tougher cancers, and it just takes more money and time.

I’ve been arguing that the “cancer is hard” story is incorrect. Targeted chemotherapy, the most popular approach for the past two decades, tends to fail because of the incredible diversity and mutability of cancer.  Approaches that focus on what cancers have in common, like their high glucose requirements or susceptibility to immune defenses, might turn out to work much better.

But, if I’m correct, why hasn’t some enterprising cancer researcher already come to the same conclusions? Even if I’m wrong, I’m not unique; a lot of my argument is just echoing James Watson’s views. Why haven’t any investors and funders decided it might be a good idea to try cancer research the way the discoverer of the double helix thinks we should do it?

This can be explained by an increase in the regulatory burden of cancer research in the past several decades. Clinical trials have become more expensive, require more paperwork, and allow less freedom of judgment from clinicians and researchers.  Only a large pharmaceutical company can afford to run Phase II and III clinical trials these days.  There’s more money in cancer research than ever, but it’s harder to try new things on sick people.  This tends to narrow cancer research to established players and drug classes.

In the world of early-stage tech startups, success follows a power-law distribution. Investors gain more money by funding a handful of huge successes than they lose by giving small investments to a lot of things that don’t work out. So it makes sense that, for instance, YCombinator keeps casting a wider net, accepting startups at earlier stages, and actively seeking outliers and mavericks. They want to make sure they don’t miss the next AirBnB.

It would seem to make sense that investing in drug candidates would work similarly; you don’t want to miss the next Gleevec either.  But if the cost of testing is too high, casting a wide net becomes much more expensive. You can’t just give the founders of an early-stage biotech company a little funding to see if they can do something awesome with it.  And so, medical research becomes much more conservative.

Regulation Has Increased Costs and Slowed Drug Development

90% or more of a typical drug’s costs come from Phase III clinical trials. So it makes sense to focus on the costs and barriers associated with clinical trials, to see if they’ve gone up over time and what the consequences have been.

As of 2005, the R&D cost of the average drug was $1.3 billion. In 1975, that figure was $100 million. (That is, drug trials have gotten on average more than 13 times more expensive over the past forty years.)  Phase III trials are becoming longer, involving more procedures and more hours of work, and have lower enrollment and retention due to more stringent enrollment criteria and trial protocols.  

Protocols for clinical trials are now over 200 pages long on average. The combined costs are $26,000 per patient.  The annual rate of cost increase is itself increasing, from an annual increase of 7.3% in 1970-1980 to 12.2% from 1980-1990 (inflation-adjusted.)  The estimated cost per life-year saved from current clinical cancer trials is approximately $2.7 million.

It now takes an average of 12-15 years from drug discovery to marketing, compared to an average of 8 years in the 1960’s.  Before the 1962 Kefauver-Harris amendment that vastly increased FDA powers, it took only 7 months. For oncology drugs, just the preclinical work takes 6 years; once early clinical trial data are in, it takes 26-27 months to proceed to Phase II or III; and it takes an average of 14.7 clinical trials (Phase I, II, or III) to get a drug approved.

Running a clinical trial requires protocols to be approved by the FDA, the NCI (National Cancer Institute, the primary funder of cancer research in the US), and various IRBs (institutional review boards, administered by the OHRP, or Office of Human Research Protections.)  On average, “16.8% of the total costs of an observational protocol are devoted to IRB interactions, with exchanges of more than 15,000 pages of material, but with minimal or no impact on human subject protection or on study procedures.”  Adverse events during trials require a time-consuming reporting and re-consent process. While protocols used to be guidelines for investigators to follow, they are now considered legally binding documents; so that, for instance, if a patient changes the dates of chemotherapy to schedule around family or work responsibilities, that is considered to be a violation of protocol that can void the whole trial.

To handle this regulatory burden, an entire industry of CROs (contract research organizations) has grown up, administering trials and handling paperwork to make the experimental drug look good to federal regulators. Like tax preparers, CROs have an incentive to keep the regulatory process complex and expensive.

The result of all this added cost is that fewer drugs get developed than otherwise would.  Sam Peltzman’s 1973 study of drug availability and safety before and after the 1962 Kefauver-Harris amendment (which significantly enhanced FDA powers) found that a model of drug development predicted a post-1962 average of 41 new drugs approved per year, while the actual average was 16 new drugs approved per year. The pre-1962 average number of drugs approved per year was 40.

 

This is a graph of the number of new drug applications approved by the FDA every year from 1944 to the present. Note that the number of drugs approved has been largely flat since the 1962 Kefauver-Harris amendment, though the decline in drug approvals appears to precede the law by several years.

Increased Drug Regulation Has Not Meaningfully Decreased Risk

Peltzman’s study on the Kefauver-Harris amendment found that there was little evidence suggesting that more ineffective drugs reached the market pre-1962 compared to post-1962.

Comparing the US to Great Britain and Spain, each of which approve more drugs per year than the US, the other countries have no higher rates of postmarket withdrawals of drugs, suggesting that the extra regulatory scrutiny is not providing us with safer drugs.

Toxic death rates haven’t dropped much in Phase I trials. In 6639 patients, comprising 211 trials, between 1972 and 1987, the toxic death rate was 0.5%. In 11,935 patients, comprising 460 studies, between 1991 and 2002, the toxic death rate was also 0.5%.  

Between 1999 and 2006, the number of adverse drug reactions recorded in the US has actually been increasing, particularly  as the proportion of elderly patients taking many drugs has increased.

The most common severe drug interactions are often from old, well-known drugs, like insulin, warfarin, and digoxin.  “Antibiotics, anticoagulants, digoxin, diuretics, hypoglycaemic agents, antineoplastic agents and nonsteroidal anti-inflammatory drugs (NSAIDs) are responsible for 60% of ADRs leading to hospital admission and 70% of ADRs occurring in hospital.”  Increasing regulation on new drugs isn’t going to stop the problem of increasing adverse drug reactions, because most of those come from old drugs.

Cost-Benefit Tradeoffs Support Looser Regulations On Drugs

Gieringer’s 1985 study estimated the loss of life from FDA-related delay of drugs since 1962 to be in the hundreds of thousands.  This only includes the delay of drugs that were eventually approved, not the potentially beneficial drugs that were never approved or never developed, so it’s probably a vast underestimate.

In a recent paper, “Is the FDA Too Conservative Or Too Aggressive?“, the authors apply a Bayesian decision analysis to evaluate the overall cost of a trial based on the disease burden of Type I vs. Type II errors.

The classical approach used by the FDA is to constrain experiments to a maximum 2.5% risk of Type I error for all tests, and then choose a power for the alternative hypothesis by making the sample size large enough.  That is, no drug can be approved if there is a greater than 2.5% chance that it is ineffective.

This doesn’t make sense from a disease risk standpoint, because for very severe diseases, the risk of not trying a drug that might work is higher than the risk of trying a drug that doesn’t work.  The authors use data from the U.S. Burden of Disease study, which measures Years Lived with Disability to compute the “optimal” level of acceptable risk of inefficacy for drugs for different diseases. For instance, in pancreatic cancer, the BDA-optimal risk of Type 1 error is 27.9%, since the disease is so deadly.

Cancer in general, being both common and deadly, is an especially good area for looser drug regulation.  If a new therapy increased the cure rate of lung cancer by just 1% (through improved adjuvant therapy) and increased the average life expectancy of uncured patients by just 3 months, the [five-year] regulation-induced delay would cost more than 2,000,000 life-years worldwide.

Even this cost-benefit framing may be understating the case for FDA and OHRP reform, though. The problem seems to be less that the standards for efficacy are too high, than that the costs of compliance are too high because of redundant and excessive required documentation.  It would in principle be possible to streamline the process of conducting clinical trials without reducing its rigor.

We Think About Risk Wrong

In medical contexts, people often talk about the unknown as disrespectable. An “unapproved” drug, an “untested” drug, an “unproven” drug, a treatment that is “not indicated”, all sound unsettling.  Nobody wants to play cowboy in life-and-death situations.

But this kind of language is not actually about reducing risk.  Reality is probabilistic; all choices have potential risks and potential benefits. There’s no real wall, out in the universe, between the “safe/known” and the “unsafe/unknown”; that’s a human framing, akin to the Ellsberg paradox or the bias of ambiguity aversion. People prefer known risks to unknown risks.

In other words: death and disease are scary, and rightly so, but people will tend to be less frightened of risks that seem normal and natural (people have always died of cancer) than of risks that seem outlandish or like somebody’s fault (taking an experimental drug that might or might not work).  Chosen risk, conscious risk, stepping into the unknown, is viewed as worse than the risk of passively allowing harm to occur.  Even if the objective risk-benefit calculations actually work out the other way.

This is an instinct worth fighting.  Cancer is a common disease, yes; but the “normalcy” of it can blind us to the horrifying death toll.  As Bertrand Russell said, the mark of a civilized man is the capacity to read a column of numbers and weep.

Fear of action isn’t actually about making people safer. It’s about making people feel safer, because they aren’t looking at the whole picture.  It’s about making people feel like they can’t be blamed.

It’s Too Hard to Do Transformative Biomedical Research Today

Derek Lowe, an always insightful observer of the pharmaceutical scene,  comments on the VC firm Andreessen Horowitz’s first foray into biotech, “In this business, you work for years before you can have the tiniest hope of ever selling anything to anyone. And before you can do that, you have to (by Silicon Valley standards) abjectly crawl before the regulatory agencies in the US and every other part of the world you want to sell in. Even to get the chance to abase yourself in this fashion, you have to generate a mountain of carefully gathered and curated data, in which every part of every step must be done just so or the whole thing’s invalid, go back and start again and do it right this time. The legal and regulatory pressure is, by Valley standards, otherworldly.”

It shouldn’t be.

I am not a policy expert, so I don’t know what the appropriate next steps are.  What kinds of reforms in FDA and OHRP rules have a reasonable chance of being passed?  I don’t know at this point, and I hope some of my readers do.

I do know that committed activists can change things. In 1992, after a decade of heroic advocacy by AIDS patients, the FDA created the “accelerated approval” process, which can approve drugs for life-threatening diseases after Phase II studies.

We have to find a way to continue that legacy.