« "Mitt Romney left the group, 'Pro-Choicers.'" | Main | Fareed Zakaria Is Part of the Reality-Based Community »
October 22, 2007
Was RAND Health Insurance Study Wrong?
News that there were serious methodological flaws in the RAND health insurance study is actually very, very important. The RAND health insurance study remains the source for almost all speculation about how individuals react to different types of health insurance. When we say that higher co-pays make people cut care indiscriminately, we're going off of their evidence. When some say that health outcomes weren't much better with no co-pays, they're going off of RAND's evidence. When HSA supporters say that higher co-pays didn't degrade health status at all, and thus we should cut insurance spending across the board, they're going off of RAND's evidence. The problem is, RAND's evidence may not have been very good:
Of the various responses to cost sharing that were observed in the participants of the RAND HIE, by far the strongest and most dramatic was in the relative number of RAND participants who voluntarily dropped out of the study over the course of the experiment. Of the 1,294 adult participants who were randomly assigned to the free plan, 5 participants (0.4 percent) left the experiment voluntarily during the observation period, while of the 2,664 who were assigned to any of the cost-sharing plans, 179 participants (6.7 percent) voluntarily left the experiment. This represented a greater than sixteenfold increase in the percentage of dropouts, a difference that was highly significant and a magnitude of response that was nowhere else duplicated in the experiment.
“What explains this? The explanation that makes the most sense is that the dropouts were participants who had just been diagnosed with an illness that would require a costly hospital procedure. … If they dropped out, their coverage would automatically revert to their original insurance policies, which were likely to cover major medical expenses (such as hospitalizations) with no copayments … As a result of dropping out, these participants’ inpatient stays (and associated health care spending) did not register in the experiment, and it appeared as if participants in the cost-sharing group had a lower rate of inpatient use. … the cost-sharing participants who remained exhibited a lower rate of inpatient use than free FFS participants, not because they were responding to the higher coinsurance rate by forgoing frivolous hospital care but instead because they did not need as much hospital care, since many of those who became ill and needed hospital care had already dropped out of the experiment before their hospitalization occurred. …
So when we say that higher co-pays didn't degrade health outcomes, it turns out that many of those facing a health crisis and assigned to the high co-pay set dropped out of the study. So we have no idea what their outcomes would've been. Sick individuals in the low co-pay bracket, by contrast, did not drop out of the study, and so the comparison between the two brackets is deeply flawed.
For most folks, this probably seem really wonky and in-the-weeds. But it's almost impossible to overstate how much pull the RAND study has in health policy circles. Jason Furman's whole paper on cost sharing? Largely based on the RAND study. Robin Hanson's theories about slashing medical care in half? Largely based on the RAND study. And now it turns out that the RAND study has some compromised data. I don't know which co-pay subgroups saw the most folks dropping out, and I don't know how the data would change if they had stayed in. It's possible that the RAND study's actual conclusions are right (if, at this point, somewhat outdated). But it's no longer clear that we can simply assume that that's true.
October 22, 2007 in Health Care | Permalink
Comments
Hurrumph! Lies, Damn Lies and Statistics!
Too bad we don't have a federal Dept. of Health & Human Services that was actually, you know, concerned about health and human services - and therefore gathered reliable data on plans, costs, and outcomes.
I guess this leaves us only somewhat worse off without any believable data since political lies will have to fill in more substantially for what we thought was empirical truth. The WSJ EB and CATO are on solid ground with whatever crap they print in the future.
Posted by: JimPortlandOR | Oct 22, 2007 12:57:11 PM
Ezra,
If this has you worked up into a frenzy, I'd really suggest that you spend some more time looking at the current state of clinical and cost-effectiveness studies of which you seem to be a big proponent. Much more difficult extrapolations and assumptions are often needed to come to those studies' conclusions, not even mentioning that the methodology itself (e.g. measures of utility benefits) are far less clear than those used in the RAND study.
I've said this before, but clinical/cost-effectiveness studies are really 15-20 years away before being ready for prime-time use. And by using the term "prime-time use"-- I mean they would still have more issues than what you've raised here on frequent occassion.
Posted by: wisewon | Oct 22, 2007 1:03:15 PM
Kind of a "survivor bias" situation.
Posted by: luci | Oct 22, 2007 1:19:48 PM
I find this fascinating, because I ran into a very similar problem while doing my doctoral dissertation, which was an outcome study of two different treatment programs for abusive/neglectful families. Both programs had a high dropout rate, and both I and my committee recognized this as a serious problem that had to be addressed in the discussion (the results looked unrealistically good because the families that weren't benefitting were more likely to drop out). Indeed, this is a common problem in social science research in general. What blows my mind is that neither the RAND researchers, or other people reviewing their research, have thought to point this out before now. I suspect this reflects the political context: the loathing that elite groups have for any kind of universal system has led them to uncritically accept the RAND results, and it's only now becoming respectable to question them.
Posted by: beckya57 | Oct 22, 2007 1:40:07 PM
It doesn't really matter why the people dropped out. Any difference that big should have been a red flag, even if expected large expenses weren't the only reason for dropping out. Why? Because in the real world you don't get to drop out of the ongoing experiment on paying for medical care. And the alternative isn't going back to your umbrella-coverage insurance plan either.
(If it turns out that Ezra's speculation is correct, btw, that may not even mean that copays are a bad thing, only that copays for big non-elective medical care are a bad thing. As for whether copays for other kinds of care discourage useful health-maintenance, you'd need a lot longer than 5 years.) I'm also a little surprised that this wasn't addressed way back when, because even willingness to enroll in such a study would have been conditioned on expected health needs -- it would be nice to know, for example, how many of those dropouts came immediately after randomization.
Posted by: paul | Oct 22, 2007 2:13:45 PM
If true, this is a pretty earth-shattering finding. I'm just about to read the paper, but I suspect that it's very unlikely that the RAND researchers did not know about this. It's not exactly uncommon to grapple with this stuff in econometrics.
Posted by: Zach | Oct 22, 2007 3:35:12 PM
This just bolsters the case for a new giant RAND-style health insurance experiment.
Posted by: Jeff Westcott | Oct 22, 2007 3:57:43 PM
I find it hard to believe that no one has discussed this yet. This study is very widely read and used as a classroom example by economists. Bias introduced by attrition is not a new idea, though papers damaged by it still get published, especially medical studies.
My guess: this issue has been debated already but the debate happened before the internet and is therefore not googleable--so it's almost like it never happened.
Homework: what are the bounds on the true results of the experiment given the level of attrition seen in the study (i.e. if everyone who atritted were sick, what would that do to the results)?
Posted by: slog | Oct 22, 2007 4:14:53 PM
Aaron Swartz wrote about this RAND study this year, which is what made me aware of it in the first place:
http://sciencethatmatters.com/archives/30
Among other things, he commented:
The RAND study was by far the biggest study of this kind, but other studies find similar results. One analysis found that regions whose Medicare programs give out more money (when the underlying healthiness of the residents is held constant) see no increase in survival rates. A replication found the same results in VA hospitals. Cross-national comparisons find "the impact of public spending on health is ... both numerically small and statistically insignificant". Correlational studies find "Environmental variables are far more important than medical care." And there are more where that came from.
Posted by: Jamie McCarthy | Oct 22, 2007 6:12:59 PM
Another bum rap, Ezra. The RAND researchers followed up on the drop-outs and incorporated their subsequent health status into the study. There's a bit of debate about whether the follow up was sufficient, but the basic criticism is specious.
Posted by: JasonR | Oct 22, 2007 8:00:15 PM
Read the original report. In particular, see page vi under "Possible Artifacts and Biases."
Now, whether they were successful in dealing with this particular source of bias -- they certainly were aware of it -- is beyond my expertise.
Posted by: idlemind | Oct 22, 2007 10:11:22 PM
Actually, the real discussion of the effect of drop-outs starts on page 14, in the section titled Retention in the Experiment. The authors explain why they believe the effect is small and does not change their findings.
I find it ironic that beckya57 cites this alleged flaw as evidence of "the loathing that elite groups have for any kind of universal system" that causes them to "uncritically accept the RAND results" when in fact the real problem is her own uncritical acceptance of the assertion that the study is seriously flawed.
Posted by: JasonR | Oct 22, 2007 10:37:06 PM
I dont think nyman's explaination is all that compelling. One of the main reasons people stop participating in these types of long term studies is that they just get fed up with the hassle of doing so. I suspect those on the free plan wouldve been much more willing to just put up with the inconveniance than those who had comparable or superior health insurance than what they got for participating in the study.
Posted by: pimp hand strikes! | Oct 23, 2007 6:33:55 AM
The proper way to fix medical care is to make it a universal, single payer national health care system that is actuarially advance funded using the actuarial cost method known as The Entry Age Normal Cost method, and by so doing invest a portion of the assets in common stock.
The first part will reduce costs from $2 Trillion annually to around $1.2 -1.3 trillion; the second will reduce costs another $800 billion to bring down the total costs in time to about $400-500 billion annually.
This is a reduction of 75-80%.
The first part of these cost reductions will take about ten years but there will be an increase initially of perhaps $150-200 billion for several years before the costs come down, but no tax increase will be necessary if we do this gradually.
I'm a retired health care and pension-consulting actuary and the only actuary to ever say this. That's because of massive conflicts of interest in my profession and failure to make actuaries accountable to the public.
Hillary's 7 points for health care reform and similar ones by the McKinsey study on health care reform are correct--but they will take a decade to do. Go online and read them.
As you read them please note that most, if not all, cannot be done unless you have a national health care system--like all other developed and civilized nations have had for years, some for decades.
This accounts for why their total health costs on a per capita basis are a fraction of ours and why they have had better outcomes in many cases.
However they too need to actuarially advance fund their systems if they wish to avoid ever-increasing costs as nations age--older people consume more health care and not just a little--a lot.
One credible estimate, by the employee Benefit Research Institute has that for the US the total health costs at age 65 and over as $1.3 million and that seems to be in right ballpark. For other countries these costs are much lower, but still quite high and they increasing at more than inflation.
Actuarial advance funding is a little secret that the health insurance industry and my profession do not want you to know. In some cases good health actuaries never learned it in the first place, as it is a byproduct of pension funding and they are two different specialties in my profession.
For the bad actuaries, after all, if they constructed an sex and age based morbidity table, just as mortality tables were developed some 350 years ago, one might confront the fact that the current system has such high and ever-increasing costs that it is obvious to all, even non-actuaries, that our mostly private system is a joke—60 % of the $2 trillion we spend annually on health care is private and a good part of another $400 billion, another 20%, we spend on Medicaid is too--and thus it is completely unsustainable, and that health actuaries and the companies they work for are the problem and not the solution.
Some of the larger ones will have an important role in this national health care system but not as insurers, as administrators and watchdogs and to help keep costs down by measuring outcomes and this has to be done by making them not just responsible but most of all accountable.
Accountability is something they have not been in the past.
I think only around a dozen or so at most will survive.
All those little bitty health insurance companies that have been ripping off the public with their individual health insurance policies will die as they should have long ago.
Lastly, by putting Medicare in legitimate competition with these new health care firms—and require the latter to include everyone and not to cherry-pick—and both will improve—Medicare by becoming more pro-active while these private firms by bringing their much higher costs down—thus benefiting the nation and the people enormously.
The actuarial advance funding is also necessary to fix Social Security and the source of the additional money will come from these reforms in health care and the monies invested—around $200 billion together—will be a major source of steady patient capital for many decades.
This will be invested partly in common stock once we have invested enough money for short term cash benefit outflow and this in turn will strengthen our economy in myriad important ways.
Because of these reforms, no tax increase will be necessary.
On the contrary, in a decade the annual costs of Social Security and this national health care system will be gradually be brought down to around 3% of pay for each—the combined employer-employee payroll tax. This 3% is the Entry Age Normal Cost.
The Initial Unfunded Past Service liability of both systems will be calculated by this actuarial cost method and then systematically paid off just like our home mortgages, from the US General Account, and again no tax increases will be necessary--the amounts will come from the aforementioned health care reform.
Our US firms then will be able to compete far more successfully than they now do with this albatross around their necks.
Posted by: Andy Lang | Oct 23, 2007 9:37:38 AM
why not read the response and judge for yourselves?
Posted by: lee | Nov 13, 2007 4:05:14 PM
Hi everyone,
Just wanted to introduce myself. This seems like a nice place and I look forward to hanging out here :)
Pam
http://sacredheartdietforreal.blogspot.com/
Posted by: pamelakworkoutgirl | Jan 13, 2008 6:10:46 AM
As the de facto administrator of the Security Fix blog, I've spent many an hour deleting spammy links left in the comments section -
- comments that usually lead back to the same kinds of Web sites you most commonly see advertised in junk e-mail.
Posted by: Garri Azz | Feb 16, 2008 12:13:15 AM
<a href=http://ertwerhsdzgnh.es>ertwerhsdzgnh.es</a> conditions <a href=http://ertwerhsdzgnh.es>ertwerhsdzgnh.es</a> 4383 http://ertwerhsdzgnh.es
Posted by: Suefemubs | Mar 23, 2009 6:50:09 AM
<a href=http://members.ebay.fr/ws/eBayISAPI.dll?ViewUserPage&userid=cialis-achat>Acheter cialis</a> medicaments disponibles par commande http://members.ebay.fr/ws/eBayISAPI.dll?ViewUserPage&userid=cialis-achat Livraison discrete et rapide. cialis en ligne Bon Marche Cialis ;) <a href=http://members.ebay.fr/ws/eBayISAPI.dll?ViewUserPage&userid=cialis-achat> cialis soft generique</a>
Posted by: demindicaPece | Mar 29, 2009 8:32:24 AM
The comments to this entry are closed.
