Designing a system to fund health research

by Alan Cohen

Hypothetical representation of the frequency of different degrees of ethical research conduct under two funding schemes with differing amounts of pressure. The different categories of ethical conduct should be seen as a continuum, not as distinct. It is clear that even a small shift in the average level of ethical conduct could have a large effect on the conduct of science.

Thursday, I attended a meeting at the University of Sherbrooke in which ~20 health researchers could discuss some proposed changes to the funding system of CIHR (the Canadian equivalent of the US National Institutes of Health) with Alain Beaudet, President of CIHR (i.e., the big cheese). I had read the documents summarizing the proposed changes, and I was optimistic. I emerged from the meeting even more encouraged and highly impressed – Dr. Beaudet understands the underlying principles that make a funding system succeed or fail to a degree that I have rarely seen. This is because he does not simply view it as a budget to be allocated, nor as a set of priorities to be achieved. He understands that this is a system, in the sense that there are many different actors and incentives and rules and constraints, all of which interact in certain ways to produce outcomes that, while not deterministic, can be largely predicted based on the structure of the system.

And I like systems (in this sense), so I’ve been thinking about this for a while. So here’s a primer on CIHR funding, the problems it has now, the proposed changes, and why they’ll (probably) work. This may seem a bit technical or wonky to some of you, but I think it will be more interesting if you read with this general question in mind: How can we design a way to distribute funds for research that creates the most good research per dollar spent, given that researchers and administrators are the imperfect humans they are? 

The current system and its problems

Problem 1: Too much pressure

Currently, Canadian health researchers get money to perform research and (at most medical schools) for their salaries by applying for grants. These grants can come from private foundations such as the Gates Foundation or the Cancer Research Society, from industry such as the pharmaceutical companies, and from governmental granting agencies, mostly CIHR and some provincial organizations. Because we don’t get tenure in the traditional sense and our salaries come mostly from grants rather than the university budget, our jobs depend on being able to bring in grants. Universities compile statistics and have budgets based on the amount of grant money their researchers bring in, so they are always anxious to hire researchers who will bring in large amounts (and to get rid of those who are not productive).

In theory this is good, since it creates competition for posts and rewards productive researchers. The problem is that the stakes are so high for each researcher that a culture develops of trying to game the system to get grants, and the grants often go to those who are good at the game rather than good at research. This culture is generally not present in departments such as biology, sociology, etc. where the salaries are “hard-money,” i.e., funded by the university through a tenure system.

I have experienced this culture difference everywhere I’ve worked, but most strikingly in St. Louis. I was in the biology department at the University of Missouri-St. Louis, with many friends in Ecology and Evolution at Washington University. All of us and all of our professors pursued research for the love of it. Yes, profs applied for grants, but the job security took a lot of the pressure off, and even if one grant didn’t come through there was usually a way to make ends meet through creative budgeting. And the research culture was impeccably honest: yes, people wanted big publications and to be recognized for their work, but they wanted to be recognized for real achievements, not for the semblance of it.

In contrast, friends in Ph.D programs at the Wash U medical campus (under profs funded by soft money) told me numerous stories of fabricated data, data ignored when it didn’t conform to the professor’s hypothesis, and a backstabbing, competitive culture in the labs. I wouldn’t claim that everyone is like this under soft money, nor even close to a majority, but I do think the culture difference affects everyone to some degree. Because everyone else applying for grants is playing the game, you’ll only win if you do too. In most cases, no one really lies or breaks the rules, but everyone writes grants that are tailored to “sell” based on certain keywords and changes in emphasis. We know that once we have the money we can do what we want, so no one hesitates to slightly misrepresent their intended work in order to get the money.

One of the best concrete examples of this is in budgeting. For most grants, we are required to write a detailed budget and to justify the expenses, but in the end we can spend the money in any way that follows the general funding guidelines, regardless of what we wrote in our budget. So the writing of the budget becomes largely an exercise in marketing: show that you will train X students, etc. I was actually advised by a CIHR committee chair to write my budgets strategically in this way, even if I didn’t expect to spend the money in exactly that fashion.

Faking a budget is not the same as faking research, but a culture that accepts these misrepresentations is one in which a few more people at the extreme of the distribution will fake their results in order to get published and win at the game. More importantly, researcher bias is an important factor in generating false results. Even those who don’t intend to produce bad results may be more tempted to find what they need to find for their careers to succeed, and in sum this could result in a significant bias toward validating conventional wisdom even when it is not true. The few cheaters at the extreme may produce less harm than all the mostly-honest but pressured researchers in the middle.

Problem 2: Too much paperwork

Another well-meaning but counter-productive aspect of the funding system is targeted criteria to meet specific goals. For example, collaborations across provinces, studies of aboriginal populations, and studies of gender differences are all considered important, and many grants can have a larger chance to be funded if they include these aspects. There is nothing wrong with identifying such priorities and trying to encourage them, but the mechanism is often counter-productive. Everyone wants a higher chance of funding, and everyone tries to include as many of these extra credit aspects as possible, even when it is not really relevant to the research. People who are good at playing the game may even make a convincing case for this aspect of their research. In the end, it adds more paperwork, it becomes hard for reviewers to tell who is really fulfilling the criteria, and, most importantly, it creates research projects that are contorted to conform to many criteria outside their primary scientific goal.

The amount of work it takes to put together a full grant for CIHR is amazing if you have never done this. In addition to the 11 to 13 page main description of the project, there are budgets, at least two different summaries (sometimes more), letters from collaborators, CVs from all main participants, signed forms, a letter from the university specifying working conditions, and so forth. My completed grant submission last fall was a 227-page PDF file. By way of comparison, a similar form submitted to NSERC (the Canadian equivalent of the US National Science Foundation) was only 35 pages. Each document alone may be simple, but they tend to be interdependent in complex ways, and often require other people to complete certain tasks, such that one person not completing a task delays the whole process. In 2011 I spent at least three months working full time doing almost nothing other than preparing grants. An Australian estimate puts the cost of preparing a grant at around $17,000. Funding success rates in Canada are generally 15-20%, meaning that the average cost to get funded is about $100,000 (=$17,000/15-20%). And in order to have a remote chance of getting this money, you need to already be a high-performing researcher, the sort of person who would almost certainly conduct useful research even without being forced to jump through these hoops. All of which is to say that the current system results in an astounding waste of time and money in order to distribute funds.

The good news: solutions on the way

In the process of preparing my first grants as an independent researcher, I quickly arrived at an understanding of the problems described above. Luckily, it appears CIHR has also arrived at a similar understanding of the problems, and a few others as well, such as reviewer burn-out. They do a nice job of describing some of the basic challenges, and they propose some clear and promising solutions. The most important changes will be (1) to add a new type of funding based on individuals rather than projects (see below); (2) to streamline the review process and eliminate all extraneous information not absolutely critical to review; and (3) to create a College of Reviewers rather than a series of committees such that the politics of the committees cannot pervert the grant writing or review process.

For me, the most interesting and important proposed change is the funding of individuals, such as is done by the Howard Hughes Medical Institute and the Wellcome Trust. Under this program, successful researchers can apply for seven years of funding not based on a detailed, specific proposal, but based on a history of productive research and a general research theme. Researchers who get this grant cannot apply for most other CIHR grants. The grant should be less work to prepare, and happens only once every seven years, meaning that total burden of grant preparation will go down substantially for these researchers. The only problem for me is that the current plan is to fund 20-30% of researchers this way, a number that will mean  70-80% will still have to struggle through multiple grants for specific projects!

What the proposed changes will do less well is to address is the culture of competitiveness and high-stakes applications in medical research. Funding researchers rather than projects goes in the right direction, but only if enough researchers get funded to remove some of the pressure. Nonetheless, this is a clear step in the right direction for Canadian taxpayers, for researchers, and for those who depend on the results of our research.