ABSTRACT
Rigorously evaluating the implementation of a large spontaneous initiative in medical care clinics involves issues of timing, funding, partnering with those leading the initiative, and collecting both outcome and implementation data without interfering with either the initiative or normal clinic operations. DIAMOND provides an example of both answers and problems.
BACKGROUND
It is increasingly obvious that the gold standard for research of randomized controlled trials has serious flaws, including poor external validity, prohibitively high costs, intrusion in medical practice, limited implementation and spread of results, and inability to evaluate implementation and organizational context. As a result, there is growing interest in observational studies of usual care or natural experiments to improve care that have both sufficient scientific rigor to produce valid results and sufficient information about the implementation process to facilitate replication and spread. However, in order to design and conduct such studies (especially those that evaluate a spontaneous large scale improvement initiative), one needs:
• A partnership with leaders of the natural experiment
• A creative but rigorous research design
• A way to obtain funding on a fast timeline
• An approach to collecting both outcome and implementation data in a way that doesn’t interfere with either the initiative or normal clinic operations.
The DIAMOND Initiative (Depression Improvement Across Minnesota – Offering a New Direction) innvolved agreement by all payers in Minnesota except Medicare to provide a new monthly payment to primary care clinics for adults with depression if they met certain criteria and if that clinic had been trained and certified as providing the collaborative care demonstrated in many controlled trials to be more effective than usual care. Fortunately, we have a regional quality improvement collaborative (ICSI or Institute for Clinical Systems Improvement) to serve as trusted convener and trainer for participating clinics. This Initiative provided an opportunity to test innovative solutions to all of the issues described above.
METHODS
1. Partnership between researchers and implementers
The research project PI was involved with the development of the Initiative from its earliest beginnings and was a member of its newly formed steering committee. As such, he was able to work with Initiative leaders to develop an implementation plan that also allowed for a scientifically rigorous observational study design and with the payers and clinics to design data collection methods that would provide answers to the questions of the steering committee, and therefore build their support for the study.
2. Creative but rigorous research design
The Initiative planned to implement the care and payment changes in five sequences over two years. Each sequence of 5-24 clinics (for a total of 84) was trained over six months and implemented DIAMOND as a new group of clinics began training. This staged implementation allowed for a multiple baseline, staggered implementation design that strengthened the usual before-and-after design by using each clinic as its own control and the different sequences as controls on secular trends. Since all participating clinics were identified and assigned to a sequence at the onset (thanks to the partnership relationship with the Initiative), it was possible to recruit patients with depression from all the clinics on an ongoing (weekly) basis throughout the two year training and implementation period plus nine months after the last sequence implemented. Recruited patients were surveyed at baseline for depression severity, care received, satisfaction, productivity, and quality of life. A follow-up survey six months later
allowed evaluation of remission and response rates and changes in care reports, satisfaction, productivity, and quality of life. Each patient also provided consent to obtain their payer records to evaluate impact on healthcare utilization and costs.
3. Funding on a fast timeline
Submission of an investigator-initiated proposal to NIH to evaluate the impact and implementation of this Initiative was made possible by the long planning time required to get agreements and specifications for such a large and status quo-changing Initiative. It was also facilitated by interest among steering committee members in having research answers to their questions about the Initiative and by having the PI involved closely in the effort from its beginning. The latter permitted submission of the research proposal to NIH before all the Initiative details were completed, although it seemed likely to be implemented.
4. Innovative and unobtrusive data collection
Instead of identifying research subjects in participating clinics during care, all seven payers in the state (except Medicare) agreed to submit lists of their members with new anti-depressant prescriptions from Initiative clinics to the study team after a one-week opportunity for them to opt out of the study after receiving a letter from the payer informing them of their selection. These payers continued to do this on a weekly or biweekly basis throughout the 33 month study period. With this information, the study survey center called potential subjects, screened them for eligibility (treatment for depression in an Initiative clinic with a PHQ9 score of 7 or more), and administered the rest of the survey questions to those who were eligible. The study also obtained permission from leaders of all 24 medical groups with clinics in the Initiative to survey a physician and administrative leader from each of the 84 clinics about implementation specifics and change factors prior to implementation and again at 12 and 24 months after. We also requested Initiative implementation costs from the payers, medical groups, and ICSI as well as operating costs/patient for one month on 1-5 occasions, depending on sequence.
RESULTS
The proposal received a very good score from the NIMH review panel and was funded in time to develop and implement the desired data collection systems by one month before the first sequence clinics implemented the new DIAMOND collaborative care system. If resubmission had been required, it would have been too late to do this. Over the course of 33 months, the payers provided identifying information about 24,065 member patients. Because of incorrect contact information or inability to reach these patients within the required 3 week window, only 60% could be contacted, and 34% of these refused or were incapable of answering our questions. That left 9,498 patients to be screened, 26% of whom were found to be eligible and 99.7% of those consented to have their insurance records released for analysis, resulting in 2,423 enrolled in the study. Approximately half of these subjects were recruited prior to their clinics’ implementation of DIAMOND and half were recruited after that. Comparison of available characteristics of the various potential and enrolled groups of subjects showed no clinically significant differences among them. The research/Initiative partnership that worked so well in developing the research proposal functioned even better during study operations. The study PI and project manager are members of the Initiative Steering Committee as well as other subcommittees. They present updates on Study status and lessons at every Steering Committee meeting and have provided Steering Committee members with a schedule of when each set of data will be available and analyzed so results can be reported to them. In turn, Steering Committee leaders are members of the Study Coordinating Team and provide reports on the Initiative at each of its meetings. Although it wasn’t planned, the Initiative Steering Committee asked (and the Study agreed) if the Study could collect operating costs to help it understand the financial aspects of the Initiative
Four years into the study, we have had extraordinary cooperation from all Initiative participants. Of the 553 clinic and medical group leaders (half physicians and hallf managers) asked to complete implementation surveys, 550 have done so (99.4%), even those in cliniics that have dropped out of the Initiative. Implementation and operating cost data collections were similarly successful and have been reported to the Initiative in aggregate form. Besides submitting potential subject lists to the Study throughout 33 months, the payers have also submitted preliminary activation and utilization data from consenting patients and it is currently under analysis.
PROBLEMS
The only potentially serious problem with the design is that Initiative data shows that only about 20% of eligible patients with depression are being activated into DIAMOND care. If that holds true for study subjects, we could have insufficient intervention patients among the postimplementation sample to adequately test the impact of the Initiative on patients. In order to minimize that problem, we added a request to the payers to also identify members with a claim for DIAMOND care, so they could be recruited as subjects as well.
REASONS FOR SUCCESS
1. Pre-existing collaborative relationships for quality improvement among the stakeholders
2. A neutral convening and facilitating organization like ICSI
3. Pre-existing trusting relationships of payer, medical group, and ICSI leaders with the PI
4. Involvement of the PI in the earliest phases of development of the Initiative, and a parallel involvement of ICSI non-research colleagues in the earliest phases of Study development
5. An external expert consultant with credibility in depression improvement research who was willing to support both the Initiative and the Study
6. Involvement of experts who also understand and support partnership research from around the country as co-investigators and consultants
7. Research deference to the operational needs and specific concerns of Initiative participants
8. Cross membership of Study and Initiative leadership on mutual steering committees
9. Willingness of the Study team to add research efforts to newly identified Initiative needs
10. Presence of an appropriate and available funding vehicle and a willing funder