Evaluation of the Open Operating Grant Program – Long Descriptions

Figure 1-1: Impact of supported papers produced by OOGP-funded researchers vs. OECD health research comparators (2001-2009)
2001 2002 2003 2004 2005 2006 2007 2008
Supported by OGP Funding 1.4 1.39 1.4 1.48 1.46 1.44 1.47 1.63
Canada 1.23 1.21 1.2 1.19 1.21 1.21 1.25 1.31
Australia 1.15 1.12 1.12 1.16 1.16 1.17 1.22 1.26
France 0.98 1.01 1 1.02 1.06 1.06 1.14 1.16
Germany 1.07 1.07 1.08 1.09 1.12 1.11 1.18 1.23
Netherlands 1.3 1.28 1.32 1.32 1.35 1.32 1.41 1.44
Switzerland 1.41 1.39 1.39 1.41 1.48 1.4 1.48 1.57
United Kingdom 1.21 1.25 1.24 1.25 1.27 1.26 1.3 1.41
United States 1.33 1.33 1.32 1.3 1.3 1.3 1.34 1.36

« Back to Figure 1-1

Figure 1-2: Scientific impact of OOGP-supported research papers (ARC)
Average Relative Citation
2000-2005 ARC score 1.44
2006-2009 ARC score 1.54
Overall ARC score (2000-2009) 1.5

« Back to Figure 1-2

Figure 1-3: Journal article productivity per year of grant duration by pillar of respondent
1st Research Theme/Pillar Average number of publications per year of grant
Biomedical Research 2.34
Clinical Research 2.03
Health Systems & Services Research 1.31
Soc, Cultural, Enviro, Popln Hlth Research 2.38
Total 2.26

« Back to Figure 1-3

Figure 1-4: Publication behavior by grant duration - When do supported researchers publish?
CSY CSY+0 CSY+1 CSY+2 CSY+3 CSY+4 CSY+5 CSY+6 CSY+7 CSY+8 CSY+9
All Durations (N=450) 7.52% 12.31% 16.77% 20.37% 17.53% 11.99% 6.37% 3.82% 2.12% 0.58%
5-year Grants (N=72) 10.81% 15.45% 15.31% 19.24% 17.28% 11.94% 6.18% 2.25% 0.84% 0.14%
3-year Grant (N=297) 6.93% 11.95% 17.73% 21.42% 17.09% 11.78% 5.66% 3.75% 2.54% 0.69%

« Back to Figure 1-4

Figure 1-5: Books/book chapters and reports published as a result of OOGP grants
1st Research Theme/Pillar Books/Book Chapters Published Reports/Technical Reports Published
Biomedical Research .99 .24
Clinical Research 1.05 .01
Health Systems & Services Research .08 .38
Soc, Cultural, Enviro, Popln Hlth Research 1.13 .55
Total .96 .25

« Back to Figure 1-5

Figure 2-1: Impact of applicants for two years following competition by application status and Canadian papers in health fields by publication year (2000-2009) (ARCs)
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
Successful applications 1.69 1.5 1.38 1.46 1.57 1.55 1.56 1.51 1.74 1.64
Unsuccessful applications 1.43 1.47 1.32 1.45 1.52 1.39 1.4 1.44 1.53 1.58
Applications by researchers who are never funded 1.39 1.25 1.47 1.45 1.32 1.2 1.39 1.45 1.47 1.2
Canada 1.24 1.21 1.19 1.19 1.18 1.2 1.2 1.25 1.33 1.33

« Back to Figure 2-1

Figure 2-2: Average Relative Citations of supported papers of funded researchers by peer review committee percentile ranking and publication year (2001-2009) (ARCs)
2001 2002 2003 2004 2005 2006 2007 2008 2009
Always Top Ranked (n=74) 0.66 2.12 1.34 1.68 1.86 2.07 1.8 2.11 2.05
Sometimes Top Ranked (n=470) 1.65 1.47 1.57 1.55 1.54 1.61 1.64 1.75 1.7
Never Top Ranked (n=956) 1.4 1.24 1.21 1.41 1.4 1.22 1.35 1.58 1.5

« Back to Figure 2-2

Figure 2-3: Previously funded status of OOGP renewal applications with FRNs, by Competition (2000-2010)
First version With more than one version
FY 2000-2001 48.97% 51.03%
FY 2001-2002 54.40% 45.60%
FY 2002-2003 55.56% 44.44%
FY 2003-2004 66.41% 33.59%
FY 2004-2005 66.84% 33.16%
FY 2005-2006 68.48% 31.52%
FY 2006-2007 75.44% 24.56%
FY 2007-2008 76.81% 23.19%
FY 2008-2009 76.17% 23.83%
FY 2009-2010 76.44% 23.56%

« Back to Figure 2-3

Figure 2-4: Concordance between independent and committee stage assessments
Bottom 20% at committee stage 60-80% at committee stage 40-60% at committee stage 20-40% at committee stage Top 20% at committee stage
Was in top 20% at Independent stage 0% 1% 4% 22% 75%
Was in 20-40% at Independent stage 1% 6% 24% 52% 21%
Was in 40-60% at Independent stage 5% 21% 49% 22% 3%
Was in 60-80% at Independent stage 16% 55% 21% 4% 1%
Was in bottom 20% at Independent stage 78% 17% 2% 1% 0%

« Back to Figure 2-4

Figure 2-5: Origins of top 5% of committee stage applications
Top 20% at independent stage 20-40% at independent stage 40-60% at independent stage 60-80% at independent stage
Top 5% at committee stage 94.59% 4.69% 0.60% 0.12%

« Back to Figure 2-5

Figure 2-6: True Positive applications perform better than the other three comparison groups
Average Relative Citation
True negative 1.45
False positive 1.59
False negative 1.54
True positive 1.63

« Back to Figure 2-6

Figure 3-1: Percent of funded researchers saying research results have had impacts
Percent of respondents who said research had… Total
%
Health system/care consumer level impacts 22.1%
Health system/care organization level impacts 26.0%
Health system/care provider level impacts 34.4%
Generated subsequent research by others 44.2%
Generated subsequent research by project team members 88.2%

« Back to Figure 3-1

Figure 3-2: Percent saying OOGP research has influenced stakeholders to a considerable or great extent
Community/municipal organizations
 0.3%
1.7%
Federal and provincial reps 1.7%
Health system/care manager
 1.7%
Health care professional orgs 2.3%
Media
 2.5%
Patients/consumers 4.5%
Health system/care practitioners 6.2%
Industry
 8.7%
Formal stakeholders 26.5%
Researchers/academics 61.4%

« Back to Figure 3-2

Figure 4-1: Average number of full-time equivalent trainees per year of grant by competition year
Year of competition Competition year 2000 Competition year 2001 Competition year 2002 Competition year 2003 Competition year 2004
Total average per year 1.54 1.38 2.43 1.73 3.81

« Back to Figure 4-1

Figure 4-2: Proportion of annual expenditure on the OOGP by pillar 2000/01 – 2010/11
Expenditures (New methodology) 2000-2001 2001-2002 2002-2003 2003-2004 2004-2005 2005-2006 2006-2007 2007-2008 2008-2009 2009-2010 2010-2011
Biomedical 61% 74% 79% 81% 82% 82% 81% 80% 80% 79% 78%
Clinical 5% 7% 7% 7% 7% 7% 8% 8% 9% 10% 10%
Health Systems & Services 1% 2% 3% 3% 3% 3% 3% 4% 4% 4% 4%
Social, Cultural, Environmental or Population Health 2% 2% 4% 5% 6% 6% 7% 8% 7% 7% 7%
Not Specified 31% 14% 8% 4% 2% 1% 0% 0% 0% 0% 0%

« Back to Figure 4-2

Figure 4-3: Barriers and challenges to Pillar III/IV applicants in the OOGP

There are three broad categories of barriers and challenges faced by Pillar III/IV applicants to the OOGP. These are (a) diverse projects cutting across disciplines and methodologies; (b) higher proportion of applications rated as "unfundable" and lower average peer review scores; and (c) renewal behaviours.

« Back to Figure 4-3

Figure 4-4: Likelihood of an application being deemed non-fundable (odds ratios)
Likelihood
Theme IV 2.14
Theme III 2.15
Theme II 1.68
Theme I 1

« Back to Figure 4-4

Figure 5-1: Open Operating Grant Program: Number of applications and success rates, 2000-2010
Funded Fundable, Not Funded Not Fundable Successe Rate (%)
2000-01 801 500 1,077 34%
2001-02 833 604 1,104 33%
2002-03 935 723 1,217 33%
2003-04 880 915 1,284 29%
2004-05 911 915 1,096 31%
2005-06 957 1,167 1,241 28%
2006-07 916 1,382 1,374 25%
2007-08 829 1,743 1,322 21%
2008-09 816 1,568 1,241 23%
2009-10 797 1,599 1,284 22%

« Back to Figure 5-1

Date modified: