Ten years of NIHR research training: who got an award? A retrospective cohort study

Principal findings

The key predictors of application success were: the award level and type of programme applied to, and the applicant’s previous experience of having had an award. One characteristic of the host institution was important: applications from institutions with, or associated with, a medical school were more likely to be successful. After adjusting for factors such as having a medical school, applications from GT and RG institutions were not found to be more successful than those from other institutions, and applications from HEIs were found to be less successful than those from NHS Trusts and other organisations. Differences in success rate according to the professional background of the applicant were only apparent in the classification tree modelling: in the subgroup of applications for senior awards from people who had not previously held an NIHR award, AHPs and other HPs fared better than medics, who in turn fared better than nurses/midwives, non-HPs and some other groups. It was notable that gender was not a predictor of success in any of the analyses.

Strengths and limitations

This paper reports analyses of a well-maintained administrative dataset which contained information on a number of factors potentially associated with award success. In interpreting the study results, it is important to be aware that the dataset necessarily reflects the eligibility characteristics of the award schemes in operation over the 10-year period: some awards (especially at predoctoral level) were only open to some types of applicant; some awards (especially those combining research with clinical practice) could only be held in certain types of institution; and not all schemes were open for the full 10-year duration of this study. These eligibility criteria particularly affect the interpretation of the classification tree results, and merit further consideration because of their potential policy implications. For example, particular care is needed when eligibility criteria constrain the factors which might differentiate within one subgroup but not within another; for example, within the large group of early career researchers, where association with a medical school appears to be less relevant to success for applicants to the ICA and IAT schemes than for applicants to the NIHR Fellowship scheme. Such a conclusion would, however, be unjustified because nearly all the ICA and IAT awards—but not the NIHR Fellowship awards—are held in institutions associated with a medical school, reducing the potential for within-group distinction on that basis.

Over and beyond formal eligibility considerations, potential predictive factors were found not to be independent of each other; but this was expected and addressed to a substantive extent by the regression modelling and classification tree. However, more subtle selection effects (eg, who got put forward for which award and by what kind of host institution) are also likely to have been in operation and these cannot be examined using routinely collected administrative data. Other published research (eg, Burns et al4) which uses a similar kind of data source has indicated the same kinds of interpretive limitations.

The inherent variability of classification tree modelling should also be considered when interpreting the potential predictive factors identified in our analyses. As a form of multivariable analysis, the classification tree method is dependent on the parameters selected and the values of the input variables. Selection of different parameters and input variables may result in a slightly different tree structure; there is an interdependence between the input variable, model parameters and the output variables.10 The parameters selected in our classification tree analysis were based on a pragmatic approach to obtaining robust and interpretable summaries of the data. As each node and branch of the tree is an element of knowledge about the relationship between the output variable (award success) and the input (predictor) variables, our intention was to construct a tree with sufficient detail to identify the main effects, without over complicating the interpretation of the results.

Previous research has tended to be designed around specific potential predictors of success, such as gender, or specific subgroups of applicants such as clinical academics. For example, Waisbren et al,5 Brown et al3 and Burns et al4 all focused on gender differences in grant funding. This study was broader ranging and employed two different analytic approaches to minimise the likelihood of misinterpreting the findings. By contrast with other published work, including Burns et al,4 in this dataset and using these analytical approaches, no direct association was found between gender and award success. The success rate for male and female applications was equitable (22% vs 21%, table 2). However, in similarity with Waisbren et al5 we observed that the numbers of applications made by males and females explained differences in the numbers of awards made at different seniority levels.

The distribution of medical schools helped explain differences in success rates between types of host institution often considered to be of higher or lower status. It has long been assumed research intensive universities, such as those within the RG or GT, are at an advantage at securing NIHR funding.1 However, our modelling has shown that the disparity in research funding across HIE groups is explained by the presence of an associated medical school. This is reasonable given that the premise of the NIHR is to support applied health and care research within the UK National Health Service (NHS).

Finally, the available dataset was limited in terms of the available applicant demographic data. Applicant data on ethnicity, age, full time/part time, and use of RDS support is not routinely collected at application stage and was therefore not available for analysis. Similarly, applications are judged individually and are therefore not routinely categorised into topic areas and types of research at the application stage. While the findings of this report suggests that funding applications for NIRH Academy personal awards are treated equitably, further scrutiny of application success by additional applicant factors and topic areas would support the development of future funding strategies.

Implications of the findings

Overall, the evidence suggests the NIHR Academy succeeded in its objective of treating all applications equitably. Factors associated with type of award (seniority level and type of programme) were the most important predictors of success, together with a specific characteristic of the host institution (ie, association with a medical school). Success rates did not differ according to the gender of the applicant, and applications from doctors were not more likely to be successful than applications from other professions.

There were some specific circumstances in which combinations of personal characteristics did seem to be relevant: among applicants who had not held a previous award, the success rate of applications for the more senior awards differed according to the applicant’s professional background, AHPs/other HPs being more successful than the other groups. Taken together, these findings suggest an essential fairness in how the quality of a submitted application is assessed, but they also raise questions about variation in the opportunity to submit a high-quality application.

Nevertheless, it must be acknowledged that there are likely to be factors associated with funding success that we did not have access to. For example, this study cannot address the question of how the presence of a medical school improves the likelihood of success of an application. We also cannot know to what extent host institutions provide support and mentoring for preparing an application. The companion study, however, by using a qualitative methodology, does identify some candidate mechanisms, and taken together, the two sets of findings provide valuable insight as to how research capacity development initiatives might be targeted in the future.

Further research

Other studies have found that some topic areas and types of research are more likely to find favour with funders than other areas. In addition, we were unable to evaluate variation in application success by applicant factors such as ethnicity, age or institution factors such as access to RDS support. It was not possible to examine the role of these variables in the present study, and they represent a gap which future research should seek to rectify. The relationship between applicant gender, award seniority and application outcome (funded/rejected) should be further explored and evaluated within the context of national gender distribution by seniority to identify the inflection point where applications for personal fellowship awards from women drop below the median. In the longer term, the impact of any initiatives prompted by the present study findings would need to be evaluated.

Leave a Reply