This key research evidence about the GP selection process outlines the development of the GP selection system and shows how it follows best practice.
History of the national GP selection system
Prior to the development of the national GP selection system, research showed that various selection methods were applied across different NHS England (NHSE) local offices and deaneries which lacked standardisation.
Directors of Postgraduate GP Education were therefore given responsibility for developing a selection and training system that ensured the process of recruitment demonstrates fair and open competition, and observes best practice in providing equal opportunities for all applicants (NHS Executive (2000) Enhanced Management Arrangements for General Practice Vocational Training, 1999/230 Ed).
Subsequently, a collective commitment was made to work towards standardisation of recruitment methods across NHSE local offices and deaneries which resulted in the national process.
National GP selection system
In keeping with best practice selection from other occupations, a multi method job analysis was conducted by Professor Fiona Patterson (Work Psychology Group) and colleagues in 2000. A thorough job analysis determines the relevant knowledge, skills, abilities and attitudes associated with competent performance in a job role and informs the design of selection methods which are accurate (reliable), relevant (valid), objective and standardised. The Patterson analysis was used as the basis of a new, national, selection process, and continues to evolve from that base.
The GP selection process currently uses national machine markable (MMT) short-listing tests, together with selection centre methodology which has been developed incrementally. The development and successful implementation of the MMTs, with clinical problem-solving and situational judgement components, has been a significant innovation in postgraduate specialty selection.
The importance of validation studies
UK general practice has undertaken long-term validation of its selection methods to ensure accuracy, fairness and cost effectiveness.
Evaluation from the candidate’s perspective consistently demonstrates that applicants have confidence in the relevance and fairness of the process.
This section is a resource for accessing key research evidence regarding the GP selection process which has successfully completed several years of national recruitment since 2006. Whilst not exhaustive, the research evidence presented below outlines the development of the GP selection system and indicates how it follows best practice.
Development
A competency model for general practice: implications for selection, training, and development, 2000
In the UK, organisational psychology principles are used to identify and develop selection criteria and methods by identifying core and specialty specific competencies. Early work included this 2000 paper: 3 studies from a single Midlands based Practice N= 31-35, in 2000. 11 competencies underlying good GP were identified as necessary for job performance.
Read the full article by Patterson et al. (2000) on the British Journal of General Practice website.
Using job analysis to identify core and specific competencies: implications for selection and recruitment, 2008
Using job analysis in this way Patterson et al. (2008) later undertook three multisource, multimethod independent studies to explore core and specific competencies in anesthesia, obstetrics and gynecology and pediatrics. The outcome comprised 14 general competency domains common to all specialties. However, there were significant between‐specialty differences in both definitions of domains and the ratings of importance attached to them.
Read the full article by Patterson et al. (2008) on Wiley Online Library.
The predictive validity of selection for entry into postgraduate training in general practice: evidence from three longitudinal studies, 2013
The 2008 study was replicated by Patterson, Tavabie et al. (2013) to explore competencies for general practice training which resulted in 11 competency domains, of which empathy and perspective taking, communication skills, clinical knowledge and expertise and professional integrity were rated as the most important domains.
Read the full article by Patterson, Tavabie et al. (2013) on the British Journal of General Practice website.
Development of competency models for assessors and simulators in high-stakes selection processes, 2014
Patterson et al. (2014) later extended this competency model approach to examine specific knowledge, skills and attributes linked to simulations in British GP Selection Centres (Stage 3 of the national Recruitment process).
Read the full article by Patterson et al. (2014) on Taylor and Francis Online.
Design, Pilot and Implement
The involvement of lay people in selection to general practice training schemes, 2005
This 2005 paper showed high correlation between lay assessor and medical assessor scores, reflecting a shared agreement in the best candidates. In the main, these ‘Lay’ people were found to have some medical or educational connection. Lay assessors continue to be used in Selection Centres as part of the GP recruitment process, and many medical disciplines have now adopted a similar approach.
Read the full article by Burke et al. (2005) on ResearchGate.
Evaluation of three short-listing methodologies forselection into postgraduate training in general practice, 2009
This compares three short-listing methodologies: clinical problem-solving test (CPST); structured application form (AFQ) and the novel situational judgement test (SJT). The optimum short-listing methodology was found to be a combination of the CPS and the SJT. The SJT replaced the AFQ in GP selection in 2007. The SJT/CPST combination remains the short-listing methodology for selection into GP training. NB this paper also describes the UK Selection Centre approach, referring to the three job-relevant simulations at the time (patient consultation, group and written simulation exercises). The simulations have since changed in type/format.
Read the full article by Patterson et al. (2009) on Wiley Online Library.
Evaluation and Validation
Evaluation of three short-listing methodologies forselection into postgraduate training in general practice, 2009
SJT Reliability. This paper is described above, but included in this section for a different function. In this pilot SJT internal reliability was reported in the range of r.0.80–0.83.
Read the full article by Patterson et al. (2009) on Wiley Online Library.
Can knowledge tests and situational judgement testspredict selection centre performance?, 2012
The CPST and SJT are shown to be predictive of performance in job simulations at Selection Centre (SC). The study provide evidence for their continued use in selection for general practice training. Subsequently, their predictive validity was shown to hold good for exit level GP training.
Read the article by Ahmed et al. (2012) on Wiley Online Library.
Evaluating cognitive ability, knowledge tests andsituational judgement tests for postgraduate selection, 2012
In terms of operational validity and candidate acceptance, the combination of the current CPST and SJT proved to be the most effective administration of tests in predicting selection outcomes. In terms of construct validity, the SJT was the best single predictor of subsequent performance in the Selection Centre (SC).
Read the full article by Koczwara et al. (2012) on Wiley Online Library.
Development of competency models for assessors and simulators in high-stakes selection processes, 2014
This article on selection in the context of an assessment or selection centre is a qualitative study exploring competency models to improve uniformity and calibration of the overall process. Most of the world literature was (and remains in 2018) quantitative, so this is unusual.
Read the full article by Patterson et al. (2014) on Taylor and Francis Online.
Acceptability and Feasibility
Evaluating candidate reactions to selection practicesusing organisational justice theory, 2011
Candidates generally have positive reactions during both short-listing and SC phases. All selection methods were considered job relevant and both phases fair. The CPST was considered more job relevant than the SJT, and the SC phase was rated as fairer than the short-listing stage. These findings supported previous research in other occupational settings. Candidate reactions were found to improve between 2007 and 2009 for short listing (machine marked) methods whilst the then-current CS was consistently rated as the most job-relevant selection method. Since this study SJTs have become even more widely accepted and are now used routinely from the undergrad stage.
Read the full article by Patterson et al. (2011) on Wiley Online Library.
Evaluating cognitive ability, knowledge tests andsituational judgement tests for postgraduate selection, 2012
This paper is described above, and included in this section for a different function. In this large sample (n.2947), there was a good agreement among respondents (>60%) that the content of SJTs was clearly relevant to GP training and appropriate for the entry level they were applying for. One third agreed that the test gave them sufficient opportunity to indicate their ability for training and would help selectors differentiate between applicants.
Read the full article by Koczwara et al. (2012) on Wiley Online Library.
How effective are selection methods in medical education? A systematic review, 2016
A systematic review of selection methods reported that there is acceptability evidence in the organizational psychology literature for the SJT (not specific to this exam).
Read the full article on Patterson et al. (2016) on Wiley Online Library.
Predictive Validity
Do general practice selection scores predict success at MRCGP? An exploratory study, 2012
The score for overall performance at selection achieved statistically significant correlation with examination performance (r.0.49) for the Applied Knowledge Test (AKT) and r.0.53 for the Clinical Skills Assessment (CSA) modules of the exit, licensing MRCGP exams.
Read the full article by Ahmed et al. (2012) on Taylor and Francis Online.
The predictive validity of selection for entry into postgraduate training in general practice: evidence from three longitudinal studies, 2013
Patterson et al. found that the CPST and the SJT correlate with the AKT at r=0.73 and 0.43, respectively (uncorrected), and the CSA at r=0.38 and 0.43, respectively (uncorrected). The Selection Centre was found to correlate with the CSA at r=0.32.
Read the full article by Patterson et al. (2013) on the British Journal of General Practice website.
Exploring the Relationship between General Practice Selection Scores and MRCGP Examination Performance, 2015
This detailed analysis for the GMC showed findings comparable to, and replicating, previous validity study results. The CPST and SJT correlate 0.73 and 0.46 with the AKT, again uncorrected for restriction of range. The correlations for the CSA are 0.49 and 0.54, respectively, which is slightly stronger than previously found. Similarly, scores from the Selection Centre correlate at r=0.42 (uncorrected) with the CSA. All of these results, including previously reported findings, are statistically significant at p < 0.001. Findings predicted applicants with top CPST/SJT test scores as also doing well at SC. The suggestion was made to ‘fast track’ those candidates into offers of training, bypassing the SC.
Read the full article by Patterson et al. (2015) on the Semantic Scholar website.
Reviews
Evaluation of GP Specialty Selection, 2016
An in-depth review of the national selection process for GP training at that time, with reference to cost effectiveness and suggestions to consider for continued quality improvement. Overall affirming, with significant constructive challenges that were subsequently considered and, where appropriate, addressed.
Read the full article by Davison et al. (2016) on the University of Birmingham website.
Utility of selection methods for specialist medical training: A BEME (best evidence medical education) systematic review: BEME guide no. 45, 2017
A systematic review of international measures of selection in postgraduate medicine. Very affirming for the UK GP system. Offers suggestions for a research agenda which resonates with the 2018 updated Ottawa consensus statement.
Read the full article by Roberts et al. (2017) on Taylor and Francis Online.
2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions, 2018
Latest Ottawa consensus statement. This notes that most research has been on the psychometric behaviour of selection instruments, calling for more discussion as to the strategic aims which should underpin development.
Read the full article by Patterson et al. (2018) on Taylor and Francis Online.
