Predictors of ophthalmology career success (POCS) study
•,,.
...
Abstract
Objective Ophthalmology is the busiest outpatient specialty with demand predicted to rise over 40% in the next 20 years. A significant increase in the number of trainee ophthalmologists is required to fill currently vacant consultant posts and meet the UK’s workforce demands by 2038. Our aim was to understand what determines success in ophthalmology training, in order to inform future ophthalmologists, refine recruitment and facilitate workforce planning.
Methods and Analysis This was a retrospective longitudinal cohort study using routinely collected data available from UK Medical Education Database (UKMED) (https://www.ukmed.ac.uk/). Data were analysed on 1350 candidates who had applied for ophthalmology specialty training (OST) between 2012 and 2018, as well as 495 candidates who had attempted Fellow of the Royal College of Ophthalmologists (FRCOphth) Part 1 between 2013 and 2018. Participants who had not obtained their primary medical qualification from the UK medical schools were excluded. Primary outcome measures included gaining a place on the OST programme and passing the FRCOphth Part 1 examination on first attempt.
Results Higher education performance measure decile scores at medical school are strongly predictive in securing an OST post and passing the part 1 examination first time (p<0.001). Candidates who attempt FRCOphth Part 1 prior to their ST1 application are more likely to get a place on OST on first attempt. Socioeconomic factors, gender and ethnicity do not influence success in OST entry. Male trainees are more likely to pass FRCOphth Part 1 on their first attempt.
Conclusion This study is the first quantitative assessment of the factors that determine success in OST recruitment and ophthalmology postgraduate examinations in the UK. Similar studies should be undertaken in all other medical and surgical specialties to understand what factors predict success.
Key messages
What is already known about this subject?
Candidates choosing ophthalmology as a career will demonstrate a high level of commitment prior to their application.
What are the new findings?
Academic performance at medical school is predictive of success in gaining a place on ophthalmology specialty training (OST) and passing the part 1 fellowship examination on first attempt.
Attempting the Fellow of the Royal College of Ophthalmologists (FRCOphth) Part 1 examination increases the chances of getting a place on the OST training programme on the first application. Socioeconomic factors, ethnicity and gender do not appear to influence success in OST entry. Males are more likely to pass the FRCOphth Part 1 on the first sitting.
How might these results change the focus of research or clinical practice?
The predictors of ophthalmology career success (POCS) study provides evidence that can help to positively influence recruitment and the examination process for future trainees and postgraduate bodies.
Introduction
Ophthalmology is the busiest outpatient specialty in the National Health Service. The Royal College of Ophthalmologists (RCOphth) has predicted that demand for specialist eye care will rise by 30%–40% over the next 10 years.1 The RCOphth also reported a pressing need to train more doctors to meet the current ‘severe’ national shortage of ophthalmologists.1 2
While the demand for more trainee ophthalmologists poses an immediate challenge to the health system, it presents an opportunity for us to reflect on the state of the current ophthalmology specialty training (OST) recruitment process. In particular, understanding what determines success in ophthalmology will help to support future OST applicants, refine recruitment and facilitate workforce planning.
OST has long been considered an attractive, but highly competitive career choice. In 2019, there were 356 applications for 110 ST1 Ophthalmology posts, rendering a competition ratio of 3.24.3 As a run-through specialty, it leads directly to a certificate of completion of training, thus providing a streamlined route to consultancy. Candidates are enticed by the combination of medicine and surgery and moreover, ophthalmology lends itself to a more desirable work–life balance than other surgical specialties.4 In a US survey, ophthalmologists demonstrated high career satisfaction rates, with 93% stating that they would rechoose their own specialty.5 Unlike other surgical specialties, ophthalmology attracts a high number of female doctors. Notably, 31% of consultants in ophthalmology are female; almost three times higher a figure than that reported by the Royal College of Surgeons.1 6
In the UK, ophthalmology trainees are selected through a national recruitment programme. This selection process is comprised of an interview and portfolio review. The interview features a critical appraisal, clinical scenario, improvement of patient care and communication station. A recent addition to the interview process is a multispecialty recruitment assessment. A maximum of 200 points can be gained from the interview and 100 points from the portfolio review, in which candidates are formally scored on their experience of teaching, audit, research, prior examination success and commitment to the specialty.
The FRCOphth Part 1 is the first of four RCOphth mandated examinations required to gain the Fellowship in Ophthalmology (Fellow of the Royal College of Ophthalmologists (FRCOphth)). During the period of data collection for this study, the scoring criteria for the OST selection process was revised. Between 2012 and 2015, the raw score for the FRCOphth Part 1 was 3 points, and 2 points in 2016. From 2017 to 2018, candidates were again awarded a score of 3 points for successful completion of the examination. All scores within the portfolio were doubled for 2018 due to the introduction of digital scoring, so it did not change the weighting of the examination.7
This study aimed to explore the factors that lead to success in securing an OST post and passing postgraduate examinations on first attempt. It is anticipated that this study will encourage other specialties to adopt a similar methodology to understand the predictors of success within their respective fields.
Methods
Design
This was a retrospective longitudinal cohort study of graduates from all medical schools in the UK. The study used routinely collected data available from UK Medical Education Database (UKMED) (https://www.ukmed.ac.uk/).
UKMED8 is a medical education research database that collates data on UK medical students and trainee doctors, in order to highlight the path of doctors through school, university and their career. A wide range of data is collected by UKMED, including candidates’ demographic details, examination results and annual review of competency progression (ARCP) outcomes. The development and rationale for UKMED has previously been described by Dowell et al.9
Study population
Online supplemental figure 1 shows the flow of data through the study. In line with UKMED statistical disclosure controls, all numbers are reported to the nearest multiple of 5.10 For the analysis of applications to OST, the cohort was defined as anyone who made one or more applications to OST on the ORIEL recruitment system between 2012 and 2018 and had their application(s) in the UKMED table ORIEL_RECRUIT_OUTCOMES.8 ORIEL is the system used to process applications to specialty training in the UK. It is managed by Health Education England on behalf of the four UK nations.11
Data analysis was undertaken on 1350 candidates who obtained their primary medical qualification (PMQ) from UK medical schools. This was because comprehensive demographic records and postgraduate performance information were only available for the UK cases.
For analysis of the membership examinations, the cohort included anyone who had taken the FRCOphth Part 1 between 1 August 2013 and 31 July 2018; the period covered by the General Medical Council (GMC)’s postgraduate examination data collection and held in the UKMED table EXAM_TOTAL_SCORES.8 UKMED postgraduate examination data are left censored, and examination attempts occurring before 1 August 2013 are not present. To conduct further analysis, only first attempts were taken from those with a UK PMQ. The dataset contained 655 cases who had attempted the part 1 examination and had this recorded as their first attempt, of which 495 had obtained their PMQ in the UK.
Data in UKMED are held at the level of individual applications. Our unit of analysis was the applicant. We summarised the recruitment outcomes for each specialty and year the doctor applied to specialty training. Therefore, for each year of application, we could determine which specialties the doctor applied to and whether they were offered a place on the given training programme in that application year. The year of application for each individual was ranked, and the case with the first rank (ie, the earliest year) was taken as the first application. We used the first application to OST, which for 1215 candidates (90% of 1350) was in the same application year as their first application to any specialty training programme.
Statistical analysis
Source data were retrieved from the UKMED database and an SPSS file was created, with one row per person. Analysis was conducted using SPSS V.26. Univariate analysis was initially conducted to understand the relationship between demographic, educational and trainee behaviour predictor variables and the outcomes of interest. Variables that were statistically significant (after applying a Bonferroni correction for multiple testing) were then included in a logistic regression model. This was to understand which variables independently predicted the outcomes of interest, after controlling for the other variables included in the model.
Data management—measures
Socioeconomic and demographic measures
The following variables were included: sex, ethnicity, higher education statistics agency (HESA) disability summary, indices of multiple deprivation (IMD) quintile, participation of local areas (POLAR) quintile, bursary (for taking UKCAT test), bursary and parental education and SEC combined or graduate on entry. SEC is the socioeconomic background of the student’s parent, step-parent or guardian who earns the most. For students over 21, it comes from their own occupation. The UK Clinical Aptitude Test (UKCAT) is an admissions test used by certain UK Universities for their medical and dental degree programmes.
UKMED includes measures that are derived from the postcode on application to medical school. IMD quintiles are created by ranking each small area within the four nations, with a lower score indicating greater deprivation. These scores are put into quintiles with 1 being the most deprived. POLAR quintile is a measure of how many 18 year olds from an area started a higher education course during a specified time period. Polar quintiles range between 1 and 5, with one being the lowest score and 5 the highest. For both IMD and POLAR quintiles, the reference data closest to the proceeding year of the student commencing medical school were used.8
Academic measures
Examination relative to application
The year of first application was obtained and the date of the first attempt at FRCOphth Part 1 was compared with the deadline date for applications for that year. For example, for 2014, the OST application deadline was 1 November 2013. If a candidate took the examination before that date they were classified as taking it before their OST application. The November date was based on the documentation in the Health Education England Applicant Handbook provided each year. As examination data were not available before 1 August 2013, this variable was only available for those applying for the first time from 2014 onwards (N=970).
Only applied to OST
If the applicant had no applications to specialties other than OST in the ORIEL dataset, in the first year they applied to OST they were classified as only having applied to OST.
Educational performance measure (EPM) score
Students in a graduating cohort are ranked on their medical school performance through their EPM. Medical schools decide which assessments to include in the EPM; these assessments must meet specific criteria and are published on the school’s website.12 In 2012, foundation programme applicants were ranked by their medical schools into EPM quartiles, whereas applicants from 2013 onwards were ranked into deciles. To allow cases from 2012 and later years to be combined in one analysis, EPM quartiles and deciles were converted into normal deviate scores, the approach taken by Garrud and McManus.13 Normal deviate scores were used as means of placing quartiles and deciles on the same scale. The midpoints for each relevant section under the normal curve were taken to obtain the Z-score. For example, if the first 25% of cases occur under the part of the curve that has a midpoint of −1.0491, so the bottom 12.5% of cases would be to the left of −1.0491.
Programme specialty at the time of the examination
This was derived by linking the examination data file to the national trainee survey (NTS) data. Data were linked to the NTS record that fell within the year of the examination return. For example, if the examination date was between the 1 August 2017 and 31 July 2018, the 2018 NTS record was used to define the candidate’s specialty and training level at the time of the examination. This is the approach used by the GMC’s progression reports.14
We used the Strengthening the Reporting of Observational Studies in Epidemiology cohort checklist when writing our writing report.15
Results
Success in obtaining an OST programme place on first attempt
Univariate analysis was undertaken for the 1350 graduates of the UK medical schools who applied to OST, looking at each demographic variable of interest. Overall, 37.8% of UK applicants were successful on their first application in the period of interest (2012–2018). Online supplemental table 1 details the results of this analysis. The following variables were significant at p<0.05 after applying the Bonferroni correction for running 18 tests:
Applicants attempting the FRCOphth Part 1 prior to commencing training were more likely to be offered a place.
Applicants only applying to OST were more likely to be offered a place.
Applicants applying in 2012 were more likely to be offered a place.
Younger applicants were more likely to be offered a place.
Applicants from certain medical schools were more likely to be offered an OST post, for example, applicants from Cambridge were particularly successful.
There was no difference in success in obtaining an OST post on first attempt for many of the variables assessed. These included sex, ethnicity, HESA disability summary, IMD quintile, POLAR quintile, SEC combined or graduate on entry, bursary (for taking UKCAT test), bursary, parental education, course type or intercalation. These variables are defined in the UKMED data dictionary – VW_UKMED_PERSON_FULL table.8
Those who were offered a place on an OST programme had a higher mean EPM normal deviate score than those who were not offered a place (0.370, (N=560) compared with −0.236, (N=295)). F=101.633, p<0.001.
Variables that were statistically significant were used in a multivariate logistic regression model presented in table 1. Younger applicants were more likely to be successful. An increase of 1 in the EPM normal deviate score equated to a 2.5 times greater likelihood of being offered a place on OST. Attempting the part 1 examination prior to application meant that someone was 2.6 times more likely to be offered a place on OST. These differences were present after controlling for medical school. Medical school was significant overall when controlling for the other variables included in the model, but there was no significant effect for any individual school. Only applying to OST and the year of first application were not significant predictors, once included in a multivariate model.
Table 1
|
Logistic regression predicting offer at first application to OST
On the first application, 42% (565/1350) applied only to OST. Overall, 785 candidates applied to multiple specialties in their first year of application to OST. The most common applications in their first year of any application to specialty training, which was not always OST, were to the following specialties: general practice (GP) (385), core surgical training (285), core medical training (235) and clinical radiology (150). Note that the first year of application to OST was the first year of application to any specialty training in 90% of cases (N=1350), 10% of those in the study applied to other specialties in recruitment years before their first application to OST.
Results in FRCOphth Part 1
Online supplemental table 2 gives univariate results. The following groups had higher pass rates on the FRCOphth Part 1: men and those who took the examination while in an ophthalmology training programme. White candidates were close to significance at p=0.052 following Bonferroni correction.
Those who passed the examination on first attempt had a higher mean EPM normal deviate score than those who did not pass (0.463, (N=165) compared with −0.172 (N=215)). F = 55.376, P <0.001.
Variables that were statistically significant or close to significance were used in the multivariate logistic regression model presented in table 2. As was the case with entry into OST, an increase of one in the EPM normal deviate score suggested that a candidate was 2.5 times more likely pass the FRCOphth Part 1 on first attempt. Men were 2.5 times more likely to pass on first attempt and ethnicity was not a significant factor, when included in a multivariate model. Candidates who were already on an OST programme when they took the examination were more likely to pass on first attempt, after controlling for EPM deviate score, compared with those not in OST.
Table 2
|
Logistic regression predicting pass at first attempt of FRCOphth Part 1
Discussion
Success on application to OST
Academic performance
High educational performance measure (EPM) scores at medical school are strongly predictive in securing an OST programme place and passing the FRCOphth Part 1. It is known that trainees who pursue more competitive specialties tend to have higher EPM scores.16 At present, EPM is used by foundation schools for recruitment purposes, but not by specialty training bodies. This is for a number of reasons; EPM scores are only available for UK applicants and they are not consistently derived across all the UK medical schools. Unlike the United States Medical licencing examination, which is taken by all medical students, there is no national standardisation to the EPM.
EPM is known to be a significant predictor of success in completing foundation training.17 In addition, medical school academic outcomes are linked to performance on the Membership of the Royal College of Physicians (MRCP) examination, and being on the GMC Specialist Register.18 Postgraduate schools may therefore wish to provide some weighting for the EPM in the recruitment process for UK graduates. To eliminate institutional bias, it may be possible in the future to adjust the EPM to make it more nationally comparable using ‘peer competition rescaling’.19
Medical school attended
This study showed an association between medical school and first time OST entry at the univariate level only. In the univariate analysis, there was an association between applicants from the universities of Cambridge, Oxford and Imperial and higher rates of first time OST entry. This trend disappeared at the multivariate level. Interestingly, McManus et al found that schools that taught more general practice did have more graduates entering GP training. However, increased medical school teaching of psychiatry, surgery and anaesthetics did not result in more specialist trainees.20
Applicant behaviour
Commitment to the specialty is an important predictor of success at gaining a place on OST. Univariate analysis found that candidates applying for OST only and those who attempted FRCOphth Part 1 prior to their OST application were more likely to gain entry on the first attempt. Multivariate analysis only found that candidates who took the FRCOphth Part 1 before applying were more likely to be successful. This is because applying to OST only and taking the examination early were related; 61% of those applying to OST only had attempted the examination compared with only 35% of those applying to other specialties. Overall, these findings are in keeping with the literature, which reports that doctors who choose to pursue ophthalmology generally demonstrate interest early on in their careers.21
Candidates in our study who applied to OST alone may have had more conviction about their career choice, as demonstrated by choosing to attempt the FRCOphth Part 1 examination prior to application. Although FRCOphth Part 1 is not a prerequisite for OST entry, the RCOphth allocates points for its successful completion in the OST application process.7 Our analysis included OST applications from 2012 to 2018. Attempting the FRCOphth Part 1 prior to OST application was associated with first time entry into OST, regardless of the application criteria at the time at which the candidate applied.
Demographic factors
In this cohort, younger applicants were more likely to be accepted onto OST on first attempt. Older candidates may have taken longer to make a career choice or have had difficulties progressing through their education or training.
This study evaluated the influence of socioeconomic factors, ethnicity and gender on OST entry. No significant association was found between socioeconomic factors, gender, ethnicity and entry to OST.
That there were no sex differences in success agrees with Woolf et al, who only found a sex difference for GP and paediatrics, with women being more likely to be appointed.22
Ethnicity did not influence success in this cohort; a finding that is not in keeping with the wider literature. For example, Kumwenda et al found that applicants from white ethnic backgrounds were significantly more likely to be allocated to a higher choice foundation school than black or Asian applicants.23
The lack of association between socioeconomic factors and success at being offered an OST place may be due to an element of self-selection. Other researchers have found that doctors from families where neither parent was educated to degree level were less likely to pursue a hospital specialty and more likely to choose general practice, compared with their peers with a family background in higher education. Moreover, private school educated doctors were 1.8 and 1.4 times more likely to train in surgical or medical specialties (relative to general practice), respectively, than those who attended a state school.24 This study did not analyse if those applying to OST were demographically different to those applying to other specialties, as our dataset was restricted only to candidates who applied to OST.
Examination success
Demographic factors
In this cohort, sex was associated with examination success, but ethnicity was not. Male trainees were 2.5 times more likely to pass FRCOphth Part 1 on their first attempt; a finding which is reflected in other specialties. Males sitting the Part A Membership of the Royal College of Surgeons examination were almost three times as likely to pass than females.25 McManus et al reported that men do better on part 1 and part 2 of the MRCP(UK) examinations.26
Woolf et al have previously published widely on the phenomenon of differential attainment. Their 2011 meta-analysis showed that ethnic minority medical graduates in the UK had 2.5 times higher odds of failing examinations compared with their white peers.27 However, no association was found between ethnicity and passing FRCOphth Part 1 on first attempt in this study’s cohort (when included in the multivariate model that adjusts for EPM deciles).
Candidate behaviour
Candidates who were already in OST were more likely to pass the FRCOphth Part 1 examination first time. Based on this data, it could be argued that it may not be in trainees’ best interests to take the examination before starting OST, given the cost and time requirement. Indeed, the FRCOphth Part 1 is known to be one of the most expensive of the common postgraduate examinations.28 It is also considered to be a very difficult examination to pass with an average overall pass rate of 46%.29
However, this is offset by this study’s previously stated finding; that candidates who had previously taken FRCOphth Part 1 were more likely to get an OST place on first attempt. Overall, it would appear that although FRCOphth Part 1 is more difficult to pass outside of training, attempting the examination is more likely to assist with first time OST entry.
Academic performance
As with selection into OST, EPM deciles were highly predictive of passing the part 1 examination on first attempt. As McManus et al note, there is an academic backbone to medicine and those that do well academically at the start of the careers generally continue to be successful.30
Strengths and limitations
This study is the first of its kind to evaluate the factors that predict success in ophthalmology training and has applicability to other specialties. A key strength of this study is that it is based on a complete dataset of all OST applications within the UK from 2012 to 2018.
For the purpose of this study, successful candidates are defined as those who gain a place on OST or pass postgraduate examinations on the first attempt. Indeed, first time achievement is not an absolute predictor of long-term success and many doctors do not succeed on first attempt. It is also understood that the self-selected nature of our cohort may account for the lack of correlation between socioeconomic factors and success at gaining a place on OST.
This study has not explored motivations for applying to OST over other specialties using the career intention items in the NTS. Nor has it explored whether the quality of a candidate’s ophthalmology training or designated deanery are associated with passing FRCOphth Part 1 on first attempt.
Conclusion
A number of factors that predict success in gaining a place in OST and in passing postgraduate examinations on the first attempt were identified. Subsequent studies will continue to follow this cohort through OST to understand how trainees perform in later examinations, including the refraction examination and FRCOphth Part 2. It is also possible to monitor ARCP outcomes and see whether certain trainees take longer to complete their training. Whether or not early academic achievement influences long-term success on the OST programme will be an interesting finding for the RCOphth and other Royal Colleges. Finally, it is hoped that UKMED studies such as this can help to understand whether the quality of OST training affects a trainee’s career progression and examination success. Quality of training may be measured by a number of factors, including the level of clinical and educational support received, protected teaching time and opportunities for continuous professional development. These measures are available on the GMC’s annual NTS survey.31
UKMED data are refreshed annually, and it would be possible to run this analysis each year as the cohort size increases. Current and prospective OST candidates can learn from previous UKMED data trends in order to decide when to attempt examinations and to help make informed career choices. Periodic analysis of UKMED OST applicant and trainee data is also important to ensure that recruitment processes and examination success remain fair and equitable in subsequent years.
