Response to the Independent Review of the Assessment Processes of the RCoA
We welcome the findings of an independent review of all examinations delivered by the Royal College of Anaesthetists, conducted by Professor John McLachlan and published in full today.
We will address the recommendations, many of which align closely with those of our recent internal review of the FRCA, working closely with the Faculties of Pain and Intensive Care Medicine. We are committed to delivering fair, robust and high-quality examinations that are aligned with GMC standards and best contemporary practice and that uphold and improve standards of patient care. The reviews will inform our developmental roadmap, and we will implement changes with care over the coming months and years, taking into account the needs of candidates and consulting with stakeholders.
We understand that the FRCA exams are high stakes assessments that have a significant impact on candidates’ personal and professional lives. For that reason, it was essential for the review to capture the experiences and opinions of our members and other stakeholders, particularly those of current and past candidates. Their input in the review process has been invaluable, as has that of our examiners. We are grateful to everyone who has participated in the review process and thank them for their openness and honesty. We also extend our thanks to Professor McLachlan for undertaking the review.
While it is likely that changes will be made to the FRCA exam format in future, the current FRCA examinations remain valid, reliable, fit for purpose and are approved by the GMC. We will ensure exam candidates are given at least 12 months’ notice of any significant changes to the FRCA exams in order to facilitate good preparation and support. Therefore, anaesthetists in training and prospective candidates should not change the way they prepare for the current sets of exams until formally notified of any changes.
Scope of the independent review
We commissioned the review to provide independent assurance and insights into the development and delivery of our assessment processes, alongside our regular internal reviews of the FRCA. The review considered all aspects of the FRCA, FFICM and FFPMRCA exams, including an error in the delivery of the September 2021 FRCA Final Written Exam and an unexpectedly low pass rate for the October 2021 FFICM Objective Structured Clinical Examination (OSCE).
We have published a summary of our proposed timetable for implementation of the changes resulting from the review, and we highlight the main themes relevant to the FRCA examinations here. We will keep our members and the GMC informed of our progress through regular updates on our website and member communications.
Response to the September 2021 FRCA Final Written Exam
The review concluded that the error in the delivery of the September 2021 FRCA Final Written Exam was derived in large part from the stresses of converting written assessments to online versions due to the pandemic, and that such errors are unlikely to occur again. It also found that actions taken by the College to mitigate the effects were as reasonable as possible in the circumstances.
We understand that this does not lessen the impact of the errors that were made, and we apologise again for the difficulties and stress caused to candidates. We are fully committed to ensuring errors do not reoccur. In line with Professor McLachlan’s recommendations, we have taken action to invest in additional capacity in our exams team and made improvements to our pre-delivery processes for written exams. We will make further improvements to the FRCA in terms of resourcing and our platforms, processes and support for candidates, as outlined below.
Our programme of improvement
We are committed to pursuing excellence in all that we do, including the governance and delivery of the FRCA and Faculty examinations. We fully endorse the recommendations from Professor McLachlan in relation to examiner culture, including ensuring our codes of conduct are a key part of examiner training and continuing our efforts to increase the diversity of examiner pools.
In considering the recommendations from both the independent review and our internal review, we will:
- Invest additional resources in our examinations function to implement changes arising from the review and strengthen our capacity to deliver exams. We have established an Examinations Development and Assurance Group (EDAG) to lead the development, quality and alignment of exam processes across the FRCA and Faculty exams, where appropriate. This group will have an independent chair and comprise examiners from all exams delivered by the College, members of the exams team, lay and trainee representatives and a specialist educational assessment expert.
- Give candidates and anaesthetists in training a greater role in our assessment processes, including more representation on College committees engaged in assessment. These representatives will also play a central role in supporting communication with candidates.
- Review the purpose of our assessments in accordance with the changes made in the curriculum and the changing nature of clinical practice in our specialties. We will consult with members and stakeholders to define this.
- Implement a unified approach to the use of, and training in, standard-setting across the FRCA exams and all assessments delivered by the College.
- Undertake new research to inform our assessment design process. We are committed to investigating the impact of gender, ethnicity and educational background on exam performance. We also aim to undertake validity research to compare exam performance with performance in the workplace as estimated by trainers and supervisors with performance in RCoA assessments.
We understand that members may have questions about the independent review, and the action we are taking as a result. We welcome your feedback and encourage you to attend our Let’s Talk member engagement online event on Wednesday 8 March at 19.00.
Members can also share their views with us using our online form. This feedback will be shared with our Examinations Development and Assurance Group.
Dr Fiona Donald
President, Royal College of Anaesthetists
Frequently asked questions
Candidates and trainers do not need to change their revision/ study plans for the current exams. While it is likely that changes will be made to the exams formats in future, all our current examinations remain valid, reliable, fit for purpose and are approved by the GMC. We will ensure exam candidates are given at least 12 months’ notice of any significant changes to exams in order to facilitate good preparation and support. Prospective candidates should not change the way they prepare for the current sets of exams until formally notified of any changes.
The College and Faculty exams continue to be valid, reliable and fit for purpose. The structure, format and standard setting methods for these exams have been approved by our regulator, the General Medical Council (GMC), and we have maintained close contact with them throughout the review process. Any changes we make to these aspects of the exams will require approval from the GMC.
Best practice in examinations changes over time in line with research in assessment, assessment theory and changes in medical education. This is why our examinations are reviewed internally every 3 – 5 years to ensure they continue to reflect best practice, the needs of patients and the needs of the healthcare system. We are committed to the continuous improvement of our examinations, and the recommendations from both the internal review of the FRCA and the independent review conducted by Professor McLachlan provide us with a roadmap for further development and improvement.
We very much welcome trainee and candidate participation in our exams. Trainee representatives are present on the Examinations Committee as well as College Council and Faculty Board. The Examinations Department also engages regularly with our Anaesthetists in Training Committee, the Anaesthetists in Training Representative Group (ATRG) and the FICM Specialty Registrars Sub-Committee (StRSC) with updates on exams. Trainees have also supported us in the development of training materials for examiners and for exam candidates. All candidates receive a post exam survey to feedback on examination delivery and content and this feedback is reviewed by the operational teams for Primary, Final and the Faculties, and during moderation and item analysis meetings, as is any ad hoc feedback received.
We will provide updates on changes to the exams by email to the membership as well as on our website and social media channels. Regular updates to the Anaesthetists in Training Committee, the Anaesthetists in Training Representative Group (ATRG) and the FICM Specialty Registrars Sub-Committee (StRSC) will ensure that representatives from these groups also support the feedback of changes to trainees and our wider membership.
The assessment blueprints for the FRCA examinations are located in the core level and intermediate training syllabuses for Primary and Final (core and intermediate) on the RCoA website. Revised syllabuses for the 2021 curriculum will be available from summer 2023 and relevant key capabilities from the 2021 curriculum are already referenced in feedback to candidates for the written examinations.
The assessment blueprint for the new 2021 ICM curriculum: Supporting Excellence is available on the FICM website. To help candidates with their exam preparation, a detailed syllabus is also provided.