Final CRQ Examination decision

Published: 04/10/2021

The College acknowledges and apologises for the additional stress and uncertainty to candidates who attempted the Final FRCA CRQ examination on 14 September 2021. After internal investigations and working with our online exam provider, TestReach, to understand more about what happened, we are now able to provide more context to the solution provided by the College. Please be assured that the steps to this solution entailed extensive scrutiny of the evidence and the options available to us, and the decision made was felt to be necessary and the fairest available.

As soon as the impact of the problem with the written paper became clear, a group of senior College Examiners and Council Members met to discuss possible solutions. This group included the College President; Chair and Vice-Chair of the Final FRCA; Chair of College Examinations Committee; the Chair of Training Board; Trainee representatives from Council, and senior officers at the College. In looking for the best solution, this group considered the welfare of candidates, the validity and reliability of the exam and the fairness and practicality of various alternatives.

Two facts were clear from the outset. Firstly, although the character limit restriction was resolved and the majority of candidates had their examinations restarted, some were not restarted with the character fix in place. These papers were reviewed but it was clear that they were not going to be a valid assessment of those candidates. Additionally, there was some variation in the extra time provided to those candidates who were restarted, and there were differences in the way the issue and solution was reported to candidates during the exam.

Secondly, the security of the exam had been compromised during the restarting of the examination after the character limit fix. We therefore concluded that the lack of consistency in test administration and breach in test security had impacted the reliability of the CRQ and it was no longer a valid tool to measure a candidate’s true ability.

Having reached that conclusion, it was apparent that no part of the CRQ could be marked for use as feedback or to contribute to the overall score on the written assessment, and therefore marking the CRQ for candidates who fail based on just the MCQ was also discounted as an option. The option of preparing a new CRQ and resitting all candidates was discussed. Discussions included writing a new paper or developing one from questions used previously.  The past paper option was not considered possible due to the relativeness newness of the CRQ examination form and limited question bank, which would impact our ability to provide a valid and secure exam. Both options would have taken some time to finalise, leaving candidates, who have already prepared long and hard, having to redouble their revision efforts for some weeks. The involved time delay would have removed the ability of pass candidates to progress to the Final SOE in December.

The above considerations led the group to the conclusion that the fairest, most reliable, and valid way forward, whilst also disadvantaging and delaying as few as possible, was to allow progress to the oral component of the Final FRCA based on the MCQ result alone.

The decision to base results on the MCQ was made after reviewing the raw data received from TestReach and on statistical modelling of results from past sittings to review potential impact on candidates and the ways of mitigating this impact.

We understand that some candidates are justifiably upset by these events. I hope that this explanation helps to assure you that the decision to proceed with the written exam based on the MCQ alone was taken with the best interests of candidates at heart and to ensure, as far as possible, validity, reliability, and fairness.

Dr Mark Forrest

Chair of the Examinations Committee