A new study published in United Kingdom Medical Journal Tested to see if artificial intelligence (AI) can pass the exam for the Royal College of Radiology (FRCR) Scholarship.
Radiologists in the UK (UK) must pass the FRCR exam before completing their training. Assuming that AI can pass the same tests, it can replace radiation experts. The final FRCR exam has three components and candidates are required to pass each component in order to pass the exam.
In the component of quick reporting, candidates must analyze and interpret 30 rays in 35 minutes and report at least 90% accurately to pass this part of the exam. This course measures candidates for accuracy and speed. There is an argument that AI will excel in accuracy, speed, radiation and binary results. Therefore, a quick reporting session of the FRCR exam can be a good setting to test AI capabilities.
Study: Can Artificial Intelligence Pass the Royal College of Radiology Scholarship Exam? A study of the accuracy of multiple reader judgments. Image Credit: SquareMotion / Shutterstock
About the study
In the current study, researchers evaluated whether AI candidates were able to pass the FRCR exam and perform better than the human radiologists who performed the same exam. The authors used 10 fake FRCR checks for analysis since RCR refused to share retired FRCR quick report review cases. Radiography was chosen to reflect the same level of difficulty as the actual exam.
Each mock exam has 30 rays covering all body parts from adults and children. About half have a pathology and the rest have no abnormalities. Previously successful FRCR candidates (Ray Readers) who have passed the FRCR exam in the last 12 months are selected through social media, word of mouth and email.
X-ray readers have completed a short survey that captures demographic information and previous FRCR exam attempts. Anonymous radiographs are provided through an online imaging platform (digital imaging and medical communication in DICOM). Radiologists were given one month (May 2022) to record their interpretations for the ten fake exams online.
Radiologists have given an assessment of 1) how the fake exam representation relates to the actual FRCR exam, 2) their performance, and 3) how they think AI will work well. Also, 300 anonymous rays were offered to the AI candidate called Smarturgences, developed by Milvue, a French AI company.
AI equipment is not specified in the analysis of radiographs, skeletal, abdominal and axial. However, it is provided with these radiographs for fairness throughout the participants. Scores for AI devices are calculated in four ways. In the first scenario, only AI interpretable radiographs were scored, excluding unexplained radiographs. Unexplained radiographs were usually scored as abnormal and erroneous in the second, third, and fourth scenarios.
A total of 26 radiologists, including 16 women, were recruited, most of whom were between 31 and 40 years old. Sixteen radiologists have completed their FRCR exams in the last three months. Most participants cleared the FRCR exam on their first attempt. The AI device will go through two test tests in the first scenario. In the second scenario, the AI will pass a sample check.
In the third and fourth scenarios, the AI candidate will fail the exam. The total accuracy and accuracy variability for AI is 83.6%, 75.2% and 79.5% in the first scenario, AI is the most competent candidate in one exam but ranks second until the last.
Assuming that the strict scoring criteria best reflect the actual examination, which is the case in the fourth scenario, the overall sensitivity, accuracy and accuracy of AI are 75.2%, 62.3% and 68.7%, respectively. In comparison, the scientists’ brief estimates of sensitivity, accuracy, and accuracy were 84%, 87.5%, and 85.2%, respectively.
No radiologist has passed all the fake exams. The highest-ranking radiologists passed nine exams, while the three lowest-ranking radiologists passed only one. On average, a radiologist can pass four tests. Radiologists rated the mock exam as more complex than the FRCR exam. They rated their performance 5.8 – 7.0 on a 10-point Likert-type scale and AI performance between 6 and 6.6.
“On this occasion, the artificial intelligence candidate could not pass any of the 10 fake exams when marked on strict criteria similar to human counterparts, but it can pass a number of fake exams,” the researchers said. “If a special transmission is made by the RCR to exclude images that it has not trained.”
Of the 42 unexplained radiographs in the AI candidate data set, one resulted for mislabeling the basal pneumothorax on a normal abdominal radiograph. More than half of the radiologists misdiagnosed 20 rays; Of these, the AI device correctly diagnosed 10 radiation but misinterpreted the rest. In total, almost all radiologists correctly analyzed 148 rays, of which 134 were correctly interpreted by AI candidates.
To summarize, the AI went through two mock exams when a special occupation was offered, namely the exclusion of unexplained images. However, AI will not pass if control is not granted. Although AI does not outperform radiation experts, its accuracy remains high, given the complexity and mix of cases.
In addition, AI ranks highest in a test, better than three radiation experts. It should be noted that the AI correctly diagnosed half of the X-rays that its peers misinterpreted. However, AI candidates still need more training to achieve the same level of performance and skills as the average radiologist, especially for cases where AI cannot interpret.