Dear Editor in Chief,
We read with interest the recent article by Guzelce et al. evaluating GPT 4 and GPT 4o on Turkish Board of Emergency Medicine (TBEM) sample questions.[1] While the study contributes a timely perspective to the growing literature on large language model (LLM) performance in medical education, several methodological concerns warrant consideration before broader interpretations are drawn.
Most critically, the entire evaluation rests on only 25 questions, a sample so small that a single correct or incorrect answer shifts performance by four percentage points. The observed accuracy difference between GPT 4 (80%) and GPT 4o (88%) was unsurprisingly nonsignificant (P = 0.45), yet the authors’ conclusion that both models “performed above the TBEM passing threshold” risks conveying an unwarranted confidence. With several domains represented by only one question each, the domain level error analysis presented in Figure 1 is statistically uninformative.
A less discussed but equally important concern involves the asymmetric testing conditions. GPT 4 processed image based questions using text descriptions embedded in question stems, while GPT 4o directly interpreted uploaded images via its multimodal architecture. The fact that both models achieved identical scores on all four image based questions does not validate GPT 4’s image comprehension; it may simply reflect that the text stems already contained sufficient diagnostic information, inadvertently removing the visual challenge entirely. This confound undermines any meaningful comparison between the two architectures on visual reasoning.
Furthermore, only two testing iterations per model were performed. This is insufficient to characterize response stability, especially for GPT 4, which demonstrated answer variability between runs. More rigorous consistency testing, across randomized prompt sequences and temperature settings, would be necessary before drawing conclusions about real world reliability.
Finally, the study’s exclusive use of publicly available official sample questions raises a training data contamination concern that the authors do not address. These questions may have been included in GPT’s pretraining corpus, artificially inflating observed accuracy beyond what the models would achieve on unseen examination content.
We commend the authors for addressing a regional gap in the artificial intelligence in medical education literature. However, we urge readers and educators to interpret these findings as preliminary signals rather than performance benchmarks. Future studies should employ larger, stratified, and potentially confidential question sets, standardized prompt protocols, and blinded scoring methodologies to establish robust conclusions about LLM utility in specialty board preparation.