Comparative evaluation of LLMs in orthopedic surgery

Aims/objectives: This study aimed to evaluate the performance of leading large language models (LLMs) in orthopedic surgery, with a focus on diagnostic accuracy, radiographic interpretation, subspecialty-specific performance, consistency, and gender bias. Unlike previous investigations that focused solely on ChatGPT, we assessed Claude-3-Sonnet, GPT-4o, Gemini-1.5, and Meta's LLaMA Vision-Instruct models against Orthopaedic In-Training Examination (OITE) questions. Methods: We tested each LLM on 2906 multiple-choice questions from the OITE question bank. Questions were categorized by subspecialty and by presence of images. Models provided answer choices, confidence scores (1–4), and justifications. Each question was administered three times to evaluate consistency. Statistical analyses included Z-tests, t-tests, ANOVA, chi-square tests, and logistic regression. Rasch transformation enabled comparison to PGY-1 and PGY-5 resident performance. Gender bias was evaluated based on performance differences across gender-specific cases. Results: GPT-4o achieved the highest accuracy (72.8 %), outperforming all other models. Performance improved with larger model sizes across all vendors. All models showed diminished accuracy on image-based questions (mean 10.9 % lower, p < 0.001). Confidence scores and triplicate agreement were associated with accuracy; their combination yielded the most reliable outputs (up to 83.4 % accuracy). Several models exhibited worse performance on female patient questions, indicating possible gender bias. Conclusion: LLMs demonstrate varying levels of accuracy in orthopaedic applications, with GPT-4o approaching PGY-5 performance, strictly in the context of standardized exam questions. Current models underperform on image-based questions and exhibit gender bias. Response confidence and consistency can flag reliable outputs. Continued development, bias mitigation, and fine-tuning with diverse, domain-specific datasets are essential for integration of LLMs into orthopaedic education and practice.