Comparative Evaluation of Four Large Language Models in Turkish Dentistry Specialization Exam

Ömer Ekici

Makale detayları

Comparative Evaluation of Four Large Language Models in Turkish Dentistry Specialization Exam

Yazarlar:

Ömer Ekici

Yayıncı:

Selcuk University

Ana Başlıklar:

Dentistry (Other)

Dergi:

Selcuk Dental Journal Vol.12 Issue 4, pp.6-10

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001358546

pISSN:

2148-7529

Anahtar kelimeler:

Özet:

 Background The aim of the study is to evaluate the performance of four leading Large Language Models LLMs in the 2021 Dentistry Specialization Training Exam DSE. Methods A total of 112 questions were used, including 39 questions in basic sciences and 73 questions in clinical sciences, which did not include the figures and graphs asked in the 2021 DSE. The study evaluated the performance of four LLMs: Claude-3.5 Haiku, GPT-3.5, Co-pilot, and Gemini-1.5. Results In basic sciences, Claude-3.5 Haiku and GPT-3.5 answered all questions correctly by 100, while Gemini-1.5 answered by 94.9 and Co-pilot by 92.3. In clinical sciences, Claude-3.5 Haiku showed an overall correct answer rate of 89, Co-pilot 80.9, GPT-3.5 79.7 and Gemini-1.5 65.7. For all questions, Claude-3.5 Haiku showed a correct answer rate of 92.85, GPT-3.5 86.6, Co-pilot 84.8 and Gemini-1.5 75.9. While the performance of LLMs in basic sciences was similar p0.134, there was a statistically significant difference between the performances of LLMs in clinical sciences and all questions p0.007 and p0.005, respectively. Conclusion In all questions and clinical sciences, Claude-3.5 Haiku performed best, Gemini-1.5 performed worst, and GPT-3.5 and Co-pilot performed similarly. The 4 LLM models examined showed a higher success rate in basic sciences than in clinical sciences. The results showed that AI-based LLMs can perform well in knowledge-based questions such as basic sciences but perform poorly in questions that require knowledge as well as clinical reasoning, discussion, and interpretation, such as clinical sciences. Keywords Artificial intelligence, Dentistry, Dentistry specialization training, Large language model.

 Background The aim of the study is to evaluate the performance of four leading Large Language Models LLMs in the 2021 Dentistry Specialization Training Exam DSE. Methods A total of 112 questions were used, including 39 questions in basic sciences and 73 questions in clinical sciences, which did not include the figures and graphs asked in the 2021 DSE. The study evaluated the performance of four LLMs: Claude-3.5 Haiku, GPT-3.5, Co-pilot, and Gemini-1.5. Results In basic sciences, Claude-3.5 Haiku and GPT-3.5 answered all questions correctly by 100, while Gemini-1.5 answered by 94.9 and Co-pilot by 92.3. In clinical sciences, Claude-3.5 Haiku showed an overall correct answer rate of 89, Co-pilot 80.9, GPT-3.5 79.7 and Gemini-1.5 65.7. For all questions, Claude-3.5 Haiku showed a correct answer rate of 92.85, GPT-3.5 86.6, Co-pilot 84.8 and Gemini-1.5 75.9. While the performance of LLMs in basic sciences was similar p0.134, there was a statistically significant difference between the performances of LLMs in clinical sciences and all questions p0.007 and p0.005, respectively. Conclusion In all questions and clinical sciences, Claude-3.5 Haiku performed best, Gemini-1.5 performed worst, and GPT-3.5 and Co-pilot performed similarly. The 4 LLM models examined showed a higher success rate in basic sciences than in clinical sciences. The results showed that AI-based LLMs can perform well in knowledge-based questions such as basic sciences but perform poorly in questions that require knowledge as well as clinical reasoning, discussion, and interpretation, such as clinical sciences. Keywords Artificial intelligence, Dentistry, Dentistry specialization training, Large language model.

 Background The aim of the study is to evaluate the performance of four leading Large Language Models LLMs in the 2021 Dentistry Specialization Training Exam DSE. Methods A total of 112 questions were used, including 39 questions in basic sciences and 73 questions in clinical sciences, which did not include the figures and graphs asked in the 2021 DSE. The study evaluated the performance of four LLMs: Claude-3.5 Haiku, GPT-3.5, Co-pilot, and Gemini-1.5. Results In basic sciences, Claude-3.5 Haiku and GPT-3.5 answered all questions correctly by 100, while Gemini-1.5 answered by 94.9 and Co-pilot by 92.3. In clinical sciences, Claude-3.5 Haiku showed an overall correct answer rate of 89, Co-pilot 80.9, GPT-3.5 79.7 and Gemini-1.5 65.7. For all questions, Claude-3.5 Haiku showed a correct answer rate of 92.85, GPT-3.5 86.6, Co-pilot 84.8 and Gemini-1.5 75.9. While the performance of LLMs in basic sciences was similar p0.134, there was a statistically significant difference between the performances of LLMs in clinical sciences and all questions p0.007 and p0.005, respectively. Conclusion In all questions and clinical sciences, Claude-3.5 Haiku performed best, Gemini-1.5 performed worst, and GPT-3.5 and Co-pilot performed similarly. The 4 LLM models examined showed a higher success rate in basic sciences than in clinical sciences. The results showed that AI-based LLMs can perform well in knowledge-based questions such as basic sciences but perform poorly in questions that require knowledge as well as clinical reasoning, discussion, and interpretation, such as clinical sciences. Keywords Artificial intelligence, Dentistry, Dentistry specialization training, Large language model.

 Amaç Çalışmanın amacı 2021 Diş Hekimliği Uzmanlık Eğitimi giriş sınavında DUS önde gelen dört Büyük Dil Modeli LLM nin performansını değerlendirmektir. Gereç ve Yöntemler 2021 DUS sınavında sorulan şekil ve grafik içermeyen temel bilimlerde 39 soru ve klinik bilimlerde 73 soru olmak üzere 112 soru kullanıldı. Çalışmada Claude 3.5 Haiku, GPT-3.5, Copilot ve Gemini 1.5 olmak üzere dört LLMnin performansı değerlendirildi. Bulgular Temel bilimlerde Claude-3.5 Haiku ve GPT-3.5 tüm soruları 100 doğru cevaplarken, Gemini 1.5 94,9 ve Copilot 92,3 oranında cevapladı. Klinik bilimlerde toplamda Claude 3.5 Haiku 89, Copilot 80,9, GPT-3.5 79,7 ve Gemini 1.5 65,7 doğru cevap oranı sergiledi. Tüm sorularda ise Claude 3.5 Haiku 92,85, GPT-3.5 86,6, Copilot 84,8 ve Gemini 75,9 doğru cevap oranı gösterdi. Temel bilimlerde LLMlerin performansı benzer iken p0.134, klinik bilimlerde ve tüm sorularda LLMlerin performansları arasında istatistiksel açıdan anlamlı farklılık görüldü sırasıyla p0.007 ve p0.005. Sonuç Tüm sorularda Claude 3.5 Haiku en iyi performansı gösterirken, Gemini 1.5 en kötü performansı gösterdi, GPT 3.5 ve Co pilotun performansı benzer bulundu. İncelenen 4 LLM modeli temel bilimlerde klinik bilimlere göre daha yüksek bir başarı oranı gösterdi. Sonuçlar, yapay zeka tabanlı LLMlerin temel bilimler gibi bilgiye dayalı sorularda iyi performans sergileyebileceğini ancak klinik bilimler gibi bilgi ile birlikte klinik muhakeme, tartışma ve yorum gerektiren sorularda daha düşük performans sergilediğini gösterdi. Anahtar Kelimeler Yapay zeka, Diş Hekimliği, Diş Hekimliği uzmanlık eğitimi, Büyük dil modeli.

Makale detayları

Comparative Evaluation of Four Large Language Models in Turkish Dentistry Specialization Exam

Yazarlar:

Ömer Ekici

Yayıncı:

Selcuk University

Ana Başlıklar:

Dentistry (Other)

Dergi:

Selcuk Dental Journal Vol.12 Issue 4, pp.6-10

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001358546

pISSN:

2148-7529

Artificial intelligence

Dentistry

Dentistry specialization training

Large language modelYapay zeka

Diş Hekimliği

Diş Hekimliği uzmanlık eğitimi

Büyük dil modeli.

Sayı

Makale detayları

Yazarlar: Ömer Ekici

Yayıncı: Selcuk University

Ana Başlıklar: Dentistry (Other)

Dergi: Selcuk Dental Journal Vol.12 Issue 4, pp.6-10

Belge Tipi: Makale

Yayın Tarihi: 9/2025

Belge Dili: İngilizce

Document ID (Org No.): 2001358546

pISSN: 2148-7529

Sayı

Yazarlar:

Ömer Ekici

Yayıncı:

Selcuk University

Ana Başlıklar:

Dentistry (Other)

Dergi:

Selcuk Dental Journal Vol.12 Issue 4, pp.6-10

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001358546

pISSN:

2148-7529