Artificial intelligence meets medical expertise: evaluating GPT-4's proficiency in generating medical article abstracts

Ergin Sağtaş; Furkan Ufuk; Hakkı Peker; Ahmet Baki Yağcı

Makale detayları

Artificial intelligence meets medical expertise: evaluating GPT-4's proficiency in generating medical article abstracts

Yazarlar:

Ergin Sağtaş | Furkan Ufuk | Hakkı Peker | Ahmet Baki Yağcı

Yayıncı:

Pamukkale University

Ana Başlıklar:

Radyoloji ve Organ Görüntüleme

Dergi:

Pamukkale Tıp Dergisi Vol.17 Issue 4, pp.756-762

Belge Tipi:

Makale

Yayın Tarihi:

9/2024

Belge Dili:

İngilizce

Document ID (Org No.):

2001315059

pISSN:

1308-0865

Anahtar kelimeler:

Özet:

 Purpose: The advent of large language models like GPT-4 has opened new possibilities in natural language processing, with potential applications in medical literature. This study assesses GPT-4s ability to generate medical abstracts. It compares their quality to original abstracts written by human authors, aiming to understand the effectiveness of artificial intelligence in replicating complex, professional writing tasks. Materials and methods: A total of 250 original research articles from five prominent radiology journals published between 2021 and 2023 were selected. The body of these articles, excluding the abstracts, was fed into GPT-4, which then generated new abstracts. Three experienced radiologists blindly and independently evaluated all 500 abstracts using a five-point Likert scale for quality and understandability. Statistical analysis included mean score comparison inter-rater reliability using Fleiss Kappa and Bland-Altman plots to assess agreement levels between raters. Results: Analysis revealed no significant difference in the mean scores between original and GPT-4 generated abstracts. The inter-rater reliability yielded kappa values indicating moderate to substantial agreement: 0.497 between Observers 1 and 2, 0.753 between Observers 1 and 3, and 0.645 between Observers 2 and 3. Bland-Altman analysis showed a slight systematic bias but was within acceptable limits of agreement. Conclusion: The study demonstrates that GPT-4 can generate medical abstracts with a quality comparable to those written by human experts. This suggests a promising role for artificial intelligence in facilitating the abstract writing process and improving its quality.

 Purpose: The advent of large language models like GPT-4 has opened new possibilities in natural language processing, with potential applications in medical literature. This study assesses GPT-4s ability to generate medical abstracts. It compares their quality to original abstracts written by human authors, aiming to understand the effectiveness of artificial intelligence in replicating complex, professional writing tasks. Materials and methods: A total of 250 original research articles from five prominent radiology journals published between 2021 and 2023 were selected. The body of these articles, excluding the abstracts, was fed into GPT-4, which then generated new abstracts. Three experienced radiologists blindly and independently evaluated all 500 abstracts using a five-point Likert scale for quality and understandability. Statistical analysis included mean score comparison inter-rater reliability using Fleiss Kappa and Bland-Altman plots to assess agreement levels between raters. Results: Analysis revealed no significant difference in the mean scores between original and GPT-4 generated abstracts. The inter-rater reliability yielded kappa values indicating moderate to substantial agreement: 0.497 between Observers 1 and 2, 0.753 between Observers 1 and 3, and 0.645 between Observers 2 and 3. Bland-Altman analysis showed a slight systematic bias but was within acceptable limits of agreement. Conclusion: The study demonstrates that GPT-4 can generate medical abstracts with a quality comparable to those written by human experts. This suggests a promising role for artificial intelligence in facilitating the abstract writing process and improving its quality.

 Purpose: The advent of large language models like GPT-4 has opened new possibilities in natural language processing, with potential applications in medical literature. This study assesses GPT-4s ability to generate medical abstracts. It compares their quality to original abstracts written by human authors, aiming to understand the effectiveness of artificial intelligence in replicating complex, professional writing tasks. Materials and methods: A total of 250 original research articles from five prominent radiology journals published between 2021 and 2023 were selected. The body of these articles, excluding the abstracts, was fed into GPT-4, which then generated new abstracts. Three experienced radiologists blindly and independently evaluated all 500 abstracts using a five-point Likert scale for quality and understandability. Statistical analysis included mean score comparison inter-rater reliability using Fleiss Kappa and Bland-Altman plots to assess agreement levels between raters. Results: Analysis revealed no significant difference in the mean scores between original and GPT-4 generated abstracts. The inter-rater reliability yielded kappa values indicating moderate to substantial agreement: 0.497 between Observers 1 and 2, 0.753 between Observers 1 and 3, and 0.645 between Observers 2 and 3. Bland-Altman analysis showed a slight systematic bias but was within acceptable limits of agreement. Conclusion: The study demonstrates that GPT-4 can generate medical abstracts with a quality comparable to those written by human experts. This suggests a promising role for artificial intelligence in facilitating the abstract writing process and improving its quality.

 Amaç: GPT-4 gibi büyük dil modellerinin ortaya çıkışı, tıbbi literatürdeki potansiyel uygulamalarla birlikte doğal dil işlemede yeni olanaklar sağlamıştır. Bu çalışma GPT-4ün tıbbi makale özetleri oluşturma yeteneğini değerlendirmektedir. Çalışma yapay zekanın karmaşık, profesyonel yazma görevlerini kopyalamadaki etkinliğini anlamayı amaçlamakta ve kalitelerini insan yazarlar tarafından yazılan orijinal özetlerle karşılaştırmaktadır. Gereç ve yöntem: 2021-2023 yılları arasında yayınlanan beş önde gelen radyoloji dergisinden toplam 250 orijinal araştırma makalesi seçildi. Bu makalelerin tamamı, özetler hariç, GPT-4e yüklendi ve daha sonra GPT-4 tarafından yeni özetler oluşturuldu. Üç deneyimli radyolog, kalite ve anlaşılabilirlik açısından beşli Likert ölçeği kullanarak 500 özetin tamamını kör ve bağımsız bir şekilde değerlendirdi. İstatistiksel analizde, değerlendiriciler arasındaki güvenilirliği ölçmek için Fleiss Kappa testi ve değerlendiriciler arasındaki uyum düzeylerini değerlendirmek için Bland-Altman grafikleri kullanıldı. Bulgular: Analiz, orijinal ve GPT-4 ile oluşturulan özetler arasında ortalama puanlar açısından anlamlı bir fark olmadığını ortaya koymuştur. Değerlendiriciler arası güvenilirlik açısından, orta ile önemli düzeyde uyuma işaret eden kappa değerleri bulunmuştur; değerler Gözlemci 1 ve 2 arasında 0.497, Gözlemci 1 ve 3 arasında 0.753 ve Gözlemci 2 ve 3 arasında 0.645 idi. Bland-Altman analizi hafif bir sistematik sapma göstermiş ancak kabul edilebilir uyum sınırları içinde kalmıştır. Sonuç: Çalışma, GPT-4ün insan uzmanlar tarafından yazılanlarla karşılaştırılabilir kalitede tıbbi özetler oluşturabildiğini göstermektedir. Yapay zeka kullanımı özet yazma sürecini kolaylaştırma ve kalitesini artırma konusunda önemli katkılar sağlayabilir.

Makale detayları

Artificial intelligence meets medical expertise: evaluating GPT-4's proficiency in generating medical article abstracts

Yazarlar:

Ergin Sağtaş | Furkan Ufuk | Hakkı Peker | Ahmet Baki Yağcı

Yayıncı:

Pamukkale University

Ana Başlıklar:

Radyoloji ve Organ Görüntüleme

Dergi:

Pamukkale Tıp Dergisi Vol.17 Issue 4, pp.756-762

Belge Tipi:

Makale

Yayın Tarihi:

9/2024

Belge Dili:

İngilizce

Document ID (Org No.):

2001315059

pISSN:

1308-0865

Artificial intelligence

ChatGPT

radiology

diagnosis

abstractsYapay zeka

ChatGPT

radyoloji

tanı

özet

Sayı

Makale detayları

Yazarlar: Ergin Sağtaş | Furkan Ufuk | Hakkı Peker | Ahmet Baki Yağcı

Yayıncı: Pamukkale University

Ana Başlıklar: Radyoloji ve Organ Görüntüleme

Dergi: Pamukkale Tıp Dergisi Vol.17 Issue 4, pp.756-762

Belge Tipi: Makale

Yayın Tarihi: 9/2024

Belge Dili: İngilizce

Document ID (Org No.): 2001315059

pISSN: 1308-0865

Sayı

Yazarlar:

Ergin Sağtaş | Furkan Ufuk | Hakkı Peker | Ahmet Baki Yağcı

Yayıncı:

Pamukkale University

Ana Başlıklar:

Radyoloji ve Organ Görüntüleme

Dergi:

Pamukkale Tıp Dergisi Vol.17 Issue 4, pp.756-762

Belge Tipi:

Makale

Yayın Tarihi:

9/2024

Belge Dili:

İngilizce

Document ID (Org No.):

2001315059

pISSN:

1308-0865