COMPARISON OF PERFORMANCES OF OPEN ACCESS NATURAL LANGUAGE PROCESSING BASED CHATBOT APPLICATIONS IN TRIAGE DECISIONS

İbrahim Sarbay; Göksu Bozdereli Berikol; İbrahim Ulaş Özturan; Keith Grimes

Makale detayları

COMPARISON OF PERFORMANCES OF OPEN ACCESS NATURAL LANGUAGE PROCESSING BASED CHATBOT APPLICATIONS IN TRIAGE DECISIONS

Yazarlar:

İbrahim Sarbay | Göksu Bozdereli Berikol | İbrahim Ulaş Özturan | Keith Grimes

Yayıncı:

Kirikkale University

Ana Başlıklar:

Sağlık Hizmetleri ve Sistemleri (Diğer)

Dergi:

Kırıkkale Üniversitesi Tıp Fakültesi Dergisi Vol.25 Issue 3, pp.482-521

Belge Tipi:

Makale

Yayın Tarihi:

12/2023

Belge Dili:

İngilizce

Document ID (Org No.):

2001306802

pISSN:

2148-9645

Anahtar kelimeler:

Özet:

 Objective: Being publicly available, easy to use, and continuously evolving, next-generation chatbots have the potential to be used in triage, one of the most critical functions of an Emergency Department. The aim of this study was to assess the performance of Generative Pre-trained Transformer 4 GPT-4, Bard and Claude during decision-making for Emergency Department triage. Material and Methods: This was a preliminary cross-sectional study conducted with 50 case scenarios. Emergency Medicine specialists determined the reference Emergency Severity Index triage category of each scenario. Subsequently, each case scenario was queried using three chatbots. Inconsistent classifications between the chatbots and references were defined as over-triage false positive or under-triage false negative. The primary and secondary outcomes were the predictive performance of chatbots and the difference between them in predicting high acuity triage. Results: F1 Scores for GPT-4, Bard, and Claude for predicting Emergency Severity Index 1 and 2 were 0.899, 0.791, and 0.865 respectively. The ROC Curve of GPT-4 for high acuity predictions showed an area under the curve AUC of 0.911 95 CI: 0,814-1; p0.001, while Bard showed an AUC of 0.819 95 CI: 0.692-0.945; p0.001 and for Claude this was 0.881 95 CI:0.768-0.994; p0.001. Conclusion: GPT-4, in its current form, was able to detect high acuity Emergency Severity Index scores in our case set and had close agreement with Emergency Medicine specialists, followed by Claude, while Bards agreement was relatively lower. GPT-4 and Claude provided better results than Bard in case management recommendations. We believe that studies evaluating the effectiveness and limitations of chatbots in triage are important because of their future potential.

 Objective: Being publicly available, easy to use, and continuously evolving, next-generation chatbots have the potential to be used in triage, one of the most critical functions of an Emergency Department. The aim of this study was to assess the performance of Generative Pre-trained Transformer 4 GPT-4, Bard and Claude during decision-making for Emergency Department triage. Material and Methods: This was a preliminary cross-sectional study conducted with 50 case scenarios. Emergency Medicine specialists determined the reference Emergency Severity Index triage category of each scenario. Subsequently, each case scenario was queried using three chatbots. Inconsistent classifications between the chatbots and references were defined as over-triage false positive or under-triage false negative. The primary and secondary outcomes were the predictive performance of chatbots and the difference between them in predicting high acuity triage. Results: F1 Scores for GPT-4, Bard, and Claude for predicting Emergency Severity Index 1 and 2 were 0.899, 0.791, and 0.865 respectively. The ROC Curve of GPT-4 for high acuity predictions showed an area under the curve AUC of 0.911 95 CI: 0,814-1; p0.001, while Bard showed an AUC of 0.819 95 CI: 0.692-0.945; p0.001 and for Claude this was 0.881 95 CI:0.768-0.994; p0.001. Conclusion: GPT-4, in its current form, was able to detect high acuity Emergency Severity Index scores in our case set and had close agreement with Emergency Medicine specialists, followed by Claude, while Bards agreement was relatively lower. GPT-4 and Claude provided better results than Bard in case management recommendations. We believe that studies evaluating the effectiveness and limitations of chatbots in triage are important because of their future potential.

 Objective: Being publicly available, easy to use, and continuously evolving, next-generation chatbots have the potential to be used in triage, one of the most critical functions of an Emergency Department. The aim of this study was to assess the performance of Generative Pre-trained Transformer 4 GPT-4, Bard and Claude during decision-making for Emergency Department triage. Material and Methods: This was a preliminary cross-sectional study conducted with 50 case scenarios. Emergency Medicine specialists determined the reference Emergency Severity Index triage category of each scenario. Subsequently, each case scenario was queried using three chatbots. Inconsistent classifications between the chatbots and references were defined as over-triage false positive or under-triage false negative. The primary and secondary outcomes were the predictive performance of chatbots and the difference between them in predicting high acuity triage. Results: F1 Scores for GPT-4, Bard, and Claude for predicting Emergency Severity Index 1 and 2 were 0.899, 0.791, and 0.865 respectively. The ROC Curve of GPT-4 for high acuity predictions showed an area under the curve AUC of 0.911 95 CI: 0,814-1; p0.001, while Bard showed an AUC of 0.819 95 CI: 0.692-0.945; p0.001 and for Claude this was 0.881 95 CI:0.768-0.994; p0.001. Conclusion: GPT-4, in its current form, was able to detect high acuity Emergency Severity Index scores in our case set and had close agreement with Emergency Medicine specialists, followed by Claude, while Bards agreement was relatively lower. GPT-4 and Claude provided better results than Bard in case management recommendations. We believe that studies evaluating the effectiveness and limitations of chatbots in triage are important because of their future potential.

 Amaç: Herkese açık olan, kolay kullanılan ve sürekli gelişen yeni nesil sohbet botları, Acil Servisin en kritik işlevlerinden biri olan triyajda kullanılma potansiyeline sahiptir. Bu çalışmanın amacı, acil servis triyajına karar verme sırasında Generative Pre-trained Transformer 4 GPT-4, Bard ve Claude uygulamalarının performansını değerlendirmektir. Gereç ve Yöntemler: Bu çalışma, 50 vaka senaryosu ile yürütülen kesitsel bir ön çalışmaydı. Acil Tıp uzmanları her senaryonun referans Emergency Severity Index triyaj kategorisini belirledikten sonra, her vaka senaryosu üç sohbet botu kullanılarak sorgulandı. Sohbet botları ve referanslar arasındaki tutarsız sınıflandırmalar overtriyaj yanlış pozitif veya undertriyaj yanlış negatif olarak tanımlandı. Birincil sonlanım sohbet botlarının tahmin performansı ve ikincil sonlanım ise yüksek ciddiyetteki vakaların triyajını belirlemede aralarındaki farktı. Bulgular: GPT-4, Bard ve Claudenin Emergency Severity Index 1 ve 2yi belirlemede F1 skorları sırasıyla 0,899, 0,791 ve 0,865ti. Yüksek ciddiyet tespiti için ROC eğrilerinde; GPT-4ün eğri altında kalan alanı AUC 0,911 95 GA: 0,814-1;p0.001, Bardın 0,819 95 GA: 0,692-0,945; p0.001 ve Claudenin 0,881 idi 95 GA: 0,768-0,994; p0,001. Sonuç: GPT-4, mevcut haliyle, vaka setimizde yüksek ciddiyetteki Emergency Severity Index skorlarını tespit edebildi ve Acil Tıp uzmanları ile yakın uyum gösterdi. Bunu Claude takip ederken, Bard ile uyumu ise nispeten daha düşüktü. GPT-4 ve Claude, vaka yönetimi önerilerinde Barda göre daha iyi sonuçlar verdi. Gelecekteki potansiyelleri nedeniyle, sohbet botlarının triyajdaki etkinliğini ve sınırlılıklarını değerlendiren çalışmaların önemli olduğunu düşünüyoruz.

Makale detayları

COMPARISON OF PERFORMANCES OF OPEN ACCESS NATURAL LANGUAGE PROCESSING BASED CHATBOT APPLICATIONS IN TRIAGE DECISIONS

Yazarlar:

İbrahim Sarbay | Göksu Bozdereli Berikol | İbrahim Ulaş Özturan | Keith Grimes

Yayıncı:

Kirikkale University

Ana Başlıklar:

Sağlık Hizmetleri ve Sistemleri (Diğer)

Dergi:

Kırıkkale Üniversitesi Tıp Fakültesi Dergisi Vol.25 Issue 3, pp.482-521

Belge Tipi:

Makale

Yayın Tarihi:

12/2023

Belge Dili:

İngilizce

Document ID (Org No.):

2001306802

pISSN:

2148-9645

Artificial intelligence

diagnosis

triageYapay zekâ

tanı

triyaj

Sayı

Makale detayları

COMPARISON OF PERFORMANCES OF OPEN ACCESS NATURAL LANGUAGE PROCESSING BASED CHATBOT APPLICATIONS IN TRIAGE DECISIONS

Yazarlar: İbrahim Sarbay | Göksu Bozdereli Berikol | İbrahim Ulaş Özturan | Keith Grimes

Yayıncı: Kirikkale University

Ana Başlıklar: Sağlık Hizmetleri ve Sistemleri (Diğer)

Dergi: Kırıkkale Üniversitesi Tıp Fakültesi Dergisi Vol.25 Issue 3, pp.482-521

Belge Tipi: Makale

Yayın Tarihi: 12/2023

Belge Dili: İngilizce

Document ID (Org No.): 2001306802

pISSN: 2148-9645

Artificial intelligence

diagnosis

triageYapay zekâ

tanı

triyaj

Sayı

Yazarlar:

İbrahim Sarbay | Göksu Bozdereli Berikol | İbrahim Ulaş Özturan | Keith Grimes

Yayıncı:

Kirikkale University

Ana Başlıklar:

Sağlık Hizmetleri ve Sistemleri (Diğer)

Dergi:

Kırıkkale Üniversitesi Tıp Fakültesi Dergisi Vol.25 Issue 3, pp.482-521

Belge Tipi:

Makale

Yayın Tarihi:

12/2023

Belge Dili:

İngilizce

Document ID (Org No.):

2001306802

pISSN:

2148-9645