Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics

Ertuğrul Safran; Adem Çalı

Makale detayları

Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics

Yazarlar:

Ertuğrul Safran | Adem Çalı

Yayıncı:

MediHealth Academy Yayıncılık

Ana Başlıklar:

Fizyoterapi

Dergi:

Anatolian Current Medical Journal Vol.7 Issue 5, pp.695-702

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001358261

pISSN:

2718-0115

Anahtar kelimeler:

Özet:

 Aims: Large language models LLMs such as ChatGPT are increasingly used in academic and clinical writing. While these tools can generate coherent and domain-specific text, concerns persist regarding the accuracy of their automatically generated references. In musculoskeletal rehabilitationa field heavily reliant on current evidencethe reliability of citations is especially critical. Yet, systematic evaluations of citation accuracy in AI-generated scientific content are lacking. To evaluate the reference accuracy of scientific texts generated by ChatGPT GPT-4 in response to musculoskeletal rehabilitation prompts, and to determine whether reference accuracy improves following structured post-generation verification. Methods: ChatGPT was prompted to generate four scientific paragraphs on musculoskeletal rehabilitation topics manual therapy, ACL reconstruction, low back pain, and rotator cuff repair, each including 10 references with DOIs. A total of 40 references were analyzed using a 3-point scoring system 0fabricated, 1partially correct, 2fully accurate, which was used to assess citation quality. After initial evaluation, ChatGPT was asked to verify and revise its references. Scores before and after this step were compared descriptively and with Wilcoxon signed-rank tests to assess statistical significance, and effect sizes r were calculated to estimate the magnitude of improvement. Results: Only 7.5 of references were fully accurate in the initial generation, while 42.5 were completely fabricated. The remaining 50 were partially correct. After verification, the proportion of fully accurate references rose to 77.5. Wilcoxon signed-rank testing confirmed a statistically significant improvement in accuracy across all prompts W561.0, p0.001, r0.60. The most common errors included invalid DOIs, fabricated article titles, and mismatched metadata. Conclusion: ChatGPT can generate coherent scientific content, but its initial references are frequently inaccurate or fabricated. Structured post-generation verification significantly improves reference accuracy, as confirmed by statistical testing. These findings suggest that LLMs may be integrated as drafting tools in academic and clinical musculoskeletal contexts, but only when accompanied by strict human-led verification of citations.

 Aims: Large language models LLMs such as ChatGPT are increasingly used in academic and clinical writing. While these tools can generate coherent and domain-specific text, concerns persist regarding the accuracy of their automatically generated references. In musculoskeletal rehabilitationa field heavily reliant on current evidencethe reliability of citations is especially critical. Yet, systematic evaluations of citation accuracy in AI-generated scientific content are lacking. To evaluate the reference accuracy of scientific texts generated by ChatGPT GPT-4 in response to musculoskeletal rehabilitation prompts, and to determine whether reference accuracy improves following structured post-generation verification. Methods: ChatGPT was prompted to generate four scientific paragraphs on musculoskeletal rehabilitation topics manual therapy, ACL reconstruction, low back pain, and rotator cuff repair, each including 10 references with DOIs. A total of 40 references were analyzed using a 3-point scoring system 0fabricated, 1partially correct, 2fully accurate, which was used to assess citation quality. After initial evaluation, ChatGPT was asked to verify and revise its references. Scores before and after this step were compared descriptively and with Wilcoxon signed-rank tests to assess statistical significance, and effect sizes r were calculated to estimate the magnitude of improvement. Results: Only 7.5 of references were fully accurate in the initial generation, while 42.5 were completely fabricated. The remaining 50 were partially correct. After verification, the proportion of fully accurate references rose to 77.5. Wilcoxon signed-rank testing confirmed a statistically significant improvement in accuracy across all prompts W561.0, p0.001, r0.60. The most common errors included invalid DOIs, fabricated article titles, and mismatched metadata. Conclusion: ChatGPT can generate coherent scientific content, but its initial references are frequently inaccurate or fabricated. Structured post-generation verification significantly improves reference accuracy, as confirmed by statistical testing. These findings suggest that LLMs may be integrated as drafting tools in academic and clinical musculoskeletal contexts, but only when accompanied by strict human-led verification of citations.

 Aims: Large language models LLMs such as ChatGPT are increasingly used in academic and clinical writing. While these tools can generate coherent and domain-specific text, concerns persist regarding the accuracy of their automatically generated references. In musculoskeletal rehabilitationa field heavily reliant on current evidencethe reliability of citations is especially critical. Yet, systematic evaluations of citation accuracy in AI-generated scientific content are lacking. To evaluate the reference accuracy of scientific texts generated by ChatGPT GPT-4 in response to musculoskeletal rehabilitation prompts, and to determine whether reference accuracy improves following structured post-generation verification. Methods: ChatGPT was prompted to generate four scientific paragraphs on musculoskeletal rehabilitation topics manual therapy, ACL reconstruction, low back pain, and rotator cuff repair, each including 10 references with DOIs. A total of 40 references were analyzed using a 3-point scoring system 0fabricated, 1partially correct, 2fully accurate, which was used to assess citation quality. After initial evaluation, ChatGPT was asked to verify and revise its references. Scores before and after this step were compared descriptively and with Wilcoxon signed-rank tests to assess statistical significance, and effect sizes r were calculated to estimate the magnitude of improvement. Results: Only 7.5 of references were fully accurate in the initial generation, while 42.5 were completely fabricated. The remaining 50 were partially correct. After verification, the proportion of fully accurate references rose to 77.5. Wilcoxon signed-rank testing confirmed a statistically significant improvement in accuracy across all prompts W561.0, p0.001, r0.60. The most common errors included invalid DOIs, fabricated article titles, and mismatched metadata. Conclusion: ChatGPT can generate coherent scientific content, but its initial references are frequently inaccurate or fabricated. Structured post-generation verification significantly improves reference accuracy, as confirmed by statistical testing. These findings suggest that LLMs may be integrated as drafting tools in academic and clinical musculoskeletal contexts, but only when accompanied by strict human-led verification of citations.

 Amaç: ChatGPT gibi büyük dil modelleri LLMler, akademik ve klinik yazımda giderek daha fazla kullanılmaktadır. Bu araçlar tutarlı ve alanına özgü metinler üretebilse de, otomatik olarak oluşturulan referansların doğruluğuna ilişkin endişeler devam etmektedir. Güncel kanıtlara büyük ölçüde dayanan kas-iskelet sistemi rehabilitasyonu alanında, atıfların güvenilirliği özellikle kritik öneme sahiptir. Ancak yapay zekâ tarafından üretilen bilimsel içeriklerde atıf doğruluğunu sistematik olarak değerlendiren çalışmalar yetersizdir. Bu çalışmanın amacı, kas-iskelet sistemi rehabilitasyonu konularında oluşturulan promptlara yanıt olarak ChatGPT GPT-4 tarafından üretilen bilimsel metinlerdeki referansların doğruluğunu değerlendirmek ve yapılandırılmış bir doğrulama sürecinin referans doğruluğunu artırıp artırmadığını belirlemektir. Yöntem: ChatGPTden manuel terapi, ön çapraz bağ ACL rekonstrüksiyonu, bel ağrısı ve rotator manşet onarımı konularında, her biri 10 DOI içeren dört bilimsel paragraf üretmesi istendi. Toplam 40 referans, 3 puanlık bir sistemle değerlendirildi: 0  uydurma, 1  kısmen doğru, 2  tamamen doğru. İlk değerlendirmeden sonra ChatGPTden referansları doğrulaması ve düzeltmesi istendi. İlk ve son durumdaki puanlar betimsel olarak karşılaştırıldı. Bulgular: İlk üretimde referansların yalnızca 7,5i tamamen doğruydu, 42,5i ise tamamen uydurmaydı. Geri kalan 50si kısmen doğruydu. Doğrulama sürecinden sonra tamamen doğru referans oranı 77,5e yükseldi. En yaygın hatalar geçersiz DOIler, uydurma makale başlıkları ve yanlış eşleşmiş metadata içeriğiydi. Sonuç: ChatGPT tutarlı bilimsel metinler üretebilse de, referansları sıklıkla hatalı veya uydurmadır. Üretim sonrası doğrulama, referans doğruluğunu önemli ölçüde artırmaktadır. Yapay zekâ araçlarının akademik ve klinik kas-iskelet sistemi bağlamlarında kullanımı sırasında, özellikle atıf geçerliliği açısından dikkatli olunmalıdır.

Makale detayları

Fabricated or accurate? Ethical concerns and citation hallucination in aI-generated scientific writing on musculoskeletal topics

Yazarlar:

Ertuğrul Safran | Adem Çalı

Yayıncı:

MediHealth Academy Yayıncılık

Ana Başlıklar:

Fizyoterapi

Dergi:

Anatolian Current Medical Journal Vol.7 Issue 5, pp.695-702

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001358261

pISSN:

2718-0115

ChatGPT

artificial intelligence

musculoskeletal rehabilitation

scientific writing

reference accuracy

hallucinated citationsChatGPT

Yapay zeka

Muskuloskeletal rehabilitasyon

bilimsel yazı

referans doğruluğu

atıf halüsinasyonu

Sayı

Makale detayları

Yazarlar: Ertuğrul Safran | Adem Çalı

Yayıncı: MediHealth Academy Yayıncılık

Ana Başlıklar: Fizyoterapi

Dergi: Anatolian Current Medical Journal Vol.7 Issue 5, pp.695-702

Belge Tipi: Makale

Yayın Tarihi: 9/2025

Belge Dili: İngilizce

Document ID (Org No.): 2001358261

pISSN: 2718-0115

Sayı

Yazarlar:

Ertuğrul Safran | Adem Çalı

Yayıncı:

MediHealth Academy Yayıncılık

Ana Başlıklar:

Fizyoterapi

Dergi:

Anatolian Current Medical Journal Vol.7 Issue 5, pp.695-702

Belge Tipi:

Makale

Yayın Tarihi:

9/2025

Belge Dili:

İngilizce

Document ID (Org No.):

2001358261

pISSN:

2718-0115