In Brief | Artificial Intelligence | Head and Neck Cancers
Evaluating ChatGPT's Reliability in Providing Oral Cancer Information
Time to read: 04:46
Time to listen: 06:23
Originally Published: 17 January 2025
Source: BMC Oral Health
Type of article: In Brief
MedED Catalogue Reference: MNIB012
Category: Artificial Intelligence
Cross Reference: Head and Neck Surgery, Oncology
Keywords: AI, ChatGPT, Oral Cancer
This article is a review of recent studies originally published in the BMC Oral Health, 17 January 2025. This article does not represent the original research, nor is it intended to replace the original research. Access the full Disclaimer Information.
Watch the 60 Second Summary
(Please let us know if you like this format by clicking here)
Listen to Full Article
Patients increasingly turn to the internet for health-related information, with Google being the most widely used search engine. The emergence of advanced AI tools like ChatGPT marks a significant shift in this trend.
These technologies have begun transforming healthcare by providing personalised and conversational experiences in diagnosis, screening, and treatment planning. ChatGPT's ability to tailor responses to individual symptoms, in particular, may enhance patient engagement over time.
However, the use of ChatGPT in healthcare raises ethical and legal concerns, including patient privacy protection, potential copyright issues, and medical-legal implications. Patients unaware of AI's limitations—such as the potential for misinformation and the inability to recognize individual differences in diagnosis and treatment—might place undue trust in AI over human physicians.
This dynamic poses challenges for physicians, including potential erosion of the patient-physician relationship and issues with patient compliance. Therefore, assessing the validity and reliability of ChatGPT's responses is crucial.
Oral cancer, the most common cancer in the head and neck region, accounts for approximately 3.2% of cancer-related deaths. Early diagnosis is vital, as it significantly influences remission rates and quality of life. Delays in diagnosis often result from factors such as low socioeconomic status, advanced age, homelessness, cognitive impairments, and misconceptions that symptoms are benign. Late-stage diagnoses necessitate more aggressive treatments and adversely affect patients' quality of life.
Given the importance of early detection, AI tools like ChatGPT may offer promise in providing reliable information to patients. Researchers sought to investigate whether the answers provided by ChatGPT to common patient questions regarding oral cancer were accurate and consistent and had been obtained from reliable sources.
Study Purpose
This study aims to evaluate the accuracy of ChatGPT-3.5's responses to frequently asked questions about oral cancer, assessing its potential as a supplementary resource for patients seeking information about this malignancy.
Study Methodology
These questions were presented to ChatGPT, and its responses were assessed by four specialists: two oral and maxillofacial radiologists and two medical oncologists.
The evaluation was conducted using a 5-point Likert scale, with each specialist scoring the responses independently.
The study involved 80 evaluations in total, with each specialist reviewing 20 responses.
Findings
Out of the 80 evaluations of 20 ChatGPT responses, 41 (51.25%) were rated as very good, 37 (46.25%) as good, and 2 (2.50%) as acceptable.
The median accuracy score for oral and maxillofacial radiologists was 4.48 (range 4.45-4.5), while medical oncologists rated the accuracy of information as 4.5 (range 4.5-4.5).
The question "Can oral cancer be treated?" received the lowest score.
The questions "What is the risk of oral cancer again?" and "How long can I live after mouth cancer?" showed the highest variation in evaluators' responses, with scores ranging from 3.0 to 5.0.
Importantly, of the 81 references to ChatGPT answers, only 13 were scientific articles, 10 were fake, and the remaining references were data from websites.
These findings align with existing literature on ChatGPT's capabilities and limitations in medical contexts. Studies have demonstrated ChatGPT's potential in clinical decision-making, reporting an overall accuracy rate of approximately 72%, with higher accuracy in final diagnoses.
However, challenges persist in ensuring the accuracy and reliability of AI-generated medical information, emphasizing the need for ongoing evaluation and refinement.
Conclusion
Incorporating AI tools like ChatGPT into healthcare offers promise for enhancing patient education and engagement. While in this study ChatGPT provided reliable information about oral cancer and did not provide incorrect information and suggestions. not all the information provided by ChatGPT was based on clinical references.
Careful consideration of these limitations and continuous oversight are essential to maintain the quality and trustworthiness of medical information provided to patients.
Disclaimer
This article is compiled from several resources researched and compiled by the contributor. It is in no way presented as an original work. Every effort has been made to attribute quotes and content correctly. Where possible all information has been independently verified. The Medical Education Network bears no responsibility for any inaccuracies which may occur from the use of third-party sources. If you have any queries regarding this article contact us
Fact-checking Policy
The Medical Education Network makes every effort to review and fact-check the articles used as source material in our summaries and original material. We have strict guidelines in relation to the publications we use as our source data, favouring peer-reviewed research wherever possible. Every effort is made to ensure that the information contained here is an accurate reflection of the original material. Should you find inaccuracies, out of date content or have any additional issues with our articles, please make use of the contact us form to notify us.
