Development of a Bilingual Diagnostic Chatbot (English and Yoruba) for Infectious Disease Symptom Checks using Large Language Models with Retrieval Augmented Generation
No Thumbnail Available
Date
2024-11-25
Authors
Dairo O.
Sanni R.T.
Okunola O.V.
Odumuyiwa V.T.
Journal Title
Journal ISSN
Volume Title
Publisher
MIRG
Abstract
This paper presents a multilingual healthcare chatbot for infectious disease diagnosis for English and Yoruba language speakers in Southwest, Nigeria. This chatbot was developed using the Mistral 7B LLaMA model, following a multi-step process. The process included collecting and preparing a dataset comprising over 500 symptoms and 100 diseases related to viral, bacterial, and fungal infections. Additionally, a substantial dataset was sourced from Hugging Face's Yoruba Text Corpus to enhance the model's linguistic capabilities. The Mistral model was then trained and tokenized to accurately recognize and interpret symptoms. Also, a Retrieval-Augmented Generation (RAG) was implemented to retrieve from the indexed dataset in real-time. The result revealed the effectiveness of the chatbot in infectious diseases diagnosis in response to the user's symptoms input. The performance evaluation using the MMLU (Medicine & Biology) benchmark revealed that Mistral 7B achieved 58.34% accuracy when tested for both English and Yoruba; Mistral 7B for English only performed at 60.10%; and LLaMA 2 7B achieved 44.34%, while LLaMA 2 13B recorded 55.30%. Mistral 7B exhibited a slight drop in accuracy when incorporating Yoruba (from 60.10% to 58.34%), likely due to the model's insufficient understanding of the Yoruba language. Despite this reduction, the fine-tuned Mistral 7B outperformed both LLaMA 2 7B and LLaMA 2 13B in overall performance. Notably, further fine-tuning of the Mistral 7B model led to a significant improvement in accuracy for Yoruba, with performance increasing to 78.34%. This result far surpassed the pretrained version as well as other LLMs such as LLaMA 2, GPT-3 Turbo, and Google Translate. The pilot users of the chatbot found the onboarding and other pages of the chatbot intuitive, simple, easy to use and its uniqueness in transitioning between English and Yoruba
Description
Keywords
Citation
Dairo O., Sanni R.T., Okunola O.V. & Odumuyiwa V.T. (2024). “Development of a Bilingual Diagnostic Chatbot (English and Yoruba) for Infectious Disease Symptom Checks using Large Language Models with Retrieval Augmented Generation". In Proceedings of the International Conference on Artificial Intelligence and Robotics (MIRG-ICAIR 2024), pp. 215-223, MIRG