Citation:

Dr. Ranjith Gopalan, Mr. Abhishek Sen, Mr. Vishal S, 2025. "Integrating Large Language Models: Enhancing Chatbot Capabilities for Training on Diverse Data Sources", ESP Journal of Engineering & Technology Advancements  5(1): 12-29.

Abstract:

The paper discusses the fundamentals of chatbot design, including key components, types of chatbots, and the importance of user experience and interaction design. It then delves into the structure and function of Large Language Models, their training, and their use cases in chatbot development. The paper explores the process of integrating LLMs with chatbot frameworks, highlighting the key steps and the services available for building chatbots, such as Amazon Bedrock, Microsoft LUIS, and Google Bard.Paper explains the fundamentals of chatbot design, provides an overview of large language models, discusses chatbot architecture for handling unstructured and structured data, and highlights the roles of AWS Titan and Anthropic Claude models in the development of retrieval-augmented generation systems. It includes a comparison of execution results, examines testing and evaluation of chatbot performance, considers ethical aspects of chatbot development, and suggests future research directions such as optimizing RA-based and knowledge graph-enhanced models to handle larger datasets and more complex queries without compromising performance.Furthermore, the paper provides a detailed comparison of the foundational models integrated with the Amazon Bedrock service, focusing on the Amazon Titan model used for knowledge base creation.

References:

[1] Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Abad, Z. S. H., Thieme, A., Yang, Z., Wang, Y., Lin, B., Gevaert, O., Li, L.-J., Jain, R., & Rahmani, A. M. (2023). Foundation Metrics: Quantifying Effectiveness of Healthcare Conversations powered by Generative AI. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2309.

[2] Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023). Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education. https://doi.org/10.32388/md04b0

[3] Adolphs, L., Shuster, K., Urbanek, J., Szlam, A., & Weston, J. (2021). Reason first, then respond: Modular Generation for Knowledge-infused Dialogue. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2111.05204

[4] Arun, A., Batra, S., Bhardwaj, V., Challa, A., Dönmez, P., Heidari, P., Inan, H., Jain, S., Kumar, A., Mei, S., Mohan, K., & White, M. (2020). Best Practices for Data-Efficient Modeling in NLG: How to Train Production-Ready Neural Models with Less Data (p. 64). https://doi.org/10.18653/v1/2020.coling-industry.7

[5] Ashfaq, M., Yun, J., Yu, S., & Loureiro, S. M. C. (2023). I, Chatbot: Modeling the determinants of users’ satisfaction and continuance intention of AI-powered service agents. https://www.sciencedirect.com/science/article/pii/S0736585320301325

[6] Bastola, A., Wang, H., Hembree, J., Yadav, P., McNeese, N. J., & Razi, A. (2023). LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2306.

[7] Cantador, I., Viejo-Tardío, J., Cortés-Cediel, M. E., & Bolívar, M. P. R. (2021). A Chatbot for Searching and Exploring Open Data: Implementation and Evaluation in E-Government. https://doi.org/10.1145/3463677.3463681

[8] Castillo-Bolado, D., Davidson, J. K., Gray, F., & Rosa, M. K. A. (2024). Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2409.20222

[9] Chair, Y., Liu, A., Rogers, A., Magdy, W., Preoţiuc-Pietro, D., Akhtar, S., Αλέτρας, Ν., Bontcheva, K., Darwish, K.-R., ElSherief, M., Garimella, K., Guerini, M., Jaidka, K., McGillivray, B., Mejova, Y., Naseem, U., Roß, B., Thorne, J., Viviani, M., … Huang, C.-Y. (2023). Findings of the Association for Computational Linguistics: ACL 2023. In Findings of the Association for Computational Linguistics: ACL 2022. https://doi.org/10.18653/v1/2023.findings-acl

[10] Chaves, A. P., & Gerosa, M. A. (2020). How Should My Chatbot Interact? A Survey on Social Characteristics in Human–Chatbot Interaction Design. In International Journal of Human-Computer Interaction (Vol. 37, Issue 8, p. 729). Taylor & Francis. https://doi.org/10.1080/10447318.2020.1841438

[11] https://dblp.uni-trier.de/db/conf/lrec/lrec2016.html#ChiarainC16

[12] Dai, S., Wang, G., Park, S., & Lee, S. (2021). Dialogue Response Generation via Contrastive Latent Representation Learning (p. 189). https://doi.org/10.18653/v1/2021.nlp4convai-1.18

[13] Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.-E., Lomelí, M., Hosseini, L., & Jeǵou, H. (2024). The Faiss library. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2401.08281

[14] Dowlagar, S., & Mamidi, R. (2021). CMSAOne@Dravidian-CodeMix-FIRE2020: A Meta Embedding and Transformer model for Code-Mixed Sentiment Analysis on Social Media Text. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2101.

[15] Fadhil, A. (2018). Domain Specific Design Patterns: Designing For Conversational User Interfaces. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.1802.09055

[16] Finch, S. E., Paek, E. S., & Choi, J. D. (2023). Leveraging Large Language Models for Automated Dialogue Analysis. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2309.06490

[17] Følstad, A., & Brandtzæg, P. B. (2020). Users’ experiences with chatbots: findings from a questionnaire study. In Quality and User Experience (Vol. 5, Issue 1). Springer Science+Business Media. https://doi.org/10.1007/s41233-020-00033-2

[18] Friedman, L., Ahuja, S., Allen, D. T., Tan, Z., Sidahmed, H., Long, C., Xie, J., Schubiner, G., Patel, A., Lara, H., Chu, B., Chen, Z., & Tiwari, M. K. (2023). Leveraging Large Language Models in Conversational Recommender Systems. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2305.07961

[19] Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2020). Dynamic Contextualized Word Embeddings. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2010.

[20] Huang, F., & Strigini, L. (2021). HEDP: A Method for Early Forecasting Software Defects based on Human Error Mechanisms. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2110.06758

[21] Isa, N. A. N. M., Jawaddi, S. N. A., & Ismail, A. (2024). Experimental Evaluation of Machine Learning Models for Goal-oriented Customer Service Chatbot with Pipeline Architecture. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2409.18568

[22] Jacky Casas, jacky.casas@hes-so.ch, Marc-Olivier Tricot, Omar Abou Khaled, omar.aboukhaled@hes-so.ch, Elena Mugellini, elena.mugellini@hes-so.ch, Philippe Cudré-Mauroux, philippe.cudre-mauroux@unifr.ch. (2023). Trends & Methods in Chatbot Evaluation. https://dl.acm.org/doi/10.1145/3395035.3425319

[23] Khan, A., Hasan, M. T., Kemell, K. K., Rasku, J., & Abrahamsson, P. (2024). Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2410.15944

[24] Konreddy, S. D. R. (2021). The Impact of NLP on Software Testing. In Journal of University of Shanghai for Science and Technology (Vol. 23, Issue 8, p. 295). https://doi.org/10.51201/jusst/21/08380

[25] Li, C., Zhang, M., Mei, Q., Wang, Y., Hombaiah, S. A., Liang, Y., & Bendersky, M. (2023). Teach LLMs to Personalize -- An Approach inspired by Writing Education. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2308.

[26] Liu, B., & Mazumder, S. (2020). Lifelong Learning Dialogue Systems: Chatbots that Self-Learn On the Job. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2009.10750

[27] Mao, K., Dou, Z., Chen, H., Mo, F., & Qian, H. (2023). Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2303.

[28] Pahune, S., & Manoj, C. (2023). Several Categories of Large Language Models (LLMs): A Short Survey. In International Journal for Research in Applied Science and Engineering Technology (Vol. 11, Issue 7, p. 615). International Journal for Research in Applied Science and Engineering Technology (IJRASET). https://doi.org/10.22214/ijraset.2023.54677

[29] Pantano, E., & Pizzi, G. (2020). Forecasting artificial intelligence on online customer assistance: Evidence from chatbot patents analysis. In Journal of Retailing and Consumer Services (Vol. 55, p. 102096). Elsevier BV. https://doi.org/10.1016/j.jretconser.2020.102096

[30] Parnin, C., Soares, G., Pandita, R., Gulwani, S., Rich, J. A. J., & Henley, A. Z. (2023). Building Your Own Product Copilot: Challenges, Opportunities, and Needs. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2312.14231

[31] Pozdniakov, S., Brazil, J., Abdi, S., Bakharia, A., Sadiq, S., Gašević, D., Denny, P., & Khosravi, H. (2024). Large Language Models Meet User Interfaces: The Case of Provisioning Feedback. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2404.

[32] Pyatkin, V., Roit, P., Michael, J., Goldberg, Y., Tsarfaty, R., & Dagan, I. (2021). Asking It All: Generating Contextualized Questions for any Semantic Role. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2109.04832

[33] Engineering (Vol. 29, Issue 11, p. 1673). World Scientific. https://doi.org/10.1142/s0218194019400163

[34] Saad-Falcon, J., Barrow, J., Siu, A., Nenkova, A., Rossi, R. A., & Dernoncourt, F. (2023). PDFTriage: Question Answering over Long, Structured Documents. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2309.08872

[35] Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models [Review of Role play with large language models]. Nature, 623(7987), 493. Nature Portfolio. https://doi.org/10.1038/s41586-023-06647-8

[36] Silva, G. R. S., & Canedo, E. D. (2022). Towards User-Centric Guidelines for Chatbot Conversational Design. In International Journal of Human-Computer Interaction (Vol. 40, Issue 2, p. 98). Taylor & Francis. https://doi.org/10.1080/10447318.2022.2118244

[37] Song, C., & Raghunathan, A. (2020). Information Leakage in Embedding Models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (p. 377). https://doi.org/10.1145/3372297.3417270

[38] Tang, X., Shin, R., Inan, H. A., Manoel, A., Mireshghallah, F., Lin, Z., Gopi, S., Kulkarni, J., & Sim, R. B. (2023). Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2309.11765

[39] Tanioka, H. (2019). A Fast Content-Based Image Retrieval Method Using Deep Visual Features. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.1908.01505

[40] Tennenholtz, G., Chow, Y., Hsu, C., Jeong, J., Shani, L., Tulepbergenov, A., Ramachandran, D., Mladenov, M., & Boutilier, C. (2023). Demystifying Embedding Spaces using Large Language Models. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2310.

[41] Vijayaraghavan, V., Cooper, J. B., & J., R. L. (2020). Algorithm Inspection for Chatbot Performance Evaluation. In Procedia Computer Science (Vol. 171, p. 2267). Elsevier BV. https://doi.org/10.1016/j.procs.2020.04.245

[42] Villalba, A. C., Brown, E. M., Scurrell, J. V., Entenmann, J., & Daepp, M. I. G. (2023). Automated Interviewer or Augmented Survey? Collecting Social Data with Large Language Models. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2309.10187

[43] Wang, K., Ramos, J., & Lawrence, R. (2024). ChatEd: A Chatbot Leveraging ChatGPT for an Enhanced Learning Experience in Higher Education. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arXiv.2401.

[44] Wang, K., Ramos, J., & Lawrence, R. (2024). ChatEd: A Chatbot Leveraging ChatGPT for an Enhanced Learning Experience in Higher Education. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2401.00052

[45] Wynn, D. C., & Maier, A. (2022). Feedback systems in the design and development process. In Research in Engineering Design (Vol. 33, Issue 3, p. 273). Springer Science+Business Media. https://doi.org/10.1007/s00163-022-00386-z

[46] Xia, P., Zhu, K., Li, H., Zhu, H., Li, Y., Li, G., Zhang, S., & Yao, H. (2024). RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2407.05131

[47] Xiong, H., Bian, J., Yang, S., Zhang, X., Kong, L., & Zhang, D. (2023). Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorial. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2309.15074

[48] Zamfirescu-Pereira, J. D., Hartmann, B., & Yang, Q. (2023). Conversation Regression Testing: A Design Technique for Prototyping Generalizable Prompt Strategies for Pre-trained Language Models. In arXiv (Cornell University). Cornell University. https://doi.org/10.48550/arxiv.2302.03154

Keywords:

LLM, Chatbots Generative AI, Meta data, Knowledge graph, Structured and unstructured data, AWS Titan, FAISS index, Anthropic Claude