A Conceptual Framework on Development of Sign Language Chatbot for E-Commerce

This study proposes and validates a conceptual framework using a sign language (SL) chatbot in the e-commerce domain to improve accessibility. Currently, the use of SL is a less-explored area in online platforms. It is difficult for any communication-challenged person to interact online for buying products and services, using a chatbot, particularly for people using SL. The objective of this study is to introduce a novel hybrid architecture using SL Recognition with a conversational-based chatbot agent via a custom Application Programming Interface (API) in e-commerce platforms. Our work proposes a combined hybrid chatbot framework model using Convolutional Neural Network (CNN) and Natural Language Processing (NLP). Python libraries such as Keras, OpenCV, and MediaPipe frameworks were used to read the signs in the system. To test this study at this initial stage, a preliminary feasibility experiment was conducted. Purposive sampling has been used to select 8 participants familiar with American Sign Language (ASL), who were tested under various conditions, including different lighting and clothing. The SL recognition module’s initial performance data has been analyzed using precision, recall, and F1 scores to assess ASL recognition and achieved an accuracy of 98%. This whole work showcases a blueprint to develop inclusive e-commerce platforms to encourage accessibility.