A Web-Based SIBI Sign Language Translator Application with Speech-to-Text Feature Using CNN and MediaPipe

Fathur Rahman; Ahmaddul Hadi; Dony Novaliendry; Agariadne Dwinggo Samala

doi:10.24036/jtip.v18i2.971

Authors

Fathur Rahman Universitas Negeri Padang
Ahmaddul Hadi Universitas Negeri Padang
Dony Novaliendry Universitas Negeri Padang
Agariadne Dwinggo Samala Universitas Negeri Padang

DOI:

https://doi.org/10.24036/jtip.v18i2.971

Keywords:

Sign language translator, SIBI, Hand gesture recognition, Convolutional neural network, Speech-to-text, MediaPipe, Web application

Abstract

This study developed a web-based application to facilitate two-way communication between individuals with hearing impairments and the general public. The application translated hand gestures based on the Indonesian Sign System into text using a Convolutional Neural Network model and real-time landmark detection. Additionally, it converted spoken language into text through speech recognition technology, which was then displayed alongside the corresponding sign language images.

The system used a camera to capture hand gestures, which were processed into landmark data and classified into letters A to Z. Voice input was processed directly in the browser without additional installations. The application was designed to be lightweight, interactive, and compatible with various devices.

Testing results showed that the gesture recognition feature achieved high accuracy, ranging from 98.71% to 100%. The speech-to-text feature also provided accurate transcription results, both for individual letters and complete sentences. Accuracy decreased at distances beyond 30 cm and in noisy environments.

The integration of gesture recognition and speech-to-text conversion in a single web platform offered an effective, accessible, and inclusive communication solution for users with special needs.

References

I. Y. Simamora, M. Zahra, W. A. Sinaga, H. E. Pandiangan, and S. F. Hasibuan, “Peran Komunikasi dalam Pembangunan Pendidikan,” J. Pendidik. Tambusai, vol. Volume 8, p. 8, 2024.

Lela Ayu Septyani, Hanik Noor Solikhah, and Arcivid Chorynia Ruby, “Analisis Penggunaan Bahasa SIBI Untuk Meningkatkan Komunikasi Siswa Tunarungu Dalam Kehidupan Sehari-hari,” J. LENTERA J. Stud. Pendidik., vol. 6, no. 2, pp. 135–140, 2024, doi: 10.51518/lentera.v6i2.210.

R. H. Alfikri, M. S. Utomo, H. Februariyanti, and E. Nurwahyudi, “Pembangunan Aplikasi Penerjemah Bahasa Isyarat Dengan Metode Cnn Berbasis Android,” J. Teknoinfo, vol. 16, no. 2, p. 183, 2022, doi: 10.33365/jti.v16i2.1752.

D. Novaliendry, K. Budayawan, R. Auvi, B. R. Fajri, and Y. Huda, “Design of Sign Language Learning Media Based on Virtual Reality,” Int. J. online Biomed. Eng., vol. 19, no. 16, pp. 111–126, 2023, doi: 10.3991/ijoe.v19i16.44671.

G. Amprimo, G. Masi, G. Olmo, and C. Ferraris, “Deep Learning for hand tracking in Parkinson’s Disease video-based assessment: Current and future perspectives,” Artif. Intell. Med., vol. 154, no. October 2023, 2024, doi: 10.1016/j.artmed.2024.102914.

Purwono, A. Ma’arif, W. Rahmaniar, H. I. K. Fathurrahman, A. Z. K. Frisky, and Q. M. U. Haq, “Understanding of Convolutional Neural Network (CNN): A Review,” Int. J. Robot. Control Syst., vol. 2, no. 4, pp. 739–748, 2022, doi: 10.31763/ijrcs.v2i4.888.

M. M. Taye, “Understanding of Machine Learning with Deep Learning :,” Comput. MDPI, vol. 12, no. 91, pp. 1–26, 2023.

N. Lubis, M. Z. Siambaton, and R. Aulia, “Implementasi Algoritma Deep Learning pada Aplikasi Speech to Text Online dengan Metode Recurrent Neural Network (RNN),” sudo J. Tek. Inform., vol. 3, no. 3, pp. 113–126, 2024, doi: 10.56211/sudo.v3i3.583.

I. Bakti and M. Firdaus, “Arsitektur Convolutional Neural Network InceptionResNet-V2 Untuk Pengelompokan Pneumonia Chest X-Ray,” J. Komput. dan Teknol., vol. 01, no. 02, pp. 35–42, 2023.

I. Šušter and T. Ranisavljević, “Optimization of MySQL database,” J. Process Manag. New Technol., vol. 11, no. 1–2, pp. 141–151, 2023, doi: 10.5937/jouproman2301141q.

J. W. Kim, J. Y. Choi, E. J. Ha, and J. H. Choi, “Human Pose Estimation Using MediaPipe Pose and Optimization Method Based on a Humanoid Model,” Appl. Sci., vol. 13, no. 4, 2023, doi: 10.3390/app13042700.

S. N. Koyineni, G. K. Sai, K. Anvesh, and T. Anjali, “Silent Expressions Unveiled: Deep Learning for British and American Sign Language Detection,” Procedia Comput. Sci., vol. 233, pp. 269–278, 2024, doi: 10.1016/j.procs.2024.03.216.

S. Supiyandi, M. Zen, C. Rizal, and M. Eka, “Perancangan Sistem Informasi Desa Tomuan Holbung Menggunakan Metode Waterfall,” JURIKOM (Jurnal Ris. Komputer), vol. 9, no. 2, p. 274, 2022, doi: 10.30865/jurikom.v9i2.3986.

J. Gupta, S. Pathak, and G. Kumar, “Deep Learning (CNN) and Transfer Learning: A Review,” J. Phys. Conf. Ser., vol. 2273, no. 1, 2022, doi: 10.1088/1742-6596/2273/1/012029.

U. Syach and S. W. M. Edi, “Perancangan Aplikasi Web Manajemen Data Produk Bisnis Perhiasan Berbasis Flask Dan Mongodb,” IT-Explore J. Penerapan Teknol. Inf. dan Komun., vol. 3, no. 2, pp. 162–176, 2024, doi: 10.24246/itexplore.v3i2.2024.pp162-176.