A Low Latency Python-Unity Integrated Framework for Real-Time BISINDO Recognition: A Comparative Analysis of Coordinate-Based Deep Learning Architectures

Automated Sign Language Recognition (SLR) is essential for bridging communication barriers between the Deaf and hearing communities. While appearance-based deep learning models achieve high accuracy on static datasets, their real-time integration into interactive graphical frontends often introduces significant computational latency and frame-rate drops. This paper presents a low-latency, cross-platform integrated framework for real-time Indonesian Sign Language (BISINDO) recognition, connecting a high-performance Python inference backend with an interactive Unity Engine frontend. By flattening multi-dimensional MediaPipe Holistic skeletal landmarks into a continuous sequential vector array (1 × 60 × 258), we eliminate resource-heavy raw RGB dependencies and preserve user privacy. We evaluate and compare four prominent deep learning architectures—LSTM, GRU, Temporal Convolutional Networks (TCN), and Transformers—under identical training configurations. To facilitate seamless real-time execution, a high-frequency cross-platform network bridge driven by connectionless User Datagram Protocol (UDP) sockets is established alongside a multi-threaded Unity listener loop. Experimental results show that the Gated Recurrent Unit (GRU) model achieves a superior offline validation accuracy of 100.0% while minimizing live real-time inference latency to 122.1 ms on consumer-grade integrated hardware. Furthermore, our streaming architecture completely eliminates network transit overhead (0 ms lag) and locks the Unity rendering pipeline at a fluid, stable 60 Frames Per Second (FPS). This work demonstrates a highly responsive, optimized framework that balances algorithmic precision with real-world interactivity for automated sign language translation. Accepted for presentation at AISDW2026 (Manuscript ID #1571317584), November 3-4, 2026.

Recommended citation: Chandra, V. P., Kwek, W. V., Tjandra, A. A. M., Tarigan, G. A., & Sutoyo, R. (2026). "A Low Latency Python-Unity Integrated Framework for Real-Time BISINDO Recognition: A Comparative Analysis of Coordinate-Based Deep Learning Architectures." The 2nd International Conference on Artificial Intelligence for Sustainability in the Developing World (AISDW2026).