Citation:

Anirudh Mustyala, Karthik Allam, 2022. "Automated Scaling and Load Balancing in Kubernetes for High-Volume Data Processing", ESP Journal of Engineering & Technology Advancements (ESP-JETA)  2(1): 24-38.

Abstract:

In today's fast-paced digital landscape, managing high-volume data processing is a critical challenge for many organizations. Kubernetes, an open-source container orchestration platform, has emerged as a powerful solution for automating scaling and load balancing, ensuring that applications remain performant and reliable under heavy loads. This abstract explores the capabilities of Kubernetes in handling large-scale data processing tasks, focusing on its automated scaling and load balancing features. Kubernetes' ability to dynamically scale resources based on demand ensures that applications can handle fluctuating workloads without manual intervention. This automation not only optimizes resource utilization but also reduces operational overhead. The platform's robust load balancing mechanisms distribute traffic evenly across containers, preventing bottlenecks and enhancing the overall system's resilience. Through real-world examples and case studies, this analysis demonstrates how Kubernetes' built-in tools and extensible architecture support efficient and effective high-volume data processing. By leveraging Kubernetes, organizations can achieve greater agility, improved performance, and heightened reliability, making it an indispensable tool for modern data-driven operations. This abstract provides a comprehensive overview of how Kubernetes' automated scaling and load balancing capabilities can be harnessed to meet the demands of high-volume data processing, highlighting its practical benefits and real-world applications.

References:

[1] Chindanonda, P., Podolskiy, V., & Gerndt, M. (2020). Self-adaptive data processing to improve slos for dynamic iot workloads. Computers, 9(1), 12.
[2] Lee, H. (2021). Scalable and High Available Kubernetes Cluster in Edge Environments for IoT Applications.
[3] Naik, N. (2017, October). Docker container-based big data processing system in multiple clouds for everyone. In 2017 IEEE International Systems Engineering Symposium (ISSE) (pp. 1-7). IEEE.
[4] Takahashi, K. (2019). A Study on Portable Load Balancer for Container Clusters (Doctoral dissertation, Graduate University for Advanced Studies, Japan).
[5] Bahiri, M. N., Zyane, A., Ghammaz, A., & Chassot, C. (2018). A new monitoring approach with cloud computing for autonomic middleware-level scalability management within IoT systems. In International Conference on Information Technology and Communication Systems (pp. 281-296). Springer International Publishing.
[6] Chindanonda, P., Podolskiy, V., & Gerndt, M. (2019, June). Metrics for self-adaptive queuing in middleware for internet of things. In 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS* W) (pp. 130-133). IEEE.
[7] Casadei, R., & Viroli, M. (2018, September). Collective abstractions and platforms for large-scale self-adaptive IoT. In 2018 IEEE 3rd International Workshops on Foundations and Applications of Self* Systems (FAS* W) (pp. 106-111). IEEE.
[8] Purswani, P. (2021, July). Self-adaptive IoT. In 2021 IEEE Symposium on Industrial Electronics & Applications (ISIEA) (pp. 1-6). IEEE.
[9] Padilla, F. J. A. (2016). Self-adaptation for Internet of things applications (Doctoral dissertation, Université Rennes 1).
[10] Apiletti, D., Barberis, C., Cerquitelli, T., Macii, A., Macii, E., Poncino, M., & Ventura, F. (2018, December). istep, an integrated self-tuning engine for predictive maintenance in industry 4.0. In 2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom) (pp. 924-931). IEEE.
[11] Zbakh, M., Bakhouya, M., Essaaidi, M., & Manneback, P. (2018). Cloud computing and big data: Technologies and applications. Concurrency and Computation: Practice and Experience, 30(12), e4517.
[12] Gotin, M., Lösch, F., Heinrich, R., & Reussner, R. (2018, March). Investigating performance metrics for scaling microservices in cloudiot-environments. In Proceedings of the 2018 ACM/SPEC International Conference on Performance Engineering (pp. 157-167).
[13] Zyane, A., Bahiri, M. N., & Ghammaz, A. (2020). IoTScal‐H: hybrid monitoring solution based on cloud computing for autonomic middleware‐level scalability management within IoT systems and different SLA traffic requirements. International Journal of Communication Systems, 33(14), e4495.
[14] Schepis, L., Cuomo, F., Petroni, A., Biagi, M., Listanti, M., & Scarano, G. (2019, July). Adaptive data update for cloud-based internet of things applications. In Proceedings of the ACM MobiHoc Workshop on Pervasive Systems in the IoT Era (pp. 13-18).
[15] Kim, H. W., Park, J. H., & Jeong, Y. S. (2016). Efficient resource management scheme for storage processing in cloud infrastructure with internet of things. Wireless Personal Communications, 91, 1635-1651.

Keywords:

Kubernetes, Data Processing, Load Balancing.