Citation:

Nishanth Reddy Mandala, 2022. "Data Engineering in Cloud-Native Architectures", ESP Journal of Engineering & Technology Advancements 2(2): 135-145.

Abstract:

The rise of cloud-native architectures has transformed the way organizations manage and process data. Data engineering in the cloud involves building scalable, resilient, and automated pipelines that can handle large volumes of data in real-time. This paper explores the design principles, tools, and methodologies for data engineering in cloud-native environments. By focusing on cloud services such as Kubernetes, Docker, serverless computing, and container orchestration, the paper highlights the advantages of cloud-native systems for modern data architectures. A case study and performance analysis are presented to demonstrate the efficiency and scalability of these architectures.

References:

[1] M. Armbrust, A. Fox, R. Griffith, et al., ”A View of Cloud Computing,” Communications of the ACM, vol. 53, no. 4, pp. 50–58, 2010.

[2] R. Buyya, C. S. Yeo, and S. Venugopal, ”Cloud Computing and Emerging IT Platforms: Vision, Hype, and Reality for Delivering Computing as the 5th Utility,” Future Generation Computer Systems, vol. 25, no. 6, pp. 599–616, 2009.

[3] A. Fox, R. Griffith, A. Joseph, et al., ”Above the Clouds: A Berkeley View of Cloud Computing,” UC Berkeley Reliable Adaptive Distributed Systems Laboratory, 2009.

[4] J. Dean and S. Ghemawat, ”MapReduce: Simplified Data Processing on Large Clusters,” Communications of the ACM, vol. 51, no. 1, pp. 107–113, 2008.

[5] D. Chappell, ”Enterprise Service Bus,” O’Reilly Media, 2009.

[6] M. Zaharia, M. Chowdhury, T. Das, et al., ”Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing,” in Proceedings of the 9th USENIX Symposium on Networked Systems Design and Implementation, 2008.

[7] P. A. Bernstein, ”Middleware: A Model for Distributed System Services,” Communications of the ACM, vol. 39, no. 2, pp. 86–98, 2006.

[8] I. Foster, Y. Zhao, I. Raicu, and S. Lu, ”Cloud Computing and Grid Computing 360-Degree Compared,” in Grid Computing Environments Workshop, IEEE, 2008, pp. 1–10.

[9] Y. Zhao, X. Ma, and D. Li, ”Data Provenance in Cloud Computing,” in International Conference on Intelligent Computing and Intelligent Systems, 2009, pp. 194–199.

[10] M. Armbrust, A. Fox, R. Griffith, et al., ”Above the Clouds: A Berkeley View of Cloud Computing,” Technical Report UCB/EECS-2009-28, 2009.

Keywords:

Data Engineering, Cloud Computing, CloudNative Architectures, Kubernetes, Docker, Serverless, Data Pipelines.