Citation:

Ranjit Kumar Gupta, Sagar Shukla, Anaswara Thekkan Rajan, Sneha Aravind, 2021. "Utilizing Splunk for Proactive Issue Resolution in Full Stack Development Projects" ESP Journal of Engineering & Technology Advancements  1(1): 57-64.

Abstract:

With the help of various sources of unstructured and structured data like system and application monitoring streams, open-source knowledge capture, and on-demand simulation output, exascale data environments are quickly approaching. With storage prices so low, the challenge today lies in turning massive data repositories into useful information. Any company, organisation, or agency can use log data as a definitive recording of what's happening, and it's frequently an underutilised resource for troubleshooting and supporting larger business goals. For the purpose of combining and indexing any type of log or machine-generated information, including complicated multi-line application logs that are complex and unstructured, Splunk offers the best software in the industry. Any machine-generated data can be gathered, stored, indexed, searched, correlated, visualised, analysed, and reported on in order to quickly, reliably, and affordably discover and address operational and security issues. Nowadays, most businesses and organisations use Information Technology (IT), which is the use of networked computers, and physical equipment to create, process, and share electronic information. While one significant danger to IT-based networks and systems is cyberattacks. Every firm has information security procedures in place since an attack that succeeds has the potential to result in significant financial loss. Logging and monitoring are crucial security measures that shield an organisation from potential threats. Specifically, threat identification is an important method for locating invaders. In this research, we examine the threat hunting utility of the Elastic stack tool and contrast it with four other tools of a similar nature. When used in a large-scale setting, the Elastic Stack tool has been shown to be both cost-effective and efficient in identifying threats and breaches of security.

References:

[1] Janos and N. Dai, “Security concerns towards Security Operations Centers”, in 2018 IEEE 12th International Symposium on Applied Computational Intelligence and Informatics (SACI), IEEE, 2018, pp. 000273–000278.
[2] A. Crowley and J. Pescatore, “The Definition of SOC-cess? SANS 2018 Security Operations Center Survey”, SANS Institute Reading Room, SANS Institute, 2018.
[3] (ISC) 2, “Cybersecurity Professionals Stand Up to a Pandemic”, (ISC) 2 Cybersecurity Workforce Study 2020, (ISC)2 , 2020.
[4] J. Pescatore and B. Filkins, “Closing the Critical Skills Gap for Modern and Effective Security Operations Centers (SOCs)”, SANS Institute Reading Room, SANS Institute, 2021.
[5] B. Filkins, “2019 SANS Automation and Integration Survey”, SANS Institute Reading Room, SANS Institute, 2019.
[6] S. Sundaramurthy, et al., “A Human Capital Model for Mitigating Security Analyst Burnout”, Eleventh Symposium On Usable Privacy and Security (SOUPS), 2015, pp. 347–359. [16] D. Murdoch, “2020 SANS Automation and Integration Survey Integration Survey”, SANS Institute Reading Room, SANS Institute, 2021.
[7] A. Cole, “SOC Automation - Deliverance or Disaster", SANS Institute Reading Room, SANS Institute, 2017.
[8] Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber. Bigtable: A distributed storage system for structured data. In Proc. 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI), pages 205–218, 2006.
[9] Greg Eisenhauer, Hasan Abbasi, Matthew Wolf, and Karsten Schwan. Event-based systems: opportunities and challenges at exascale. In Proc. Third ACM International Conference on Distributed Event-Based Systems, Nashville, Tennessee, July 2009. ACM.
[10] Greg Eisenhauer, Matthew Wolf, Hasan Abbasi, Scott Klasky, and Karsten Schwan. A type system for high performance communication and computation. In Proc. 2011 D3 science Workshop, Stockholm, Sweden, December 2011. IEEE. Associated with the 7th IEEE International Conference on e-Science.
[11] Susanne Busse et al. Federated Information Systems: Concepts, Terminology, and Architectures. Technische Universitat Berlin, 1999.¨
[12] Simon Garfunkel, Paul Farrell, Vassil Roussev, and George Dinolt. Bringing science to digital forensics with standardized forensic corpora. In Proc. Digital Forensic Research Workshop, Montreal, Canada, 2009. Elsevier.
[13] N. Bosch and J. Bosch, “Software logs for machine learning in a devops environment,” in 2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 29–33, 2020.
[14] M. Kersten, “A cambrian explosion of devops tools,” IEEE Software, vol. 35, no. 2, pp. 14–17, 2018.
[15] P. Agrawal and N. Rawat, “Devops, a new approach to cloud development testing,” in 2019 International Conference on Issues and Challenges in Intelligent Computing Techniques (ICICT), vol. 1, pp. 1–4, 2019.
[16] J. Henkel, C. Bird, S. K. Lahiri, and T. Reps, “Learning from, understanding, and supporting devops artifacts for docker,” in 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE), pp. 38–49, 2020.
[17] J. Rufino, M. Alam, and J. Ferreira, “Monitoring v2x applications using devops and docker,” in 2017 International Smart Cities Conference (ISC2), pp. 1–5, 2017.
[18] C. Ebert, G. Gallardo, J. Hernantes, and N. Serrano, “Devops,” IEEE Software, vol. 33, no. 3, pp. 94–100, 2016.
[19] R. K. Pratibha Jha, “A review paper on devops: Beginning and more to know,” pp. 1–5, 06 2018.
[20] K. Yao, M. Sayagh, W. Shang, and A. E. Hassan, “Improving state-of-the-art compression techniques for log management tools,” IEEE Transactions on Software Engineering, pp. 1–1, 2021.
[21] C.-T. Yang, E. Kristiani, Y.T. Wang, G. Min, C.H. Lai, and W.J. Jiang, “On construction of a network log management system using ELK Stack with Ceph,” J. Supercomput. 76(8), pp. 6344-6360, 2020.
[22] M. Beechey, K.G. Kyriakopoulos, and S. Lambotharan, “Evidential classification and feature selection for cyber-threat hunting,” Knowl. Based Syst. 226, 107120, 2021.
[23] W. Meng, W. Li, and L.F. Kwok, “Towards Effective Trust-based Packet Filtering in Collaborative Network Environments,” IEEE Transactions on Network and Service Management, vol. 14, no. 1, pp. 233-245, 2017.
[24] S.J. Son and Y. Kwon, “Performance of ELK stack and commercial system in security log analysis,” in Proc. IEEE Malaysia International Conference on Communications (MICC), 2017.
[25] I.Y.M. Al-Mahbashi, M.B. Potdar, and P. Chauhan, “Network security enhancement through effective log analysis using ELK,” in Proc. ICCMC, pp. 566-570, 2018.
[26] Al-Mohannadi, I. Awan, J.A. Hamar, A.J. Cullen, J.P. Disso, and L. Armitage, “Cyber Threat Intelligence from Honeypot Data Using Elastic search,” in Proc. AINA, pp. 900-906, 2018.
[27] P. P. I. Langi, Widyawan, W. Najib, and T. B. Aji, “An evaluation of twitter river and logstash performances as elasticsearch inputs for social media analysis of twitter,” in 2015 International Conference on Information Communication Technology and Systems (ICTS), pp. 181–186, 2015.
[28] O. Andreassen, C. CharrondiA¨re, and A. De Dios Fuente, “Monitoring Mixed- ˜ Language Applications with Elastic Search, Logstash and Kibana (ELK),” p. WEPGF041. 4 p, 2015.
[29] B. Purnachandra Rao and N. Nagamalleswara Rao, HDFS Logfile Analysis Using ElasticSearch, LogStash and Kibana, pp. 185–191. Singapore: Springer Singapore, 2019.
[30] C. He, “Using logstash and elasticsearch to achieve real-time statistical analysis of dspace logs,” Data Analysis and Knowledge.
[31] Kaur, J., Choppadandi, A., Chenchala, P. K., Nakra, V., & Pandian, P. K. G. (2019). AI Applications in Smart Cities: Experiences from Deploying ML Algorithms for Urban Planning and Resource Optimization. Tuijin Jishu/Journal of Propulsion Technology, 40(4), 50-56.
[32] Case Studies on Improving User Interaction and Satisfaction using AI-Enabled Chatbots for Customer Service . (2019). International Journal of Transcontinental Discoveries, ISSN: 3006-628X, 6(1), 29-34. https://internationaljournals.org/index.php/ijtd/article/view/98
[33] Kaur, J., Choppadandi, A., Chenchala, P. K., Nakra, V., & Pandian, P. K. G. (2019). Case Studies on Improving User Interaction and Satisfaction using AI-Enabled Chatbots for Customer Service. International Journal
[34] ofTranscontinental Discoveries, 6(1), 29-34. https://internationaljournals.org/index.php/ijtd/article/view/98
[35] Choppadandi, A., Kaur, J., Chenchala, P. K., Kanungo, S., & Pandian, P. K. K. G. (2019). AI-Driven Customer Relationship Management in PK Salon Management System. International Journal of Open Publication and Exploration, 7(2), 28-35. https://ijope.com/index.php/home/article/view/128
[36] AI-Driven Customer Relationship Management in PK Salon Management System. (2019). International Journal of Open Publication and Exploration, ISSN: 3006-2853, 7(2), 28-35. https://ijope.com/index.php/home/article/view/128
[37] Big Data Analytics using Machine Learning Techniques on Cloud Platforms. (2019). International Journal of Business Management and Visuals, ISSN: 3006-2705, 2(2), 54-58. https://ijbmv.com/index.php/home/article/view/76
[38] Ashok : "Choppadandi, A., Kaur, J.,Chenchala, P. K., Nakra, V., & Pandian, P. K. K. G. (2020). Automating ERP Applications for Taxation Compliance using Machine Learning at SAP Labs. International Journal of Computer Science and Mobile Computing, 9(12), 103-112. https://doi.org/10.47760/ijcsmc.2020.v09i12.014
[39] Chenchala, P. K., Choppadandi, A., Kaur, J., Nakra, V., & Pandian, P. K. G. (2020). Predictive Maintenance and Resource Optimization in Inventory Identification Tool Using ML. International Journal of Open Publication and Exploration, 8(2), 43-50. https://ijope.com/index.php/home/article/view/127
[40] Predictive Maintenance and Resource Optimization in Inventory Identification Tool Using ML. International Journal of Open Publication and Exploration, 8(2), 43-50. https://ijope.com/index.php/home/article/view/127
[41] AI-Driven Customer Relationship Management in PK Salon Management System. (2019). International Journal of Open Publication and Exploration, ISSN: 3006-2853, 7(2), 28-35. https://ijope.com/index.php/home/article/view/128
[42] Big Data Analytics using Machine Learning Techniques on Cloud Platforms. (2019). International Journal of Business Management and Visuals, ISSN: 3006-2705, 2(2), 54-58. https://ijbmv.com/index.php/home/article/view/76
[43] Fadnavis, N. S., Patil, G. B., Padyana, U. K., Rai, H. P., & Ogeti, P. (2020). Machine learning applications in climate modeling and weather forecasting. NeuroQuantology, 18(6), 135-145. https://doi.org/10.48047/nq.2020.18.6.NQ20194
[44] Tilala, Mitul, and Abhip Dilip Chawda. "Evaluation of Compliance Requirements for Annual Reports in Pharmaceutical Industries." NeuroQuantology 18, no. 11 (November 2020): 138-145. https://doi.org/10.48047/nq.2020.18.11.NQ20244.

Keywords:

Data Environments, Splunk Provides, Elastic Stack, Important Security, Log Data, Logging And Monitoring, IT, Operational Momentum, Large-Scale Environment, Cost-Efficient, Maintaining Research, Open-Source, Cyberattacks.