Architecting Secure Big Data Platforms: Insights from a Big Data and IT Security Specialist
DOI:
https://doi.org/10.64137/3107-9458/ICCSEMTI26-120Keywords:
Big Data Security, Secure Data Architecture, IT Security, Distributed Systems, Data Privacy, Zero Trust, Cloud Security, ComplianceAbstract
Nowadays organizations gather process and derive value from massive volumes of structured and unstructured data differently due to the rapid expansion of big data platforms. Organizations that are increasingly operating in distributed ecosystems, consisting of cloud infrastructures, data lakes, real-time analytics pipelines, and third-party integrations, are facing security and privacy risks of big data and these risks have been increased drastically. For example, unauthorized access, data leakage, insider attacks, and compliance violations, among other threats, are making it so that perimeter-based security models are no longer able to protect sensitive data. The central message of this article is that instead of adding security afterward, we need to be embedding security mechanisms directly into the core of big data platforms in a security-by-design architecture. Considering the input of a Big Data and IT Security Specialist, the paper develops a secure big data architecture that, inter alia, includes data encryption, fine-grained access control, identity and key management, secure data ingestion, and continuous monitoring as part of this comprehensive framework. The case of practical implementation featuring an enterprise-scale big data deployment served as the ground for the application of the proposed model, wherein the security controls were related to the working of practical operational workflows as well as regulatory requirements. Among the most important conclusions drawn is that applying security-by-design principles to data protection and compliance not only leads to better security and compliance but also improves the platform's reliability and stakeholder trust. The paper wraps up by highlighting the practical implications for architects and security practitioners, basically providing doable tools for them to construct strong, secure, and scalable big data platforms in today's enterprise environments.
References
[1] Ardagna, Claudio A., et al. "Big Data Analytics-as-a-Service: Bridging the gap between security experts and data scientists." Computers & Electrical Engineering 93 (2021): 107215.
[2] Bansal, Bijender, et al. "Big data architecture for network security." Cyber Security and Network Security (2022): 233-267.
[3] Narayanan, Uma, Varghese Paul, and Shelbi Joseph. "A novel system architecture for secure authentication and data sharing in cloud enabled Big Data Environment." Journal of King Saud University-Computer and Information Sciences 34.6 (2022): 3121-3135.
[4] Rawat, Danda B., Ronald Doku, and Moses Garuba. "Cybersecurity in big data era: From securing big data to data-driven security." IEEE Transactions on Services Computing 14.6 (2019): 2055-2072.
[5] Awaysheh, Feras M., et al. "Security by design for big data frameworks over cloud computing." IEEE Transactions on Engineering Management 69.6 (2021): 3676-3693.
[6] Ramesh, Bhashyam. "Big data architecture." Big Data: A Primer. New Delhi: Springer India, 2015. 29-59.
[7] Fetjah, Laila, et al. "Toward a big data architecture for security events analytic." 2016 IEEE 3rd international conference on cyber security and cloud computing (CSCloud). IEEE, 2016.
[8] Anwar, Memoona J., et al. "Secure big data ecosystem architecture: challenges and solutions." EURASIP Journal on Wireless Communications and Networking 2021.1 (2021): 130.
[9] Wang, Jin, et al. "Big data service architecture: a survey." Journal of Internet Technology 21.2 (2020): 393-405.
[10] Asch, Mark, et al. "Big data and extreme-scale computing: Pathways to convergence-toward a shaping strategy for a future software and data ecosystem for scientific inquiry." The International Journal of High Performance Computing Applications 32.4 (2018): 435-479.
[11] Gharaibeh, Ammar, et al. "Smart cities: A survey on data management, security, and enabling technologies." IEEE communications surveys & tutorials 19.4 (2017): 2456-2501.
[12] Nwaimo, CHIOMA SUSAN, OLUCHUKWU MODESTA Oluoha, and O. Y. E. W. A. L. E. Oyedokun. "Big data analytics: technologies, applications, and future prospects." Iconic Research and Engineering Journals 2.11 (2019): 411-419.
[13] Georgiadis, Georgios, and Geert Poels. "Enterprise architecture management as a solution for addressing general data protection regulation requirements in a big data context: a systematic mapping study." Information Systems and e-Business Management 19.1 (2021): 313-362.
[14] Hu, Jiankun, and Athanasios V. Vasilakos. "Energy big data analytics and security: challenges and opportunities." IEEE Transactions on Smart Grid 7.5 (2016): 2423-2436.
[15] Moorthy, Janakiraman, et al. "Big data: Prospects and challenges." Vikalpa 40.1 (2015): 74-96.
[16] Reddy, R. R. P. (2024). ZERO TRUST-BASED SECURE CREDENTIAL PHISHING DETECTION FRAMEWORK USING Ellsigm-GRU. Journal Homepage: http://www. ijmra. us, 14(04).
[17] Jonnalagadda, R. R., Reddy, K. K., Gunupati, K., Kumar, M., Reddy, P. R. R., & Julakanti, R. (2025, September). Design and Implementation of a Novel AI-Based Cyber Security Architecture for IoT Devices and Networks Using Machine Learning and Big Data Analytics. In 2025 International Conference on Computing and Communications (COMPUTINGCON) (pp. 1-6). IEEE.
