A Hybrid Framework of Resource Allocation using Firefly and Deep Learning in Big Data Scheduling

doi:10.23940/ijpe.24.06.p1.333343

Abstract

Abstract: Effective resource allocation is crucial for optimizing performance and efficiency in big data processing environments. In this study, we propose a novel algorithm that integrates advanced optimization techniques, including swarm intelligence-based firefly algorithm and deep learning-based resource allocation, proactive load balancing mechanisms, and holistic resource management strategies to address the complex challenges of resource allocation in large-scale big data infrastructures. The proposed algorithm is evaluated across key performance metrics, including energy efficiency, resource utilization, and SLA compliance, and compared against existing approaches. Results demonstrate significant improvements in energy efficiency, with an average power consumption of 5410 watts, average CPU utilization of 10240.125 Hz, and average SLA violation of 0.033625. These findings highlight the algorithm's effectiveness in optimizing resource allocation and enhancing system performance.

Key words: big data, resource allocation, firefly, deep learning

Rohit Kumar Verma and Sukhvir Singh. A Hybrid Framework of Resource Allocation using Firefly and Deep Learning in Big Data Scheduling [J]. Int J Performability Eng, 2024, 20(6): 333-343.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Dai W., Qiu L., Wu A. and Qiu M., 2016. Cloud infrastructure resource allocation for big data applications. IEEE Transactions on Big Data, 4(3), pp.313-324.
[2] Pimpley A., Li S., Sen R., Srinivasan S. and Jindal A., 2022. Towards Optimal Resource Allocation for Big Data Analytics. In EDBT (Vol. 22, pp. 2-338).
[3] Baresi L., Leva A. and Quattrocchi G., 2019. Fine-grained dynamic resource allocation for big-data applications. IEEE Transactions on Software Engineering, 47(8), pp.1668-1682.
[4] Satish, K.R. and Kavya, N.P., 2014, November. Big data processing with harnessing hadoop-MapReduce for optimizing analytical workloads. In 2014 International Conference on Contemporary Computing and Informatics (IC3I)(pp. 49-54). IEEE.
[5] Jiang Y., Huang Z. and Tsang D.H., 2016. Towards max-min fair resource allocation for stream big data analytics in shared clouds. IEEE Transactions on Big Data, 4(1), pp.130-137.
[6] Li J., Lu Z., Zhang W., Wu J., Qiang H., Li B. and Hung P.C., 2018. SERAC3: Smart and economical resource allocation for big data clusters in community clouds. Future Generation Computer Systems, 85, pp.210-221.
[7] Tzanetos, A. and Dounias, G., 2020. A comprehensive survey on the applications of swarm intelligence and bio-inspired evolutionary strategies. Machine learning paradigms: advances in deep learning-based technological applications, pp.337-378.
[8] Abualigah L., Gandomi A.H., Elaziz M.A., Hamad H.A., Omari M., Alshinwan M. and Khasawneh A.M., 2021. Advances in meta-heuristic optimization algorithms in big data text clustering. Electronics, 10(2), p.101.
[9] Nguyen B.H., Xue B. and Zhang M., 2020. A survey on swarm intelligence approaches to feature selection in data mining. Swarm and Evolutionary Computation, 54, p.100663.
[10] Kumar, S. and Goyal, S.K., 2022, May. Swarm Intelligence Based Data Selection Mechanism for Reputation Generation in Social Cloud. In 2022 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COM-IT-CON) (Vol. 1, pp. 583-588). IEEE.
[11] Brezočnik L., Fister Jr I. and Podgorelec V., 2018. Swarm intelligence algorithms for feature selection: a review. Applied Sciences, 8(9), p.1521.
[12] Ari A.A.A., Gueroui A., Titouna C., Thiare O. and Aliouat Z., 2019. Resource allocation scheme for 5G C-RAN: a Swarm Intelligence based approach. Computer Networks, 165, p.106957.
[13] Abraham A., Das S. and Roy S., 2008. Swarm intelligence algorithms for data clustering. In Soft computing for knowledge discovery and data mining (pp. 279-313). Boston, MA: Springer US.
[14] Senthilkumar, M. and Ilango, P., 2020. Energy aware task scheduling using hybrid firefly-GA in big data. International Journal of Advanced Intelligence Paradigms, 16(2), pp.99-112.
[15] Luo R., Liu L., Tan D. and Yin S., 2019, May. Scheduling feature selection for data-driven job shop scheduling system using improved firefly algorithm optimization. In 2019 International Conference on High Performance Big Data and Intelligent Systems (HPBD&IS)(pp. 116-121). IEEE.
[16] Harb, H. and Makhoul, A., 2019. Energy-efficient scheduling strategies for minimizing big data collection in cluster-based sensor networks. Peer-to-Peer Networking and Applications, 12, pp.620-634.
[17] Talwani S., Singla J., Mathur G., Malik N., Jhanjhi N.Z., Masud M. and Aljahdali S., 2022. Machine-learning-based approach for virtual machine allocation and migration. Electronics, 11(19), p.3249.
[18] Esmaeili H., Hakami V., Bidgoli B.M. and Shokouhifar M., 2022. Application-specific clustering in wireless sensor networks using combined fuzzy firefly algorithm and random forest. Expert Systems with Applications, 210, p.118365.