Efficient Resource Managing and Job Scheduling in a Heterogeneous Kubernetes Cluster for Big Data

doi:10.23940/ijpe.24.03.p4.157166

Abstract

Abstract: Cloud computing is an on-demand model of computing that utilizes virtualization expertise to offer cloud resources such as CPU, memory, storage, and network in virtual machines. As a result, most big data analytics in many modern enterprise applications are run from the cloud. Since resources in these private clouds are limited, getting the most out of resource applications and providing guaranteed user service by efficiently scheduling tasks and resources is the ultimate goal. However, existing big data processing system schedulers need to consider application performance and resource utilization when performing allocations. Therefore, it is difficult to design workflows for low turnaround time and high resource consumption in extensive data systems. In this paper, we propose a resource management system for efficient job scheduling, called RMS, which dynamically schedules big data jobs in Kubernetes cluster nodes for Spark applications and autonomously adjusts scheduling policies in heterogeneous node clusters to enhance application execution and resource consumption. The RMS mechanism will ensure adequate guidance and resources available in its planning objectives and satisfactory resource utilization. The experimental analysis of different RMS and performance preferences using different methods depends on the predicted completion time and the benchmark statistical result of different significant data performance indicators traces. The results show that RMS decreases the cost and scheduling overhead and improves job execution performance.

Key words: cloud computing, big data, job scheduling, resource management, kubernetes, spark

Jayanthi M and K. Ram Mohan Rao. Efficient Resource Managing and Job Scheduling in a Heterogeneous Kubernetes Cluster for Big Data [J]. Int J Performability Eng, 2024, 20(3): 157-166.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Islam M.T., Wu H., Karunasekera S., andBuyya R.Sla-Based Scheduling of Spark Jobs in Hybrid Cloud Computing Environments. IEEE Transactions on Computers, vol. 71, no. 5, pp. 1117-1132, 2021.
[2] Huang Y., Xu H., Gao H., Ma X., andHussain W.SSUR: An Approach to Optimizing Virtual Machine Allocation Strategy Based on User Requirements for Cloud Data Center. IEEE Transactions on Green Communications and Networking, vol. 5, no. 2, pp. 670-681, 2021.
[3] Ma X., Xu H., Gao H., andBian M.Real-Time Multiple-Workflow Scheduling in Cloud Environments. IEEE Transactions on Network and Service Management, vol. 18, no. 4, pp. 4002-4018, 2021.
[4] Zhang S., Wang C., andZomaya A.Y.Robustness Analysis and Enhancement of Deep Reinforcement Learning-Based Schedulers. IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 1, pp. 346-357, 2022.
[5] Li Y., Li T., Shen P., Hao L., Yang J., Zhang Z., Chen J., andBao L.PAS: Performance-Aware Job Scheduling for Big Data Processing Systems. Security and Communication Networks, vol. 2022, 2022.
[6] Yang R., Hu C., Sun X., Garraghan P., Wo T., Wen Z., Peng H., Xu J., andLi C.Performance-Aware Speculative Resource Oversubscription for Large-Scale Clusters. IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 7, pp. 1499-1517, 2020.
[7] Zhu J., Li X., Ruiz R., Li W., Huang H., andZomaya A.Y.Scheduling Periodical Multi-Stage Jobs with Fuzziness to Elastic Cloud Resources. IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 12, pp. 2819-2833, 2020.
[8] Zhou X., Liang W., Yan K., Li W., Kevin I., Wang K., Ma J., andJin Q.Edge-Enabled Two-Stage Scheduling Based on Deep Reinforcement Learning for Internet of Everything. IEEE Internet of Things Journal, vol. 10, no. 4, pp. 3295-3304, 2022.
[9] Zhu L., Huang K., Hu Y., andTai X.A Self-Adapting Task Scheduling Algorithm for Container Cloud using Learning Automata. IEEE Access, vol. 9, pp. 81236-81252, 2021.
[10] Zhang X., Li L., Wang Y., Chen E., andShou L.Zeus: Improving Resource Efficiency via Workload Colocation for Massive Kubernetes Clusters. IEEE Access, vol. 9, pp. 105192-105204, 2021.
[11] Meyer V., Kirchoff D.F., Da Silva, M.L., and De Rose, C.A. ML-Driven Classification Scheme for Dynamic Interference-Aware Resource Scheduling in Cloud Infrastructures. Journal of Systems Architecture, vol. 116, pp. 102064, 2021.
[12] Khan M., Jin Y., Li M., Xiang Y., andJiang C.Hadoop Performance Modeling for Job Estimation and Resource Provisioning. IEEE Transactions on Parallel and Distributed Systems, vol. 27, no. 2, pp. 441-454, 2015.
[13] Ghodsi A., Zaharia M., Hindman B., Konwinski A., Shenker S., andStoica I.Dominant Resource Fairness: Fair Allocation of Multiple Resource Types. In8th USENIX symposium on networked systems design and implementation (NSDI 11), 2011.
[14] Sharkh M.A., Ouda A., andShami A.A Resource Scheduling Model for Cloud Computing Data Centers. In2013 9th International Wireless Communications and Mobile Computing Conference (IWCMC), IEEE, pp. 213-218, 2013.
[15] Nathani A., Chaudhary S., andSomani G.Policy Based Resource Allocation in IaaS Cloud. Future Generation Computer Systems, vol. 28, no. 1, pp. 94-103, 2012.
[16] Chen R., Chen X., andYang, C. using a Task Dependency Job-Scheduling Method to Make Energy Savings in a Cloud Computing Environment. The Journal of Supercomputing, vol. 78, no. 3, pp. 4550-4573, 2022.
[17] Cheng F., Huang Y., Tanpure B., Sawalani P., Cheng L., andLiu C.Cost-Aware Job Scheduling for Cloud Instances using Deep Reinforcement Learning.Cluster Computing, pp. 1-13, 2022.
[18] Amer D.A., Attiya G., Zeidan I., andNasr A.A.Elite Learning Harris Hawks Optimizer for Multi-Objective Task Scheduling in Cloud Computing.The Journal of Supercomputing, pp. 1-26, 2022.
[19] Khan M.S.A. and Santhosh, R. Task Scheduling in Cloud Computing using Hybrid Optimization Algorithm. Soft Computing, vol. 26, no. 23, pp. 13069-13079, 2022.
[20] Fan Y.Job Scheduling in High Performance Computing.arXiv preprint arXiv:2109.09269, 2021.
[21] Zheng B., Pan L., andLiu S.Market-Oriented Online Bi-Objective Service Scheduling for Pleasingly Parallel Jobs with Variable Resources in Cloud Environments. Journal of Systems and Software, vol. 176, pp. 110934, 2021.
[22] Shao Y., Li C., Gu J., Zhang J., andLuo Y.Efficient Jobs Scheduling Approach for Big Data Applications. Computers & Industrial Engineering, vol. 117, pp. 249-261, 2018.
[23] Chen, K. and Huang, L.Timely-Throughput Optimal Scheduling with Prediction. IEEE/ACM Transactions on Networking, vol. 26, no. 6, pp. 2457-2470, 2018.
[24] Hou X., Kumar T.A., Thomas J.P., andLiu H.Dynamic Deadline-Constraint Scheduler for Hadoop YARN. In2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), IEEE, pp. 1-8, 2017.
[25] Wang, Y. and Shi, W.Budget-Driven Scheduling Algorithms for Batches of MapReduce Jobs in Heterogeneous Clouds. IEEE Transactions on Cloud Computing, vol. 2, no. 3, pp. 306-319, 2014.
[26] Yao Y., Wang J., Sheng B., Lin J., andMi N.Haste: Hadoop Yarn Scheduling Based on Task-Dependency and Resource-Demand. In2014 IEEE 7th international conference on cloud computing, IEEE, pp. 184-191, 2014.
[27] Niu Z., Tang S., andHe B.An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing. IEEE Transactions on Services Computing, vol. 12, no. 6, pp. 865-879, 2016.
[28] Wang, Q. and Huang, X.Pft: A Performance-Fairness Scheduler on Hadoop Yarn. In2016 7th IEEE International Conference on Software Engineering and Service Science (ICSESS), IEEE, pp. 76-80, 2016.
[29] Zaharia M., Borthakur D., Sen Sarma, J., Elmeleegy, K., Shenker, S., and Stoica, I. Delay Scheduling: A Simple Technique for Achieving Locality and Fairness in Cluster Scheduling. InProceedings of the 5th European conference on Computer systems, pp. 265-278, 2010.
[30] Tang S., Lee B.S., andHe B.Dynamicmr: A Dynamic Slot Allocation Optimization Framework for Mapreduce Clusters. IEEE Transactions on Cloud Computing, vol. 2, no. 3, pp. 333-347, 2014.
[31] Selvarani, S. and Sadhasivam, G.S.Improved Cost-Based Algorithm for Task Scheduling in Cloud Computing. In2010 IEEE International Conference on Computational Intelligence and Computing Research, IEEE, pp. 1-5, 2010.
[32] Li J., Qiu M., Niu J., Gao W., Zong Z., andQin X.Feedback Dynamic Algorithms for Preemptable Job Scheduling in Cloud Systems. In 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, IEEE, vol. 1, pp. 561-564, 2010.
[33] Huang Z., Balasubramanian B., Wang M., Lan T., Chiang M., andTsang D.H.Rush: A Robust Scheduler to Manage Uncertain Completion-Times in Shared Clouds. In2016 IEEE 36th International Conference on Distributed Computing Systems (ICDCS), IEEE, pp. 242-251, 2016.
[34] Yang Y., Zhou Y., Sun Z., andCruickshank H.Heuristic Scheduling Algorithms for Allocation of Virtualized Network and Computing Resources. Journal of Software Engineering and Applications, vol. 6, no. 1, pp. 1-13, 2013.
[35] Ghanbari, S. and Othman, M.A Priority Based Job Scheduling Algorithm in Cloud Computing. Procedia Engineering, vol. 50, no. 0, pp. 778-785, 2012.
[36] Sgall J.A New Analysis of Best Fit Bin Packing. In International Conference on Fun with Algorithms, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 315-321, 2012.
[37] Wang L., Zhan J., Luo C., Zhu Y., Yang Q., He Y., Gao W., Jia Z., Shi Y., Zhang S., andZheng C.Bigdatabench: A Big Data Benchmark Suite from Internet Services. In2014 IEEE 20th international symposium on high performance computer architecture (HPCA), IEEE, pp. 488-499, 2014.
[38] Islam M.T., Srirama S.N., Karunasekera S., andBuyya R.Cost-Efficient Dynamic Scheduling of Big Data Applications in Apache Spark on Cloud. Journal of Systems and Software, vol. 162, pp. 110515, 2020.
[39] Jyothi S.A., Curino C., Menache I., Narayanamurthy S.M., Tumanov A., Yaniv J., Mavlyutov R., Goiri I., Krishnan S., Kulkarni J., andRao S.Morpheus: Towards Automated {SLOs} for Enterprise Clusters. In12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 117-134, 2016.