Abstract
This paper aims to address the problem of scheduling large workflows onto multiple execution sites with storage constraints. Three heuristics are proposed to first partition the workflow into sub-workflows. Three estimators and two schedulers are then used to schedule sub-workflows to the execution sites. Performance with three real-world workflows shows that this approach is able to satisfy storage constraints and improve the overall runtime by up to 48% over a default whole-workflow scheduling.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Taylor, I.J., Deelman, E., et al.: Workflows for e-Science. In: Scientific Workflows for Grids. Springer (2007)
Berriman, G.B., et al.: Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand. In: Proc. of SPIE, vol. 5493, pp. 221–232 (2004)
Maechling, P., Deelman, E., et al.: SCEC CyberShake Workflows–Automating Probabilistic Seismic Hazard Analysis Calculations. In: Workflows for e-Science. Scientific Workflows for Grids. Springer (2007)
USC Epigenome Center, http://epigenome.usc.edu
Deelman, E., et al.: Pegasus: A framework for mapping complex scientific workflows onto distributed systems. Sci. Program 13, 219–237 (2005)
Litzkow, M., et al.: Condor–A Hunter of Idle Workstations. In: ICDCS (June 1988)
Blythe, J., et al.: Task Scheduling Strategies for Workflow-Based Applications in Grids. In: CCGrid (2005)
Topcuoglu, H., et al.: Performance-Effective and Low-Complexity Task Scheduling for Heterogeneous Computing. IEEE TPDS 13(3) (March 2002)
Duan, R., et al.: Run-time Optimisation of Grid Workflow Applications. In: 7th IEEE/ACM Intl. Conf. on Grid Computing (September 2005)
Sonmez, O.O.: Application-Oriented Scheduling in Multicluster Grids (June 2010), http://www.pds.ewi.tudelft.nl/~homedirsonmez/research.htm
Singh, G., et al.: Optimizing Workflow Data Footprint. Special issue of the Scientific Programming Journal dedicated to Dynamic Computational Workflows: Discovery, Optimisation and Scheduling (2007)
Papadimitriou, C.H., et al.: Combinatorial Optimization: Algorithms and Complexity, Dover, pp. 120–128 (1998) ISBN 0486402584
Eucalyptus Systems, http://www.eucalyptus.com/
Condor Team, http://www.cs.wisc.edu/condor/dagman
Juve, G., et al.: Scientific Workflow Applications on Amazon EC2, E-Science Workshops, Oxford UK (December 2009)
Wieczorek, M., et al.: Scheduling of scientific workflows in the ASKALON grid environment. SIGMOND Record 34(3) (September 2005)
FutureGrid, https://portal.futuregrid.org
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chen, W., Deelman, E. (2012). Partitioning and Scheduling Workflows across Multiple Sites with Storage Constraints. In: Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J. (eds) Parallel Processing and Applied Mathematics. PPAM 2011. Lecture Notes in Computer Science, vol 7204. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31500-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-31500-8_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31499-5
Online ISBN: 978-3-642-31500-8
eBook Packages: Computer ScienceComputer Science (R0)