Sie befinden Sich nicht im Netzwerk der Universität Paderborn. Der Zugriff auf elektronische Ressourcen ist gegebenenfalls nur via VPN oder Shibboleth (DFN-AAI) möglich. mehr Informationen...
The IceProd framework: Distributed data processing for the IceCube neutrino observatory
Ist Teil von
Journal of parallel and distributed computing, 2015-01, Vol.75 (C), p.198-211
Ort / Verlag
United States: Elsevier Inc
Erscheinungsjahr
2015
Quelle
Access via ScienceDirect (Elsevier)
Beschreibungen/Notizen
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework.
•IceProd is a lightweight distributed workflow management framework.•Uses existing middleware and protocols.•Runs at user-level and is easily adaptable to other applications.•It has been successful in managing 450k cores across 25 computing centers.•Identified areas of improvement including scalability and load balancing.