Details

Autor(en) / Beteiligte

• Aartsen, M.G.
• Abbasi, R.
• Adams, J.
• Aguilar, J.A.
• Arguelles, C.
• Baker, M.
• Barwick, S.W.
• Becker Tjus, J.
• Bernhard, A.
• Boersma, D.J.
• Bose, D.
• Botner, O.
• Bretz, H.-P.
• Brown, A.M.
• Bruijn, R.
• Casey, J.
• Christov, A.
• Christy, B.
• Clevermann, F.
• Cowen, D.F.
• Davis, J.C.
• de Vries, K.D.
• Díaz-Vélez, J.C.
• Eberhardt, B.
• Eichmann, B.
• Eisch, J.
• Finley, C.
• Flis, S.
• Gaisser, T.K.
• Glüsenkamp, T.
• Goldschmidt, A.
• Golup, G.
• Góra, D.
• Groß, A.
• Hebecker, D.
• Helbing, K.
• Hoffmann, R.
• Huelsnitz, W.
• Hultqvist, K.
• Hussain, S.
• Jagielski, K.
• Kaminsky, B.
• Kappes, A.
• Kauer, M.
• Klein, S.R.
• Köhne, J.-H.
• Kopper, S.
• Krasberg, M.
• Kriesten, A.
• Kurahashi, N.
• Lünemann, J.
• Matis, H.S.
• Meures, T.
• Montaruli, T.
• Morse, R.
• Nahnhauer, R.
• Naumann, U.
• Olivas, A.
• Omairat, A.
• Pérez de los Heros, C.
• Pfendner, C.
• Rae, I.
• Rameez, M.
• Rawlins, K.
• Redl, P.
• Resconi, E.
• Ribordy, M.
• Riedel, B.
• Rott, C.
• Ryckbosch, D.
• Santander, M.
• Sarkar, S.
• Schmitz, M.
• Schukraft, A.
• Schulte, L.
• Sheremata, C.
• Spiczak, G.M.
• Stamatikos, M.
• Stanisha, N.A.
• Stößl, A.
• Taavola, H.
• Taboada, I.
• Tepe, A.
• Tešić, G.
• Tobin, M.N.
• Toscano, S.
• Unger, E.
• van Eijndhoven, N.
• van Santen, J.
• Vehring, M.
• Voge, M.
• Waldenmaier, T.
• Wendt, C.
• Wiebe, K.
• Williams, D.R.
• Wissing, H.
• Wood, T.R.
• Xu, D.L.
• Ziemann, J.
• Zoll, M.

Titel

The IceProd framework: Distributed data processing for the IceCube neutrino observatory

Ist Teil von

• Journal of parallel and distributed computing, 2015-01, Vol.75 (C), p.198-211

Ort / Verlag

United States: Elsevier Inc

Erscheinungsjahr

2015

Quelle

Alma/SFX Local Collection

Beschreibungen/Notizen

• IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of computing resources, including grids and batch systems such as CREAM, HTCondor, and PBS. This is accomplished by a set of dedicated daemons that process job submission in a coordinated fashion through the use of middleware plugins that serve to abstract the details of job submission and job management from the framework. •IceProd is a lightweight distributed workflow management framework.•Uses existing middleware and protocols.•Runs at user-level and is easily adaptable to other applications.•It has been successful in managing 450k cores across 25 computing centers.•Identified areas of improvement including scalability and load balancing.