Chapter 3APPLICATION REQUIREMENTS FORRESOURCE BROKERING IN A GRIDENVIRONMENTMichael Russell, Gabrielle Allen, Tom Goodale, Jarek Nabrzyski, and EdSeidelMax Planck Institute for Gravitational PhysicsPoznan Supercomputing and N et working CenterAbstract We discuss the problem of resource brokering in a Grid environment from theperspective of general application needs. Starting from a illustrative scenario,these requirements are broken down into the general areas of computation, data,network, security and accounting. Immediate needs for applications to profitablystart using the G rid are discussed, along with case studies for applications fromastrophysics and particle physics.1. INTRODUCTIO NIn this chapter we identify some of the key application resource brokeringrequirements that Grids must be able to satisfy, based on our experiences inworking with teams of distributed researchers studying complex physics phe-nomena. In our research we are confronted daily with the need for a globalcomputing infrastructure and with our own dreams of building more power-ful applications. To better illustrate our point, we begin the next section witha scenario that details one of the many ways we envision using Grids in thefuture. In S ection 3, we detail the general requirements highlighted by thisscenario and others. Section 4 discusses some of the key challenge problemsthat remain to be solved before such a scenario can be realized. Section 5 de-scribes some of our own work in developing Grid solutions for the astrophysicscommunity and a second use case illustrating some of the general requirementsof high-energy physics applications. Finally, we conclude our chapter with abrief look at important issues related to the resource brokering requirements.26 GRID RESOURCE MANAGEMENT2. MOTIVATING SCENARIONow consider the following scenario, illustrated in Figure 3.1, motivatedby the very real and pressing needs of the gravitational physics communityspurred by the imminent arrival of data from a world wide network of laserinterferometric gravitational wave detectors [Gib02].The gravitational wave detector network collects a TByte of data each daythat must be searched using different algorithms for possible events such asblack hole and neutron star collisions, pulsar signals or other astronomicalphenomena. Routine real-time analysis of gravitational wave data from thedetector identifies a burst event, but this standard analysis reveals no informa-tion about the burst source or location. To obtain this location, desperatelyrequired by astrophysicists for turning their telescopes to view the event beforeits visual signal fades, a large series of pre-computed possible gravitationalwave signals, called templates, must be cross-correlated against the detectordata.Gwen, an Italian astrophysicist monitoring the real-time analysis, accessesthe detector’s Portal, where the performance tool reports that 3 TFlops/s will beneeded to analyze the 100GB of raw data within an hour and that the data willneed to be transfered to the machines before analysis can begin. Local Italianresources are insufficient, so using the brokering tool, she locates the fastestavailable m achines around the world that have networks to the gravitationalwave data repository that are fast and stable enough for the transfer of the nec-essary data in a matter of minutes. She selects five machines that together areable to perform the data analysis in the required hour, and with scheduling anddata management tools, data is moved, executables created, and the analysisbegins.The analysis proceeds smoothly until it is determined that a custom templateis required that must be created from a full scale numerical black hole simula-tion. The portal, aware that Gwen is still online and working, informs her ofthis development by instant message. Gwen is immediately able to use the bro-kering tool to search for large resources capable of running such a simulation.No single machine is available, but two machines are located that together forma large virtual machine connected by sufficient bandwidth and as a unit theywill be able to complete the distributed simulation in an additional hour. Gwenthen uses the application tools on the portal to assemble the correct executablefor this run and stages it to run immediately across the two machines. Beforeleaving for lunch she uses the broker to extend the reservations on the originalresources required for the data analysis to allow for this set back.An hour later, the simulation finishes and the needed template is automati-cally delivered to the template repository. The original data analysis can nowcontinue.Application Requirements for Resource Brokering in a G rid Environment 27Astrophysicist usesGEO600 Portal from herworkstation to analysisdata from HannoverGravitational WaveDetectorSelectappropriatecomputeresourcesStage detectordata andexecutablesNotify whenhumaninterventionrequiredSimulationsstarted on largesupercomputersContract violationprompts migrationto new resourceOutput templatesassembled anddeliveredTemplates provideaccurate wave fromgravitational wavedetector data used topositionobservationaltelescopesGEO600 PortalGEO600 PortalCommunity notifiedFigure 3.1. A Grid scenario showing the potential use of resource brokering in a gravitationalwave data analysis.28 GRID RESOURCE MANAGEMENTIn an cafe twenty minutes later, an urgent text message on her mobile phonefrom the Portal’s notification tool informs her that one machine is now over-loaded, breaking its runtime contract. The Portal confirmation to make useof a different, more expensive, machine in order to fulfill the contract. Shejudges that time is of the essence, connects with her PDA to the Portal, andinstructs the migration tool to move this part of the analysis to the indicatedmachine. Within the specified hour, a second text message tells her and hercollaborators that the analysis is finished, and provides the location of the re-sulting data, which is available for all to view. Using this data, observatoriesare now correctly instructed to position their telescopes, and astrophysicists areable to find and view an exceptionally strong gamma-ray burst, characteristicof a supernovae.This scenario illustrates how, given the appropriate resource brokering mech-anisms, a researcher need not be concerned with where and how computingresources