Parallel Programmer Productivity: A Case Study of Novice Parallel Programmers Lorin Hochstein1, Jeff Carver3, Forrest Shull2, Sima Asgari1, Victor Basili1,2, Jeffrey K. Hollingsworth1, Marvin V. Zelkowitz1,2 1University of Maryland, College Park 2Fraunhofer Center Maryland 3Mississippi State University {sima,hollings,lorin}@cs.umd.edu {basili,fshull,mvz}@fc-md.umd.edu [email protected] Abstract In developing High-Performance Computing (HPC) software, time to solution is an important metric. This metric is com-prised of two main components: The human effort required developing the software, plus the amount of machine time re-quired to execute it. To date, little empirical work has been done to study the first component: the human effort required and the effects of approaches and practices that may be used to reduce it. In this paper, we describe a series of studies that address this problem. We instrumented the development proc-ess used in multiple HPC classroom environments. We ana-lyzed data within and across such studies, varying factors such as the parallel programming model used and the applica-tion being developed, to understand their impact on the devel-opment process. 1 Introduction Historically, the major metric of success of computation in the HPC community has been speed of program execution. Recently the community has recognized the importance of the time required to develop programs as well as run them. How-ever, since HPC applications must, by definition, be high per-formance, it is critical to study programmer productivity and application performance together. The goal of this work is to better understand and quantify software development for high performance computers, to augment the existing work on im-proving performance time, and to take a more complete view of the total time to solution. Currently there is little empirical evidence to support or refute the utility of specific program-ming models, languages, and practices within the HPC com-munity. While there is a rich body of literature in the Software Engineering community about programmer productivity, much of it was developed with assumptions that do not neces-sarily hold in the HPC community. For example, while the SE community expects software specifications to evolve over the lifetime of the system, this evolution is expected to be in re-sponse to external factors such as customer-specified requests for new or changed functionality. However, in scientific com-putation, it is often insights culled from results of one program version that drive the needs for the next. This is because the software itself is helping to push the frontiers of understanding rather than automate well-understood tasks. Due to these unique requirements, traditional software engineering ap-proaches for improving productivity may be adapted for the HPC community, but are not appropriate without changes. With these challenges in mind, we have been developing and debugging a set of tools and protocols to study program-mer productivity in the HPC community. In this paper, we present both the methodology we have developed to investi-gate programmer productivity issues in the HPC domain, and some initial results of studying productivity of novice HPC programmers. The interest in the effectiveness of novice developers is justified by the nature of the traditional HPC context. Many of the applications in which high performance computers are used are quite complex and understood only by domain ex-perts. Those domain experts will often be novice HPC pro-grammers, at least when they are beginning their careers. Use-ful, evidence-based heuristics about how novice HPC devel-opers can best be brought up to speed hold out the promise of being able to address a significant obstacle to the goal of mak-ing correct HPC solutions feasible for more problems. As the first phase of this work, we are beginning with an observational approach, observing HPC code development practices and describing the results in terms of overall effort, performance, and cost. There is little previous empirical work on this topic, so this work will be used to generate a series of well-grounded hypotheses that can then be tested explicitly in later studies. 2 Related Work Two main components make up the time to solution met-ric. The first component is the human effort/calendar time required to develop and tune the software. The second compo-nent is the amount of machine time required to execute the software to produce the desired result. Metrics and even predictive models have already been developed for measuring the code performance part of that equation, under various constraints (e.g. [9, 11]). However, Permission to make digital or hard copies of all or part of this work for per-sonal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or repub-lish, to post on servers or to redistribute to lists, requires prior specific permis-sion and/or a fee. SC|05 November 12-18, 2005, Seattle, Washington, USA (c) 2005 ACM 1-59593-061-2/05/0011…$5.002little empirical work has been done to date to study the human effort required to implement those solutions. Only a handful of empirical studies have been run to examine factors influencing variables such as development time [2] or the difficulties en-countered during HPC development [12]. Some authors have commented that "little work has been done to evaluate HPC systems' usability, or to develop criteria for such evaluations" [12]. As a result, many of the practical decisions about devel-opment language and approach are currently made based on anecdote, “rules of thumb,” or personal preference. Several prior studies[4, 6] have used lines of code as the principal metric to determine effort required to compare paral-lel programming models with the assumption that fewer lines of code implies less effort. While this metric might have merit, we feel it is important to quantify the effort required to pro-duce different lines of code. Even if it turns out that the tech-nologies that require more code always require more effort, the ratio of lines of code doesn't necessarily equal the ratio of efforts, so you can't use LOCs to do your tradeoff analysis. Previous work done to study more general software de-velopment processes will provide a starting point
View Full Document
Unlocking...