Information Markets vs. Opinion Pools:An Empirical ComparisonYiling ChenChao-Hsien ChuTracy MullenSchool of Information Sciences & TechnologyThe Pennsylvania State UniversityUniversity Park, PA 16802{ychen|chu|tmullen}@ist.psu.eduDavid M. PennockYahoo! Research Labs74 N. Pasadena Ave, 3rd FloorPasadena, CA [email protected] this paper, we examine the relative forecast accuracy ofinformation markets versus expert aggregation. We lever-age a unique data source of almost 2000 people’s subjectiveprobability judgments on 2003 US National Football Leaguegames and compare with the “market probabilities” givenby two different information markets on exactly the sameevents. We combine assessments of multiple experts via lin-ear and logarithmic aggregation functions to form pooledpredictions. Prices in information markets are used to de-rive market predictions. Our results show that, at the sametime point ahead of the game, information markets provideas accurate predictions as pooled expert assessments. Inscreening pooled expert predictions, we find that arithmeticaverage is a robust and efficient pooling function; weight-ing expert assessments according to their past performancedoes not improve accuracy of pooled predictions; and loga-rithmic aggregation functions offer bolder predictions thanlinear aggregation functions. The results provide insightsinto the predictive performance of information markets, andthe relative merits of selecting among various opinion pool-ing methods.Categories and Subject DescriptorsJ.4 [Computer Applications]: Social and Behavioral Sci-ences—economicsGeneral TermsEconomics, PerformanceKeywordsInformation markets, opinion pools, expert opinions, predic-tion accuracy, forecastingPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.EC’05, June 5–8, 2005, Vancouver, British Columbia, Canada.Copyright 2005 ACM 1-58113-711-0/04/0005 ...$5.00.1. INTRODUCTIONForecasting is a ubiquitous endeavor in human societies.For decades, scientists have been developing and exploringvarious forecasting methods, which can be roughly dividedinto statistical and non-statistical approaches. Statisticalapproaches require not only the existence of enough histor-ical data but also that past data contains valuable infor-mation about the future event. When these conditions cannot be met, non-statistical approaches that rely on judg-mental information about the future event could be betterchoices. One widely used non-statistical method is to elicitopinions from experts. Since experts are not generally inagreement, many belief aggregation methods have been pro-posed to combine expert opinions together and form a sin-gle prediction. These belief aggregation methods are calledopinion pools, which have been extensively studied in statis-tics [18, 22, 36], and management sciences [8, 9, 28, 29], andapplied in many domains such as group decision making [27]and risk analysis [11].With the fast growth of the Internet, information marketshave recently emerged as a promising non-statistical fore-casting tool. Information markets (sometimes called pre-diction markets, idea markets, or event markets) are mar-kets designed for aggregating information and making pre-dictions about future events. To form the predictions, in-formation markets tie payoffs of securities to outcomes ofevents. For example, in an information market to predictthe result of a US professional National Football League(NFL) game, say New England vs Carolina, the securitypays a certain amount of money per share to its holders ifand only if New England wins the game. Otherwise, it paysoff nothing. The security price before the game reflects theconsensus expectation of market traders about the proba-bility of New England winning the game. Such marketsare becoming very popular. The Iowa Electronic Markets(IEM) [2] are real-money futures markets to predict eco-nomic and political events such as elections. The Holly-wood Stock Exchange (HSX) [3] is a virtual (play-money)exchange for trading securities to forecast future box of-fice proceeds of new movies, the outcomes of entertainmentawards, etc. TradeSports.com [7], a real-money betting ex-change registered in Ireland, hosts markets for sports, po-litical, entertainment, and financial events. The ForesightExchange (FX) [4] allows traders to wager play money onunresolved scientific questions or other claims of public in-terest, and NewsFutures.com’s World News Exchange [1] haspopular sports and financial betting markets, also groundedin a play-money currency.Despite the popularity of information markets, one of themost important questions to ask is: how accurately can in-formation markets predict? Previous research in generalshows that information markets are remarkably accurate.The political election markets at IEM predict the electionoutcomes better than polls [14, 15, 16, 17]. Prices in HSXand FX have been found to give as accurate or more ac-curate predictions than judgment of individual experts [31,32, 35]. However, information markets have not been cal-ibrated against opinion pools, except for Servan-Schreiberet. al [34], in which the authors compare two informationmarkets against arithmetic average of expert opinions. Sinceinformation markets, in nature, offer an adaptive and self-organized mechanism to aggregate opinions of market par-ticipants, it is interesting to compare them with existingopinion pooling methods, to evaluate the performance ofinformation markets from another perspective. The com-parison will provide beneficial guidance for practitioners tochoose the most appropriate method for their needs.This paper contributes to the literature in two ways: (1)As an initial attempt to compare information markets withopinion pools of multiple experts, it leads to a better un-derstanding of information markets and their promise as analternative institution for obtaining accurate forecasts (2)In screening opinion pools to be used in the comparison, wecast insights into relative performances of different opinionpools. In terms of prediction accuracy, we compare two in-formation markets