Microsoft TerraServer: A Spatial Data Warehouse Tom Barclay Microsoft Research 301 Howard St., Suite 830 San Francisco, CA 94105 415 778 8223 [email protected] Jim Gray Microsoft Research 301 Howard St., Suite 830 San Francisco, CA 94105 415 778 8222 [email protected] Don Slutz Microsoft Research 301 Howard St., Suite 830 San Francisco, CA 94105 415 778 8226 [email protected] ABSTRACT Microsoft® TerraServer stores aerial, satellite, and topographic images of the earth in a SQL database available via the Internet. It is the world’s largest online atlas, combining eight terabytes of image data from the United States Geological Survey (USGS) and SPIN-2. Internet browsers provide intuitive spatial and text interfaces to the data. Users need no special hardware, software, or knowledge to locate and browse imagery. This paper describes how terabytes of “Internet unfriendly” geo-spatial images were scrubbed and edited into hundreds of millions of “Internet friendly” image tiles and loaded into a SQL data warehouse. All meta-data and imagery are stored in the SQL database. TerraServer demonstrates that general-purpose relational database technology can manage large scale image repositories, and shows that web browsers can be a good geo-spatial image presentation system. Keywords Geo-spatial, VLDB, image databases, internet. 1. Overview The TerraServer is the world's largest public repository of high-resolution aerial, satellite, and topographic data. It is designed to be accessed by thousands of simultaneous users using Internet protocols via standard web browsers. TerraServer is an image “tile” server that delivers a set of raster images based on a users search criteria. Once an image of interest is located, users can pan, zoom in, zoom out, or display meta-data about the image they are viewing. The TerraServer is a multi-media data warehouse. It differs from a traditional data warehouse in several ways: (1) it is accessed by millions of users, (2) the users extract relatively few records (thousands) in a particular session and, (3) the records are relatively large (10 kilobytes). By contrast, classic data warehouses are (1) accessed by a few hundred users via proprietary interfaces, (2) queries examine millions of records, to discover trends or anomalies, (3) the records themselves are generally less than a kilobyte. In addition, classic data warehouse queries may run for days before delivering results. Initial results typically cause users to modify and re-run queries to further refine results. One thing the TerraServer has in common with classic data warehouses is that both manage huge databases: several terabytes of data. TerraServer’s topographic maps cover all of the United States at 2 meter resolution 10 million square kilometers), the aerial photos cover 40% of the United States today (3 million square kilometers) at one-meter resolution, and 1% of the urban areas outside the United States (1 million square kilometers) at 2 meter resolution. This report describes the design of the TerraServer and its operation over the last 18 months. It also summarizes what we have learned from building and operating the TerraServer. Our research group explores scaleable servers. We wanted first-hand experience building and operating a large Internet server with a large database and heavy web traffic. To generate the traffic we needed to build an application that would be interesting to millions of web users. Based on our exposure to the EOS/DIS project [2], we settled on building a web site that serves aerial, satellite, and topographic imagery. We picked this application for four reasons: 1. The web is inherently a graphical environment, and these images of neighborhoods are recognizable and interesting throughout the world. 2. We believed this application would generate the billions of web hits needed to test our scalability ideas. 3. The data was available. The USGS was cooperative, an since the cold war had ended, other agencies were more able to share satellite image data. The thaw relaxed regulations that had previously limited the access to high-resolution imagery on a global basis. 4. The solution as we defined it – a wide-area, client/server imagery database application stored in a commercially available SQL database system – had not been attempted before. Indeed, many people felt it was impossible without using an object-oriented or object-relational system. This paper describes the application design, database design, hardware architecture, and operational experience of the TerraServer. The TerraServer has been operating for eighteen months now. We have deployed the third redesign of the database, user interface, and process of adding new images to the database. Permission to make digital or hard copies of part or all of this work or personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. MOD 2000, Dallas, TX USA © ACM 2000 1-58113-218-2/00/05 . . .$5.00 307Figure 1. A USGS DOQImage of 3Com Park nearSan Francisco Figure 2. A USGS DRG 2-meter resolution image. Figure 4: Encarta VirtualGlobe shaded relief imageof California, 8km / pixel. 2. Application Design TerraServer is accessed via the Internet through any graphical web browser. Users can zoom and pan across a mosaic of tiles within a TerraServer scene. The user interface is designed to function adequately over low-speed (28.8kbps) connections. Any modern PC, MAC, or UNIX workstation can access the TerraServer using a standard web browser – Internet Explorer 3.0 or later, or Netscape Navigator 3.0 or later. If you have never used it, look at the TerraServer web site at http://terraserver.microsoft.com/. There are four methods by which a user locates an image: (1) Coverage Map: clicking on low resolution map of the world shaded to show where coverage exists, (2) Place Query: entering a place name, e.g. San Francisco, (3) Coordinate Query: entering the longitude and latitude of interest, or (4) Famous Place: selecting a location from a pre-compiled list of places. A successful search presents the user with a web page containing an HTML table of image tiles fetched