The 007 Benchmark*Michael J. CareyDavid J. DeWitt JeffreyComputer Sciences DepartmentUniv&sity of Wisconsin-MadisonAbstractThe 007 Benchmark represents a comprehensive test of00DBMS performance. In this paper we describe thebenchmark and present performance results from its im-plementation in three 00DBMS systems. It is our hopethat the 007 Benchmark will provide useful insight forend-users evaluating the performance of 00DBMS sys-tems; we also hope that the research community will findthat 007 provides a database schema, instance, andworkload that is useful for evaluating new techniquesand algorithms for OODBMS implementation.1 IntroductionBuilders of object-oriented database management sys-tems are faced with a wide range of design and imple-ment ation decisions, and many of these decisions havea profound effect on the performance of the resultingsystem. Recently, a number of OODBMS systems havebecome publically available, and the developers of thesesystems have made very different choices for fundamen-t al aspects of the systems. However, perhaps since thetechnology is so new, it is not yet clear precisely howthese systems differ in their performance characteristics;in fact, it is not even clear what performance metricsshould be used to give a useful profile of an OODBMS’Sperformance. We have designed the 007 Benchmarkas a first step toward providing such a comprehensive00DBMS performance profile.Among the performance characteristics tested by 007are:*DEC provided the funding that began this research. Thebulk of this work was funded by DARPA under contract num-berDAAB07-92-C-Q508 and monitored by the US Army ResearchLaboratory. Sun donated the hardware used as the server in theexperiments.Permission to copy without fee all or part of this material isgranted provided that the copies are not mede or distributed fordirect commercial advantage, the ACM copyright notice and thetitle of the publication and ite date appear, and notice is giventhat copying is by permission of the Association for ComputingMeohinery. To copy otherwise, or to republish, requires a feeand/or specific permission.SIGMOD 151931Washington, DC, USA@1993 ACM 0.89791.592.5/93/0005/001 2.,$1.50F. Naughton●●●The speed of many different kinds of pointer traver-sals, including traversals over cached data, traversalsover disk-resident data, sparse traversals, and densetraversals;The efficiency of many different kinds of updates,including updates to indexed and unindexed objectfields, repeated updates, sparse updates, updates ofcached data, and the creation and deletion of objects;The performance of the query processor (or, in caseswhere the query language was not sufficiently ex-pressive, the query programmer) on several differenttypes of queries.By design, the 007 Benchmark produces a set of num-bers rather than a single number. A single numberbenchmark has the advantage that it is very catchy andeasy to use (and abuse) for system comparisons. How-ever, a benchmark that returns a set of numbers gives agreat deal more information about a system than doesone that returns a single number.A single numberbenchmark is only truly useful if the benchmark itselfprecisely mirrors the application for which the systemwill be used.In this paper, we describe the benchmark and givepreliminary performance results from its implementa-tion in one public-domain research system (E/Exodus)and two commercially available OODB systems (Objec-tivity/DB, which is also available as DEC Object/DBVI .0, and Ontos). Due to tight space constraints, thedescriptions here are necessarily sketchy, and not all ofthe results can be presented. A more detailed bench-mark description, together with a full and final set ofperformance results for all of the participating systemslcan be found in [GDN93]. Lastly, it should be mentionedthat we had also expected to include results for anothercommercial system, the ObjectStore system from Ob-ject Design, Inc. Unfortunately, on the day before thecamera-ready deadline for this proceedings, ODI hadtheir lawyers send us a notice saying that they were dis-satisfied with the way that we had run the benchmark-ing process and that we had to drop our ObjectStore1We are currently finishing up the benchmark on anothercomercial system (02), and its performance will be includedin [CDN93]. We also invited Versant to participate in the bench-mork, but they declined to participate until the next release oftheir system was available.12results from the paper or else face possible legal action.It is unfortunate that they chose to withdraw, as ODI’Sapproach to persistence provided some interesting con-trasts with the other systems.The remainder of the paper is organized as follows.Section 2 compares the 007 Benchmark to previous ef-forts in 00DBMS benchmarking. Section 3 describesthe structure of the 007 Benchmark database. Section 4describes the hardware testbed configuration we used torun the benchmark, and gives a brief overview of thesystems tested. Section 5 describes the benchmark’s op-erations and discusses the experimental results for eachoperation as it is presented. Finally, Section 6 containssome conclusions and our plans for future work.2 Related WorkIn this section, we briefly discuss the previous bench-marking efforts that are related to 007, and we citethe reasons why we felt that there was a need for ad-ditional work in the 00DBMS benchmarking area. Amuch more in-depth treatment of related work can befound in [CDN93].2.1 Previous 00DBMS BenchmarksThe 001 Benchmark2 [CS92], commonly referred to asthe Sun Benchmark, was the first widely accepted bench-mark that attempted to predict DBMS performance forengineering design applications. Because of its early vis-ibility and its simplicity, 001 has became a de factostandard for 00DB benchmarking.Another benchmark that 007 is closely relatedto is the HyperModel Benchmark developed at Tek-tronix [And90]. Compared to 001, Hypermodel in-cludes both a richer schema (involving several differ-ent relationships and covering a larger set of basic datatypes) and a larger collection of benchmark operations(including a wider variety of lookup, traversal, and up-date operations).There are several other 00DB studies related to ourwork on 007. Ontologic used the initial Sun Bench-mark to study the performance of Vbase, their first00DB product offering [DD88]. Researchers at Al-tair designed a complex object benchmark (ACOB)for use in studying alternative client/server