A Survey of FPGA BenchmarksRaphael Njuguna, [email protected] (A project report written under theguidance of Prof. Raj Jain)DownloadAbstractNew markets are emerging for the fast growing field-programmable gate array (FPGA) industry. Standard andfair benchmarking practices are necessary to evaluate FPGA systems and determine their potential to supporttarget applications. This paper provides an extensive survey of FPGA benchmarks in both academia andindustry.Keywords: FPGA, Benchmark, Performance, Evaluation, RAW, VPR, MCNC, IWLS, PREP, Toronto 20,LINPACK, DSP, BDTI, MATLAB, MediaBench, OpenFPGA, Smith-Waterman, BLAST, EEMBC,Dhrystone, MiBench, OpenCores.Table of Contents1. Introduction2. Historical Background3. Benchmarks for Traditional FPGA Systems 3.1. RAW Benchmark Suite 3.2. VPR Benchmark 3.3. MCNC Benchmark suite 3.4. IWLS 2005 Benchmarks 3.5. PREP Benchmark Suite 3.6. Toronto 20 Benchmark suite 3.7. LINPAC Benchmark4. Benchmarks for Hybrid-FPGA Systems 4.1. Benchmarks for FPGA-based Digital Signal Processing Systems 4.1.1. BDTI Communications Benchmark 4.1.2. MATLAB Benchmarks 4.1.3. MediaBench Benchmark Set 4.2. Benchmarks for FPGA-based Biological Systems 4.2.1. OpenFPGA.org 4.2.2. Smith-Waterman Algorithm 4.2.3. BLAST 4.3. Benchmarks for FPGA-based Embedded Systems 4.3.1. EEMBC Benchmarks 4.3.2. Dhrystone Benchmark 4.3.3. MiBench Benchmark Suite5. Sources of FPGA Benchmarks 5.1. Conference Benchmarks 5.2. Open Source Benchmarks 5.3. Synthetic BenchmarksA Survey of FPGA Benchmarks1 of 135.4. Industrial Benchmarks6. Summary7. References8. List of Acronyms1. IntroductionIn the recent years, field-programmable gate array (FPGA) systems have gained popularity in manyapplications such as digital signal processing, high performance computing, biological applications, just toname a few. FPGA, a reconfigurable digital logic device, facilitates rapid prototyping and design verificationthat enable designers to develop robust hardware and software solutions. A typical FPGA design flowinvolves: creating an electronic circuit design, placing and routing connectivity of the design onto FPGAarchitecture, verification and validation of the design, and configuration of the design into an FPGA device [WikipediaFPGA]. The FPGA community relies heavily on benchmarks to evaluate performance of their hardware and softwaresolutions. Therefore, standard and fair benchmarking practices are necessary to evaluate FPGA systems anddetermine their potential to support target applications. For instance, an end user may study benchmarkresults published by various FPGA vendors to select an FPGA device that is suitable for the intendedapplication. This survey will explore utilization of benchmarks to evaluate systems that contain FPGA devicesand their associated software design tool chains.2. Historical BackgroundThe art of benchmarking in FPGA industry is as old as the industry itself. Shortly after FPGA was born in1984, a benchmark suite consisting of ten combinational benchmark circuits was reported at the InternationalSymposium on Circuits and Systems (ISCAS′85) [ Hansen99]. Four years later ISCAS′89 benchmark suitecontributed sequential circuits into the FPGA community [ Brglez89]. The need for challenging and updatedbenchmarks led to the introduction of MCNC′91 benchmarks, which were published at the MCNCInternational Workshop on Logic Synthesis, 1991 [ Yang91]. A series of benchmark suites published forconferences and workshops soon followed; they included LGSynth91, HLSynth92, PDWorkshop93,Partitioning93, just to name a few. Microelectronics Center of North Carolina (MCNC), working underACM/SIGDA grant, maintained free electronic distribution of the aforementioned benchmarks [ Brglez93]. Over the years, conference benchmarks have flooded FPGA community because they are more readilyavailable than real industrial benchmarks. Nonetheless, a few non-profit organizations have afforded FPGAindustry with benchmarks that span over diverse applications. For instance, PREP′94 benchmark suite waspublished by a consortium of companies in the programmable logic industry to demonstrate performance andcapacity of programmable logic devices [ Kliman94]. On the other hand, EEMBC, a non-profit organizationformed in 1997 to develop benchmarks for embedded systems [ EEMBC08], is an invaluable resource forFPGA system designers implementing soft-core processors. Open-source organizations such as OpenCoresallow FPGA community to share real designs, which can be used as benchmarks.3. Benchmarks for Traditional FPGA SystemsFPGAs have traditionally been used in reconfigurable and parallel computing systems. Consequently, FPGAcommunity has developed numerous benchmarks to evaluate hardware and software solutions that implementA Survey of FPGA Benchmarks2 of 13these systems.3.1. RAW Benchmark SuiteRAW benchmark suite was published by MIT′s reconfigurable architecture workstation project forperformance evaluation of reconfigurable computing systems such as FPGA [ Babb97]. It implements diversealgorithms in general purpose computing that include CPU and parallel processing benchmarks. Performanceof FPGA is based on throughput and resources required to solve a particular benchmark problem. Benchmarkresults are reported using the following metrics: solution speed (kHz), speedup relative to reference software,and speedup per FPGA [ Babb97].3.2. VPR BenchmarkVersatile place and route (VPR) is a component-level benchmark program contained in SPEC CPU2000package. It was published by Standard Performance Evaluation Corporation (SPEC) to evaluate compute-intensive integer performance of FPGA during place-and-route design process [ SPEC]. VPR demonstratesspeed and throughput of performing place-and-route design task. SPEC adopted VPR program from aresearch project that created it as a tool for packing, placement, and routing designs in FPGA [ Betz97].Although VPR program is not included in the latest SPEC CPU2006 package, it is still popular in the FPGAcommunity.3.3. MCNC Benchmark suiteMicroelectronics Center of North Carolina (MCNC) benchmark suite was published for MCNC InternationalWorkshop on Logic Synthesis, 1991. It included logic synthesis and optimization benchmark sets fromISCAS’85 and ISCAS’89 in addition to some other benchmarks collected from industry and academia. Thebenchmark suite has standardized
View Full Document
Unlocking...