ANU Computer Science Technical Reports
TR-CS-06-01
Stephen M Blackburn, Robin Garner, Chris Hoffmann, Asjad M Khan, Kathryn S
McKinley, Rotem Bentzur, Amer Diwan, Daniel Feinberg, Daniel Frampton,
Samuel Z Guyer, Martin Hirzel, Antony Hosking, Maria Jump, Han Lee, J Eliot B
Moss, Aashish Phansalkar, Darko Stefanovic, Thomas VanDrunen, Daniel von
Dincklage, and Ben Wiedermann.
The DaCapo benchmarks: Java benchmarking development and analysis
(Extended Version).
September 2006.
[POSTSCRIPT (7909125 bytes)] [PDF (4351068 bytes)]
Abstract: Since benchmarks drive computer science
research and industry product development, which ones we use and how we
evaluate them are key questions for the community. Despite complex runtime
tradeoffs due to dynamic compilation and garbage collection required for Java
programs, many evaluations still use methodologies developed for C, C++, and
Fortran. SPEC, the dominant purveyor of benchmarks, compounded this problem
by institutionalizing these methodologies for their Java benchmark suite.
This paper recommends benchmarking selection and evaluation methodologies,
and introduces the DaCapo benchmarks, a set of open source, client-side Java
benchmarks. We demonstrate that the complex interactions of (1) architecture,
(2) compiler, (3) virtual machine, (4) memory management, and (5) application
require more extensive evaluation than C, C++, and Fortran which stress (4)
much less, and do not require (3). We use and introduce new value,
time-series, and statistical metrics for static and dynamic properties such
as code complexity, code size, heap composition, and pointer mutations. No
benchmark suite is definitive, but these metrics show that DaCapo improves
over SPEC Java in a variety of ways, including more complex code, richer
object behaviors, and more demanding memory system requirements. This paper
takes a step towards improving methodologies for choosing and evaluating
benchmarks to foster innovation in system design and implementation for Java
and other managed languages.
Technical Reports <Technical-DOT-Reports-AT-cs-DOT-anu.edu.au>
Last modified: Tue May 31 12:56:01 EST 2011