4. Measuring and Reporting Performance of Benchmarks

Benchmark programs should be derived from how actual applications will execute. However, performance is often the result of combined characteristics of a given computer architecture and system software/hardware components in addition to the microprocessor. Other factors such as the operating system, compilers, libraries, memory design and I/O subsystem characteristics may also have impacts on the results and make comparisons difficult.

4.1 Measuring Performance

Two ways to measure the performance are:

1. The speed measure - which measures how fast a computer completes a single task. For example, the SPECint95 is used for comparing the ability of a computer to complete single tasks.

2. The throughput measure - which measures how many tasks a computer can complete in a certain amout of time. The SPECint_rate95 measures the rate of a machine carrying out a number of tasks.

4.2 Interpreting Results

There are three important guidelines to remember when interpreting benchmark results:

1. Be aware of what is being measured. When making critical purchasing decisions based on results from standard benchmarks, it is very important to know what is actually been measured. Without knowing, it is difficult to know whether the measurements obtained is even relevant to the applications which will run on the system being purchased. Questions to consider are: does the benchmark measure the overall performance of the system or just components of the system such as the CPU or memory?

2. Representativeness is key. How close is the benchmark to the actual application being executed? The closer it is, the better it will be at predicting the performance. For example, a component-level benchmark would not be good predictors of performance for an application that would use the entire system. Likewise, application benchmarks would be the most accurate predictors of performance for individual applications.

3. Avoid single-measure metrics. Application performance should not be measured with just a single number. No single numerical measurement can completely describe the performance of a complex device like the CPU or the entire system. Also, try to avoid benchmarks that average several results into a single measurement. Important information may be lost in average values. Try to evaluate all the results from different benchmarks that are relevant to the application. This may give a more accurate picture than evaluating the results from one benchmark alone.

4.3 Reporting Performance

There are some points to remember when reporting results obtained from running benchmarks.

• Use newer version over the older. If an updated and revised version of a benchmark suite is available, it is usually preferred over the outdated one. Generally there are good reasons for revising the original. They include, but not limited to, changes in technology, improvements in compiler efficiency, etc.

• Use all programs in a suite. There may be legitimate reasons why only a subset was used, but they should be explained. Otherwise, someone looking at the results may become suspicious as to why the other programs were not considered. Explain about the selection process, why it was not arbitrary, and why it was useful to do so.

• Report compilation mode. The compilation mode that was used is important and should be reported in every case. The effect of a certain new hardware feature may be dependent on whether it is applied to optimzed or unoptimized programs.

• Use a variety of benchmarks when reporting performance. Generally it is a good idea to use other set of programs as additional test cases. One set of benchmarks may behave differently than another set and such observations may be useful as to the next round of benchmark selections.

• List all factors affecting performance. Have enough information about performance measurements to allow readers to duplicate the results. These include:
  1. program input
  2. version of the program
  3. version of compiler
  4. optimizing level of compiled code
  5. version of operating system
  6. amount of main memory
  7. number and types of disks
  8. version of the CPU