5. Improving Benchmarks

The usefulness of a benchmark is influenced by several factors and these factors may change over time. A big factor is the ability of the benchmark to resist "cracking" or a way of tuning the performace of a benchmark by the use of different compilers, preprocessors, or benchmark-specific flags. After a benchmark has become standardized, there is tremendous pressure to improve performance by targeted optimizations or by aggressive interpretations of the rules for running the benchmark.

5.1 Optimizations that Affect Benchmark Performance

To further illustrate why benchmarks may become obsolete, it would be helpful to know why the SPEC92 benchmarks were replaced with the SPEC95.

When the benchmarks were adopted for the SPEC92 suite, several flaws were present but these were minimal since code optimization was difficult at that time. However, after the program became a benchmark, compiler authors became resourceful in optimizing around the benchmark. The following is a list of some optimizations that are used:

• Many compilers perform loop unrolling. They duplicate the code of the loop body, generate larger basic blocks that can be more easily optimized by other compilation techniques. This is a common optimization that generally benefits programs with loops.

• Via some compilers, the conditions that are checked by the if statements are transformed to a logically equivalent form that can be compiled into more efficient code.

• Some compilers optimize the load instruction over several iterations of the loop. Instead of loading a 16-bit item at a time, the compiler generates load instructions for a 32-bit or 64-bit words, storing them in later iterations in the loop. However, this is a legitmate optimization if implemented properly. It benefits some programs more than others, such as programs that possess these data type properties.

5.2 Updating Benchmarks

Since technology is always improving, benchmarks that are used to measure the effect of changes in technology also need to be improved. There are several issues that have to be addressed.

1. Runtime. The length of time that a benchmark is running is important because if the running time interval is too short, small changes or fluctuations in the measurements will have a significant impact on the observed percentage improvements. Thus SPEC improved their benchmarks by making them longer to take account of future performance.

2. Application size. Applications are growing in complexity and size, thus benchmarks may become less representative of what was run on current systems if they are not adapted for larger programs. Also it is important to mix benchmarks requiring large resources along with smaller programs.

3. Application type. Just as the size and complexity of application increases, a wider range of types of applications are growing. Thus, these other types should be considered in order to cover different varieties and increased complexities of the workload.

4. Portability. It is also important that benchmarks and tools used in the process of benchmarking are independent of the operating system.

5. Pre-emptive multitasking. Most systems can handle many tasks, such as printing or switching between applications, at the same time by splitting processing time between the multiple tasks.

6. Increasing data sizes. There is an increase in the use of audio and video intensive apllications which require more efficient I/O capabilities for higher bandwith.

7. New system resources. There are new operating systems which make it easier for developers to integrate newer capabilities into their applications. These include better 3D graphics engines, telephony protocols, and data types like audio and video which often require more computing power.

8. Moving target. It is likely that improvements in the test performance become specific to that test only. By updating benchmarks, it may encourage general improvements and make test-specific optimizations less effective.