p. 137

3 Evaluation of Information Processing System

In order to operate the information processing system, you must evaluate various functions of the system beforehand. However, the information processing system has various modes, and the usage purpose is also different. Therefore, it is difficult to evaluate on the basis of only one index or indicator. Therefore, there is a need for indexes or indicators that evaluate the information processing system from various viewpoints.

3 - 1 Evaluation of the Processing Power

The indexes (or indicators) for evaluating the processing power of the information processing system include the index (or indicators) for evaluating the processing power of the overall system and the index (or indicators) for evaluating the processing power of the CPU that is the core of the system

3-1-1 Evaluation of the Overall System

(1) Performance indicator of the processing time

The performance indicators of the processing time of the system include the turnaround time and the response time. Basically, the shorter these times, the higher the evaluation.

• • Turnaround time
  This is the time that elapses from the start of a processing request to the system until the acquisition of the complete output result. It is used as the indicator for performance evaluation of a batch processing system. · 1 question

• • Response time
  This is the time that elapses from the completion of a processing request to the system until the start of the result output. It is used as the indicator for performance evaluation of an online system.

p. 138

(2) Performance indicator of the amount of work

Throughput is the performance indicator of the amount of work in a system. Basically, the larger the amount of work that can be processed, the higher the evaluation. · 1 question

• • Throughput
  This is the amount of work that is processed within a unit time (fixed time) in the information processing system. TPS (Transactions Per Second) (i.e., number of transactions that can be processed in one second) can be used as the unit. For example, when the upper limit of the CPU utilization is 80% in a web server in which 5 milliseconds of CPU time is taken for processing one transaction, the throughput of the web server is calculated as follows:

However, this is the throughput of an individual web server, and the throughput of the entire information processing system in which several devices (i.e., servers) are combined together is the throughput of the device (i.e., server) that has the lowest processing power.

In addition to evaluation of the operating information processing system, the turnaround time/ response time and the throughput are used as target values (i.e., indicators) during development or enhancement of the information processing system. First, the necessary target values (i.e., system performance) are set. Next, the various performance requirements and system configuration (i.e., enhancement plan) are designed in order to implement the target values. This is called capacity planning. · 1 question

p. 139

[Basic procedure of capacity planning]

• 1) Collecting the workload
  Collect information, such as the type and amount of processing and the processing time that are required for the system. (The load information during peak time is particularly important.) · 2 questions
• 2) Determination of the performance requirements of the system
  Analyze the collected information, and determine the performance requirements of the system on the basis of the load forecast and creation of the prototype. · 1 question
• 3) Sizing
  Estimate the performance requirements of the hardware, software, and peripheral devices necessary for fulfilling the performance requirements of the system. · 1 question
• 4) Evaluation/improvement
  Evaluate the results of sizing, and then make improvements (i.e., tuning) if necessary. Even after the system starts operating, repeatedly perform evaluation and improvements in order to ensure that the storage capacity and the system throughput are not insufficient. · 1 question

3-1-2 Evaluation of the CPU

(1) Clock frequency

Clock frequency represents the speed at which a clock signal is generated. Since the operation of the CPU is performed in synchronization with this signal, basically, the larger the clock frequency, the higher the processing power becomes. However, since the CPI (Cycles Per Instruction), which is the number of clocks necessary for executing one instruction, may differ depending on the CPU, the performance of the CPU cannot be compared (evaluated) only through the clock frequency.

(2) MIPS (Million Instructions Per Second）

MIPS represents the number of instructions that can be executed in 1 second in units of one-million instructions. A CPU with 1 MIPS is a CPU that can execute one million instructions in 1 second. Basically, the larger the number of instructions that can be executed, the higher the evaluation becomes. However, there is no sense in using this indicator as it is for comparing CPUs of different models in which the content of the instructions to be executed is also different. MIPS is determined on the basis of the execution time of one instruction (or the cycle time

p. 140

from the receipt of an instruction until the next instruction can be received.) However, since the execution time of an instruction differs depending on the type, a method of calculating the weighted average on the basis of the occurrence rate of an instruction is used to calculate the average instruction execution time. At this time, on the basis of the existence of instructions that are used often and instructions that are not used very often, an instruction mix is used as the occurrence rate of standard instructions depending on the processing of applications. Typical instruction mixes include the commercial mix, which focuses on a transfer instruction used frequently in business processing calculations and the Gibson mix, which focuses on an operation instruction used frequently in scientific and technological calculations. Moreover, the instruction execution time for executing one instruction can also be calculated from the CPI and clock frequency.

Moreover, as a further interpretation of the unit of MIPS, it is understood that a CPU with 1 MIPS can execute one instruction in 1 microsecond. Therefore, the calculation of MIPS can be summarized as shown below.

p. 141

Nowadays, GIPS (Giga Instructions Per Second), which represents the number of instructions that can be executed in 1 second in units of giga (10⁹) instructions, may also be used.

(3) FLOPS (FLoating-point Operations Per Second)

FLOPS is a unit that represents the number of floating-point instructions that can be executed in 1 second. A CPU with 1 FLOPS is a CPU that can execute one floating-point operation in 1 second. Currently, TFLOPS (Tera FLOPS), which represents the number of floating-point instructions that can be executed in 1 second in units of tera (10¹²) instructions, is used as the performance evaluation indicator of a supercomputer and such other high-performance computer.

3-1-3 Techniques of Performance Measurement

(1) Benchmark test

A benchmark test is a method of measuring the processing performance (e.g., the time

p. 142

required) by executing a standard program (i.e., software) for an evaluation in line with the usage purpose. It is called benchmarking to perform measurement (or the measurement standard), and the program that is used is called a benchmark program. Since the measurement target of each benchmark test is different, it is necessary to understand the characteristics of the system in an integrated manner by executing several benchmark tests.

• • TPC benchmark
  This is a benchmark defined by the TPC (Transaction Processing Performance Council) that promotes standardization of an OLTP system for its evaluation, and TPC-A through TPC-D are defined depending on the applications. System cost per TPS or unit performance (cost/TPS) is used as the performance indicator. In the currently used TPC-C (for business processing OLTP), a unit set with reference to 1 minute (tpmC, Price/tpmC) is used.
• • SPEC benchmark
  This is a benchmark defined by the industrial standard body, SPEC (System Performance Evaluation Council), in order to evaluate the performance of the CPU. This benchmark test can be performed on UNIX.
  • • SPECint: A benchmark focusing on integer type operations
    
  • • SPECfp: A benchmark focusing on floating-point type operations
    
• • Other benchmarks
  
  • • Dhrystone: For evaluation of fixed-point number operations
    
  • • Whetstone: For evaluation of floating-point number operations
    
  • • Livermore Fortran Kernel: For evaluation of a vector computer/compiler
    
  • • Linpack: For evaluation of floating-point number operations (simultaneous linear
    equation solution program)

(2) Kernel program test

Kernel program test is a method of executing a simple program such as a basic computing problem, measuring the execution time, and comparing the CPU performance.

(3) Monitoring

Monitoring is a method of incorporating beforehand a mechanism for measurement in the information processing system that is the target of measurement and measuring the actual operation status.

• • Software monitoring: A method of incorporating a program (i.e., software) for
  measurement

p. 143

• • Hardware monitoring: A method of incorporating a device (i.e., hardware) for
  measurement