Notes on Benchmarking

When writing your report for the assignment/project, we expect your group to properly benchmark your code, so as to reliably demonstrate that you have managed to speed up the execution of the template.

Necessary, but not Necessarily Sufficient, Conditions

Keep the following in mind when trying to benchmark your programs:

• Repeat your measurements. A good rule of thumb is to repeat the experiments at least ten times.
• Avoid statistical noise by ensuring that your tests are not computationally trivial, e.g. a single run takes at least a second to compute.
• Compute and comment on variance, standard deviation, mean, etc.
• Repeat your experiments with different problem sizes.
• Repeat your experiments with different amounts of parallelism.
• If you make any changes to the sequential template, that require domain knowledge, implement these improvements in the sequential template as well, to prevent skewing your results.
• Estimate the number of floating point/integer operations per second and relate this to the information you have of the hardware and the observed behavior.
• Remember to explicitly state how you measure time: Are you just tracking the execution of the entire process or are you using tool-specific means?

Further Technical Comments

For details on how to benchmark MPI code, take a look at Thomas Koopman's comments on the topic.

Prevent I/O from influencing the performance of your program (for better or for worse). For programs that use the standard output stream to write out their results, you can redirect the output to the pseudo-file /dev/null. For programs that write out files (e.g. the Game of Life template) a dry run flag (-n) should be used to prevent the programs from creating files.