Statistically correct software performance tuning

How to tune software performance in a statistically correct way?

how to build the hypotheses
how to construct the experiment
how to evaluate the measurements and decide
how to make sure the results are significant