The site speed report in Google Analytics got a nice upgrade recently with the introduction of the performance report and the ability to track virtual pageviews.
The performance report shows us how the page load times are distributed and you can see that the average page load time may not the best statistic to describe that distribution. The trouble is that you can have outliers with extremely high/non-sensical values:
and these will skew the overall average page load time with one single sample:
Far better in my opinion to use the median with page load times. Just add up the individual percentages until you get to 50%, and track that value over time. In the above example the median would be in 1.5 – 2 second bucket, which is not affected by the outlier.