As I suspected, Bruce Clay confirmed that he believes page speed is in fact a ranking factor within the Google algorithm.   In an interview with WebProNews he discussed a site they worked on recently which went from position 5 to position 3 within the Google SERPS.   He says that the improvement in ranking correlates with page speed improvements only.

You can see the interview with WebProNews here:  

WebProNews gets in-depth with Bruce Clay

You can read my SEOMoz post about page speed and Google ranking here:

Optimizing Page Speed For Ranking Using Google Tools