As I suspected, Bruce Clay confirmed that he believes page speed is in fact a ranking factor within the Google algorithm. In an interview with WebProNews he discussed a site they worked on recently which went from position 5 to position 3 within the Google SERPS. He says that the improvement in ranking correlates with page speed improvements only.
You can see the interview with WebProNews here:
WebProNews gets in-depth with Bruce Clay
You can read my SEOMoz post about page speed and Google ranking here: