As Amazon and Google have proven, the longer it takes for your website to load (even by 100 milliseconds) the less that people will want to use it. And so, as Jeff points out, it makes sense that the faster your website loads the more people will use it.
Related reading: Brent’s plea for baked weblogs.