The faster your site loads, the better, mainly because people absolutely HATE slow websites. If your page takes too long to load, then the chances are that your visitor will give up and go somewhere else.
You might even be frustrated with your own website’s speed, especially while you are updating WordPress from the dashboard.
Beyond that, Google recently announced that they consider website speed when determining search engine rankings.
Check out what the experts have to say about the topic:
Google found that moving from a 10-result page loading in 0.4 seconds to a 30-result page loading in 0.9 seconds decreased traffic and ad revenues by 20% (Linden 2006). When the home page of Google Maps was reduced from 100KB to 70-80KB, traffic went up 10% in the first week, and an additional 25% in the following three weeks (Farber 2006).
Tests at Amazon revealed similar results: every 100 ms increase in load time of Amazon.com decreased sales by 1%. (Kohavi and Longbotham 2007).