Today representatives of Google Search and Microsoft’s Bing teams, Jake Brutlag and Eric Schurman respectively, presented the results of user performance tests at today’s Velocity Conference. The talk was entitled The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search. These are long-term tests were designed to see what aspects of performance are most important. To know how to improve their sites both Bing and Google need to know what tweaks to page load perceptions and realities help or hurt the user experience. This is one of the first performance tests that has actual data (and is not strictly anecdotal). The numbers may seem small, but they if you are dealing in millions/billions they add up quickly.
Here are Brutlag’s and Schurman’s final points:
- “Speed matters” is not just lip service
- Delays under half a second impact business metrics
- The cost of delay increases over time and persists
- Use progressive rendering
- Number of bytes in response is less important than what they are and when they are sent
Server-side Delays Test:
Server-side delays that slow down page delivery can significantly and (more importantly) permanently affect usage by users with the test. Both Bing and and Google ran similar tests that support this claim.
Bing’s test: Bing delayed server response by ranges from 50ms to 2000ms for their control group. You can see the results of the tests above. Though the number may seem small it’s actually large shifts in usage and when applied over millions can be very significant to usage and revenue. The results of the test were so clear that they ended it earlier than originally planned. The metric Time To Click is quite interesting. Notice that as the delays get longer the Time To Click increases at a more extreme rate (1000ms increases by 1900ms). The theory is that the user gets distracted and unengaged in the page. In other words, they’ve lost the user’s full attention and have to get it back.
Google’s Test: Google ran a similar experiment for where they tested delays ranging from 50ms – 400ms. The chart above shows the impact that it had on users for the 7 weeks they were in the test. The most interesting thing to note was the continued effect the experiment had on users even after it had ended. Some of the users never recovered — especially those with the greater delay of 400ms. Google tracked the users for an additional 5 weeks (for a total of 12).
(I’ve included more on the other tests after the jump.)
The raw numbers of Google’s Server-Side test:
Progressive rendering can have huge performance and user satisfaction gains. Bing ran a test where they sent down their easy-to-compute-and-serve header before they sent down their results. They used Chunked Transfer Encoding to deliver the bits. This resulted in more user engagement presumably due to the immediate visual response. Hopefully this response will outweigh the dev costs of implementing a more sophisticated application.
The test results:
Page Weight Increase Test:
Page weight increase had relatively little effect on users. Bing Search conducted a test where they added increasingly larger HTML comments to a page. Their normal page size is 10Kb (gzipped) and they added up to 32Kb. It was only at this highest addition that they noticed a small decrease in clicks. The test was conducted for three weeks in the US only. This test is the most search specific of them all. Search pages are among the fastest served and lightest. Also as the additional payload was just comments there was no rendering cost added in the test.
The slides for the talk have been posted. Video will be up shortly.