Allison Rhodes, the Community Manager over at OpenDNS was in touch about doing a customer profile on me for the OpenDNS website. I guess they do these kind of things if you blog about them and say nice things. They were particularly interested in commercial users of their service and I blogged about using their service at Sutton Group’s head office.
This recent attention spurred me to test the speed of their service again. In my original post, about 7 months ago, I concluded that using OpenDNS made my browsing experience about 3x faster. This time around I tested to see how much DNS resolution affects a web page’s load time. I tested with my MySpace page. The results are interesting, OpenDNS is still fast but has fallen to the bottom of the pack while Telus has made some major improvements to rise to the top.
I chose MySpace because the pages are cluttered with content coming from many different domains. My page using the default layout pulls information from 20 servers (yikes!):
- a169.ac-images.myspacecdn.com
- a27.ac-images.myspacecdn.com
- a727.ac-images.myspacecdn.com
- cache.static.userplane.com
- creative.myspace.com
- i.myspace.com
- myspace-039.vo.llnwd.net
- myspace-274.vo.llnwd.net
- myspace-369.vo.llnwd.net
- myspace-429.vo.llnwd.net
- myspace-524.vo.llnwd.net
- myspace-559.vo.llnwd.net
- myspace-772.vo.llnwd.net
- myspace-874.vo.llnwd.net
- myspace-879.vo.llnwd.net
- myspace-927.vo.llnwd.net
- myspace-954.vo.llnwd.net
- myspace-965.vo.llnwd.net
- pagead2.googlesyndication.com
- x.myspace.com
This is why slow resolvers will make you weep when loading pages like these. If each hostname takes a quarter of a second to resolve, that’s 5 seconds added to the page load. Five seconds may not sound like much but count it out loud… it’s a long time isn’t it?
This trend, web pages pulling content from many different sources is increasing (Banner ads, analytics, page counters, etc). It adds up quickly, sucking away your life milliseconds at a time.
Here are the results of my test:
This graph shows how long it takes to resolve all 20 hostnames. Essentially how much time each server adds to the page load times.
The Control is my Linksys WRT54G router running the HyperWRT Thibor firmware. The only thing faster would be to run a local dns cache locally on my testing machine (iMac). Regardless it did very well, 0.54 seconds to resolve 20 domains. Keep in mind that this is just a cache. The router is configured to query OpenDNS’s servers on cache misses. I discarded those results so everything is directly off the router.
The Telus test results are surprisingly good and bad. The 154.11.128.187 server is extremely slow adding an average of 25 seconds to the page load time. Their other server, 154.11.128.59 has improved drastically taking second place. They must have done some major upgrades to it in the last 7 months. It’s faster than both of OpenDNS’s servers now.
What is interesting is that OpenDNS has dropped to the bottom of the pack, slower than even the CityTel server. I love 204.244.99.66. It is run by a colleague out of Prince Rupert and is rarely ever down. I ping it all the time when I’m testing for network / routing connectivity.
Why has the OpenDNS servers dropped to the bottom of pack? Their growth may have something to do with it. Last July they were handling about 42 million queries a day. Today they are doing over 500 million. More than 12 times the traffic! Three seconds is not great but it’s not bad either considering the change.
However, the DNS resolution game is all about speed so I will be switching back the faster Telus server for my primary and using OpenDNS as a back up resolver. A big thumbs up to Telus for improving their services.
I feel compelled to note that I still consider using OpenDNS’s servers for other things. Their service is quite good and it wouldn’t be much of a hassle to standardize around them. My main reservations are that they are a commercial entity. That means there is a bottom line and an agenda that may not be compatible with mine in the future. For now, they’re still at least a B+ in my book. :)
Update
As requested, here is the shell script that I used.
Ben,
Thanks for the detailed write up. You can be sure we’ll be seeing what’s going on, if anything.
-david
For an unofficial test, I just put all those hostnames into a file and ran this:
Core3:~ davidu$ date; for i in `cat test.dns`; do host $i 208.67.222.222 >/dev/null 2>&1; done; date;
Thu Jan 11 15:16:28 PST 2007
Thu Jan 11 15:16:28 PST 2007
I’ll see what else I can find. Any tips on how you tested?
I tested using a custom shell script that I wrote. I use the time and dig commands. I left the shell script on my home computer. I’ll post that tonight.
You can run this in your shell. My script does essentially the same thing, just 50 times so I have a good average.
> time dig @208.67.222.222 a169.ac-images.myspacecdn.com a27.ac-images.myspacecdn.com a727.ac-images.myspacecdn.com cache.static.userplane.com creative.myspace.com i.myspace.com myspace-039.vo.llnwd.net myspace-274.vo.llnwd.net myspace-369.vo.llnwd.net myspace-429.vo.llnwd.net myspace-524.vo.llnwd.net myspace-559.vo.llnwd.net myspace-772.vo.llnwd.net myspace-874.vo.llnwd.net myspace-879.vo.llnwd.net myspace-927.vo.llnwd.net myspace-954.vo.llnwd.net myspace-965.vo.llnwd.net pagead2.googlesyndication.com x.myspace.com > /dev/null
It should spit out something similar to:
real 0m1.705s
user 0m0.011s
sys 0m0.016s
Those are the results at my work place. I have some awk/sed action going somewhere in the script too I think.
I should also note that network latency also affects the DNS request times. For example:
> time dig @4.2.2.1 {list of myspace hostnames} > /dev/null
real 0m0.771s
user 0m0.011s
sys 0m0.016s
> time dig @208.67.222.222 {list of myspace hostnames} > /dev/null
real 0m1.685s
user 0m0.011s
sys 0m0.016s
Does this mean that OpenDNS has slower resolvers than Verizon’s servers? Well packet travel time needs to be considered as well:
> ping 4.2.2.1
PING 4.2.2.1 (4.2.2.1): 56 data bytes
64 bytes from 4.2.2.1: icmp_seq=0 ttl=244 time=34.886 ms
64 bytes from 4.2.2.1: icmp_seq=1 ttl=244 time=34.265 ms
64 bytes from 4.2.2.1: icmp_seq=2 ttl=244 time=34.729 ms
…
> ping 208.67.222.222
PING 208.67.222.222 (208.67.222.222): 56 data bytes
64 bytes from 208.67.222.222: icmp_seq=0 ttl=50 time=79.360 ms
64 bytes from 208.67.222.222: icmp_seq=1 ttl=50 time=79.703 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=50 time=79.467 ms
…
As you can see the time to get to OpenDNS is almost double that of Verizon. Your mileage may vary depending on the network you are on. Obviously, the closer you are the faster it will be.
However, Verizon’s DNS resolvers are faster. Running the above dig @{ip} {hostnames} > /dev/null hides a lot of the information. If we change it to grep out the query response times:
> time dig @208.67.222.222 {big list of hostnames} | grep ‘Query time’
;; Query time: 86 msec
;; Query time: 80 msec
;; Query time: 87 msec
;; Query time: 82 msec
;; Query time: 82 msec
;; Query time: 81 msec
;; Query time: 84 msec
;; Query time: 84 msec
;; Query time: 80 msec
;; Query time: 81 msec
;; Query time: 88 msec
;; Query time: 85 msec
;; Query time: 85 msec
;; Query time: 81 msec
;; Query time: 87 msec
;; Query time: 85 msec
;; Query time: 84 msec
;; Query time: 80 msec
;; Query time: 87 msec
;; Query time: 81 msec
> time dig @4.2.2.1 {big list of hostnames} | grep ‘Query time’
;; Query time: 44 msec
;; Query time: 42 msec
;; Query time: 56 msec
;; Query time: 36 msec
;; Query time: 35 msec
;; Query time: 35 msec
;; Query time: 36 msec
;; Query time: 36 msec
;; Query time: 35 msec
;; Query time: 34 msec
;; Query time: 35 msec
;; Query time: 35 msec
;; Query time: 35 msec
;; Query time: 36 msec
;; Query time: 35 msec
;; Query time: 35 msec
;; Query time: 42 msec
;; Query time: 35 msec
;; Query time: 35 msec
;; Query time: 35 msec
This just shows that resolution time is dependent on a few factors. In the end the only thing that matters is who can get > ME < those IP addresses faster.
Ben,
Got a traceroute? You *must* be hitting our Seattle location or we got issues…
-david
David,
traceroute to 208.67.222.222 (208.67.222.222), 64 hops max, 40 byte packets
1 10.1.1.250 (10.1.1.250) 1.007 ms 0.303 ms 0.184 ms
2 m142.first-fiscal.biz-inet.com (208.181.56.142) 1.187 ms 1.254 ms 1.103 ms
3 vancbc01dr04.bb.telus.com (64.114.44.222) 10.585 ms 8.536 ms 8.354 ms
4 nwmrbc01gr01.bb.telus.com (154.11.4.66) 10.104 ms 8.842 ms 10.647 ms
5 toroonnlbr00.bb.telus.com (154.11.11.66) 55.349 ms 56.719 ms 55.205 ms
6 dcr1-so-3-2-0.chicago.savvis.net (208.175.10.89) 59.135 ms 61.513 ms 118.611 ms
7 bpr1-so-3-0-0.chicagoequinix.savvis.net (204.70.193.250) 59.286 ms 59.477 ms 79.945 ms
8 0.so-0-0-2.br6.chi2.alter.net (208.174.226.2) 59.862 ms 208.174.226.26 (208.174.226.26) 60.819 ms 0.so-0-0-2.br6.chi2.alter.net (208.174.226.2) 60.261 ms
9 0.so-5-3-0.xl1.chi2.alter.net (152.63.64.50) 60.946 ms 104.616 ms 60.130 ms
10 0.so-6-0-0.xl3.nyc4.alter.net (152.63.16.221) 78.536 ms 78.569 ms 78.919 ms
11 pos6-0.gw2.nyc4.alter.net (152.63.19.221) 78.725 ms 78.201 ms 82.137 ms
12 splicetelecom-newyork-gw.customer.alter.net (157.130.14.214) 78.831 ms 100.290 ms 81.951 ms
13 resolver1.opendns.com (208.67.222.222) 79.446 ms 79.397 ms 80.429 ms
If you guys want to have servers in Canada. I have a good relationship with Peer1. Sutton has all of our servers with them.
Here it is off of the MostlyGeek web server: (wow! 5ms…)
traceroute to 208.67.222.222 (208.67.222.222), 64 hops max, 40 byte packets
1 216.187.85.3 (216.187.85.3) 0.502 ms 0.432 ms 0.358 ms
2 oc48pos5-0.sea-dis-2.peer1.net (216.187.88.226) 3.856 ms 3.806 ms 3.734 ms
3 bpr4-so-1-0-0.seattleswitchdesign.savvis.net (208.173.48.194) 3.984 ms 3.952 ms 3.988 ms
4 bpr3-so-3-0-0.SeattleSwitchDesign.savvis.net (208.172.83.69) 3.980 ms 3.961 ms 3.860 ms
5 peer-1-networks.SeattleSwitchDesign.savvis.net (208.173.49.54) 3.984 ms 4.042 ms P4-1-1-0.r04.sttlwa01.us.bb.verio.net (208.173.50.66) 3.983 ms
6 fa-0.freedom.sttlwa01.us.bb.gin.ntt.net (129.250.12.78) 4.945 ms 4.454 ms 4.860 ms
7 resolver1.opendns.com (208.67.222.222) 5.482 ms 5.209 ms 4.859 ms
I’m curious to know how things are for you now. Here in Victoria, telus users that I have heard from are crying at the painful speed they have now. Looks like the telus name servers have fallen into tar or there is something else amiss on the telus network. I moved my DNS settings on my firewalls to OpenDNS and internet access sprang back to life!
Also, I tried your script but it would not run. When I tried the command (time -p …) on it’s own, things worked fine. I’m not much of a shell scripter so nothing is jumping out at me, but one thing that does not happen is the prompt “doing $i of $1″ never shows.
1ea98aa07804…
1ea98aa078045ad1d155…
in debian sid i have to use:
dig @ip.of.dns.server -f big.list.of.hostnames | grep “Query time”