Referers and Getting To First Base

My wife was wondering why 60% of her web site traffic was showing as “direct entry” – she couldn’t imagine all of these visitors were typing in her web site URL from scratch.  At the same time, I was working with a client that was depending on the ‘referer‘ value for some calculations.   And something wasn’t adding up to them.  I couldn’t really find a good list of bad reasons for using referers, but I did find a whole lot of partial lists.

The problem is that a lot of clicks do not come with referers – those strings of information that tells you where someone is clicking to get to you.  Being a guy, I made a sports analogy – sure, you could count hits.  But there are many ways to get to your site – just like there are seven ways to get to first base (PS don’t use sports analogies that are double entendres to explain web technology to spouses – but thats for another day).

Getting To First Base

• Hit
• Error
• Walk
• Hit by pitch
• Dropped third strike (you still got to run)
• Catcher interference
• Fielder’s choice

Getting To Your Page Without Referers

Just like the baseball list above, some reasons for arriving without referers makes sense – but others you really have to think about.  Referers both easy to hack, and are a poor way to evaluate a particular traffic source.  Statistical accuracy for traffic evaluations, sure, but referers are not a great idea for individual validation by themselves.  For example,  Meta Refresh is actually a common way to cloak redirects to hijack or bypass affiliate systems.

• Url typed by hand
• Request from local file
• javascript “window.location=” (though there are really hacky workarounds)
• Request from bookmark
• HTTPS link to HTTP target
• Desktop email client link
• New Window( popup, ‘target=’, user right-click)
• Meta refresh redirect
• Browser refresh/reload
• Browser toolbar or plugins
• Web crawlers, scrapers, bots, scripts – though many forge referers
• AJAX call
• Sourced files (CSS, javascript files, Flash – though Flash can also be used to forge referers)
• Proxy server filtering
• Server configuration for 301 and 302 redirects  (sometimes strip referer data and it is unavailable to the underlying scripting environment – PHP/Perl/Ruby etc).

And yes, referer is misspelled.  Rumor is that it was misspelled by Tim Berners-Lee, who wrote the original HTTP spec – and embedded in millions of browsers and web servers.