New Features in Acunetix WVS 8: Crawling of Websites with Different User-Agent Strings
- Friday, March 16, 2012, 6:18
- Articles, Threat Research
When you visit a website your browser sends an HTTP header called “User-Agent” to the web server. This header indicates which web browser you are using, its version number and details about your operating system and version.
Various browsers send different User-Agent strings. For example, Internet Explorer 9 sends Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0). If you are using an iPhone 4, for example, you will have a User-Agent similar to this one: Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7.
In order to improve the user experience, more and more websites display one version for users who access the website from their mobile devices and another version for users who access the website from their desktop computers. When accessed, these websites automatically know if you are using a mobile as they parse the User-Agent string. Also, some websites show some content when visited by Google, while showing other content to regular users.
For example, if you visit Facebook from a regular desktop computer you will see this page:
However, if you visit the same page from (continue reading...)