Changing Browser's User Agent using mechanize Library

In this tutorial we will learn how to change a browser's user agent using the mechanize library and will try to modify the HTTP request header.

What is a User Agent?

In computing, a user agent is anything that is acting on behalf of the user. A Borwser's User agent is a string or line of text, containing information about the browser and operating system, which is sent to the web server in the HTTP request header. This allows the website to customize the content, to best suit the capabilities of the particular device, but sharing this information also raises certain privacy issues.

In a typical HTTP request/response cycle you can view the browser's User-Agent inside the HTTP request/response header. It looks something like:

But, by using mechanize we can change the User-Agent field of the HTTP request header, to deceive(make fool of) the web server. Let's see how we can do so, using a python script we will try to set the user agent as:

user_agent = 
        [('User-agent',
        'Mozilla/5.0 (X11;U;Linux 2.4.2.-2 i586; en-us;m18) Gecko/200010131 Netscape6/6.01'
        )]

Let's have a look at the code:

#Program to change the user agent
#Verify the UserAgent with wireshark tool

import mechanize

#function to browse the web page
def change_user_agent(url, user_agent):
	try:
		#Create browser object
		browser=mechanize.Browser()
		browser.set_handle_robots(False)
		#add user agent
		browser.addheaders=user_agent
		
		#open web url
		page=browser.open(url)
		
		#read page source code
		source_code = page.read()
	
		#print source code
		print source_code
	except:
		print "Error in browsing....."

url = str(raw_input("Enter the website name: "))

#user agent details
user_agent=[('User-agent','Mozilla/5.0 (X11;U;Linux 2.4.2.-2 i586; en-us;m18) Gecko/200010131 Netscape6/6.01')]

change_user_agent(url,user_agent)

Let's run the above script, we entered the name of the website as studytonight.com:

Seems like our script executed successfully. Now let's see if the user agent is changed or not. When you will open Wireshark you will see huge chunk of traffic. Than how to filter the required one? We will use a powerful tool of linux OS, which is the 'nslookup' command.

We have done a lookup on the name of website for which we had run the above script. We now have the IP address of the required website. So now we can filter the traffic from this IP address using Wireshark. Just type: ip.addr == 104.25.205.29 in the wireshark filter bar. You will now see all the packets which have the following IP address. Now, go to the packet which has HTTP protocol and right click on it.

After Right Click → Follow → TCP stream

Although we have browsed the website using mechanize library. But we have successfully spoofed the User-Agent information sent as part of the HTTP request header. These tricks are used by security researchers to hide their identity.