Python 3 urllib: making requests with GET or POST parameters

The urllib packages are the most common and easy way of making requests in Python and offer all the necessary functionality needed to customize and add the required information to a request including GET and POST parameters.

GET requests are generally used to just fetch information from a server, any included parameters are usually to tell the server to format the data in a certain way, these parameters should not include sensitive data. GET parameters are attached to a request via a query string at the end of a URL.

POST requests are generally used to send data to a server, included parameters can sometimes include sensitive data such as usernames and passwords, although these will also require added protection such as SSL. POST parameters are attached to a request via the request body, and are more difficult to access than GET parameters.

Making a basic request using urllib without parameters

Python's urllib library is split up into several modules. To make a basic request in Python 3, you will need to import the urllib.request module, this contains the function urlopen() which you can use to make a request to a specified URL. To fetch the actual output of the request, you can use the read() function on the returned object to read the contents.

import urllib.request 
     
response = urllib.request.urlopen( "http://example.com" ) 
response_text = response.read() 

The urlopen() returns an object that can be used in a context manager, this will ensure that the response object is cleaned up properly once you are finished with it, and it helps to logically separate your response handling from the rest of your code to ensure greater readability.

import urllib.request 
     
with urllib.request.urlopen( "http://example.com" ) as response: 
    response_text = response.read() 
    print( response_text ) 

The above code will return the HTML content for the site example.com.

if no data argument is given with the urlopen() function, then the request method will be GET. If the data argument is given, then the request method will be POST.

Making a request with urllib with GET parameters

The parameters that are passed alongside a GET request are done so through the query string attached to the end of a URL, so adding your own parameters does not require any special functions or classes, all you'll need to do is make sure your query string is properly encoded and formatted.

To create your key-value pairs that will be contained in the query string, you can create a dictionary object, this object can then be encoded and formatted using urllib's urlencode() function contained within the urllib.parse module.

import urllib.request 
import urllib.parse 
    
url = "http://example.com" 
params = { 
    "param1": "arg1", 
    "param2": "arg2", 
    "param3": "arg3" 
}    
 
query_string = urllib.parse.urlencode( params ) 
 
url = url + "?" + query_string 
 
with urllib.request.urlopen( url ) as response: 
    response_text = response.read() 
    print( response_text ) 
 

The formatted string that is returned by the urlencode() function should look like this:

param1=arg1&param2=arg2&param3=arg3

When connecting it to the end of your URL, don't forget it add a '?' character to signify the start of your query string. Once you've added the query string, the final URL should look like this:

http://example.com?param1=arg1&param2=arg2&param3=arg3 

Making a request with urllib with POST parameters

The parameters that are passed alongside a POST request are done so via the request body and are a generally a little more difficult to access compared to GET parameters, however, urllib does make it quite easy for us to add POST parameters.

Similarly to how we added the GET parameters, you can create a dictionary to store the key-value pairs of your POST parameters that can then be formatted using urlencode(). We must then go through the extra step of encoding the formatted string into bytes and specifying a desired character encoding.

We can then open the request as normal using urlopen() but this time we will add our data as an extra argument, this will add our parameters to the request and change the request type to POST (the default being GET).

import urllib.request    
import urllib.parse    
    
url = "http://example.com"    
params = {    
    "param1": "arg1",    
    "param2": "arg2",    
    "param3": "arg3"    
}    
    
query_string = urllib.parse.urlencode( params )    
data = query_string.encode( "ascii" )    
    
with urllib.request.urlopen( url, data ) as response:     
    response_text = response.read()     
    print( response_text ) 

This will make a request to example.com with the 3 parameters attached to the request body.

If you have any questions feel free to leave them in the comments below!