Signup/Sign In

Understanding Web Scraping in Python - Best Practices

Posted in Programming   LAST UPDATED: NOVEMBER 7, 2023

    Web scraping is essentially a method to collect data from websites automatically. Think of it as a digital tool that can fetch and extract information from different web pages. Python is a go-to language for web scraping due to its simplicity and a set of handy libraries specifically designed for this purpose.

    Installing Python

    To begin web scraping, you need Python on your machine. Here's a quick guide to install it:

    • Visit the Python website at

    • Click on "Downloads" and select the appropriate version for your system (Windows, macOS, or Linux).

    • Download the installer and run it. Remember to tick the "Add Python to PATH" option.

    • To verify the installation, open your command line (cmd for Windows, Terminal for macOS or Linux) and type python --version. You should see the installed version number.

    Preparing for Scraping

    You'll need a couple of tools: requests to access web pages and BeautifulSoup to process the page content. Install them like this:

    • Open your command line.

    • Enter pip install requests and press Enter.

    • Then, enter pip install beautifulsoup4 and press Enter.

    Now, you're equipped to scrape!

    Your First Web Scraping Code in Python

    Let's dive into a simple scraping task: extracting titles from a blog.

    import requests
    from bs4 import BeautifulSoup
    # The target website
    url = ''
    # Fetching the webpage
    response = requests.get(url)
    # Successful response
    if response.ok:
        # Grabbing the content
        content = response.text
        # Parsing the content
        soup = BeautifulSoup(content, 'html.parser')
        # Searching for all h1 tags where titles are likely placed
        titles = soup.find_all('h1')    
        # Looping through and printing titles
        for title in titles:

    This snippet sends a request to a website, parses the page for <h1> tags - commonly used for titles - and prints out their text content neatly.

    Smart Web Scraping Practices

    When you're ready to start scraping data from the web, it's not just about writing a script and letting it loose. There's a responsible and efficient way to do it. Let's dive into some smart web scraping practices that will keep your activities smooth and sustainable.

    1. Follow the Rules

    Every website has a set of rules for bots, found in their robots.txt file. It's like the rulebook for automated access, telling you which pages you can or cannot scrape. Here's how you can check these rules with Python:

    import requests
    # The URL of the website's robots.txt file
    url = ''
    # Fetching the content of robots.txt
    response = requests.get(url)
    # If the request was successful
    if response.ok:
        # Print the contents of the robots.txt file

    This code will print out the contents of the robots.txt file from, letting you see the scraping guidelines set by the website.

    2. Be Considerate

    Bombarding a website with a ton of requests in a short time can slow it down or even cause it to crash. This is bad for the site and can get you banned. To prevent this, you should space out your requests. Here's a simple way to add a delay between requests:

    import time
    # ... your scraping code here ...
    # Wait for 5 seconds before making the next request

    Adding time.sleep(5) will pause your script for 5 seconds between requests, which is a simple way to be more polite with your scraping.

    3. Blend In

    Websites can often tell when they're being scraped. If you're doing a lot of scraping, you might want to make your requests look more like they're coming from a real user. Here's how you can change the user agent of your requests:

    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
    response = requests.get(url, headers=headers)

    By setting a User-Agent that mimics a popular browser, your script will look less like a bot and more like a human visitor.

    4. Error Handling

    Not everything will go according to plan. Websites change, and your script might encounter errors. It's important to write your code to handle these gracefully. You can easily handle errors in Python using proper exception handling.

    Here's an example of handling errors:

        response = requests.get(url)
    except requests.exceptions.HTTPError as errh:
        print ("Http Error:",errh)
    except requests.exceptions.ConnectionError as errc:
        print ("Error Connecting:",errc)
    except requests.exceptions.Timeout as errt:
        print ("Timeout Error:",errt)
    except requests.exceptions.RequestException as err:
        print ("OOps: Something Else",err)

    This code tries to make a request and catches various errors that could occur, printing out a message for each.

    Code Maintenance

    As your scraping needs grow, so will your code. Keeping it organized is key. Comment your code, use functions to organize tasks, and don't repeat yourself. Here's a snippet showing a well-organized code structure:

    def fetch_page(url):
            response = requests.get(url)
            return response.text
        except requests.RequestException as e:
            print(f"Error fetching {url}: {e}")
    def parse_titles(page_content):
        soup = BeautifulSoup(page_content, 'html.parser')
        titles = soup.find_all('h1')
        return [title.text.strip() for title in titles]
    # Use the functions
    url = ''
    page_content = fetch_page(url)
    if page_content:
        titles = parse_titles(page_content)
        for title in titles:

    This code has separate functions for fetching a page and parsing titles, making it easier to read and maintain.

    Incorporating a web scraping API can simplify these processes even more. It can handle the intricacies of making requests and parsing the data for you, which means you can focus on the logic and storage of your scraped data. With these practices in place, you'll be scraping data more effectively and responsibly.

    Wrapping Up

    Web scraping with Python is a handy technique for data collection. Proper installation of Python and the right libraries like requests and BeautifulSoup are the first steps. Always scrape with care, respecting the website's rules and managing your scraping frequency. With these basics down, you're on your way to becoming a proficient web scraper.

    About the author:
    I like writing content about C/C++, DBMS, Java, Docker, general How-tos, Linux, PHP, Java, Go lang, Cloud, and Web development. I have 10 years of diverse experience in software development. Founder @ Studytonight