Web Crawler Application Python Project
What is a Web Crawler?
Web crawlers are classified as generic web crawlers, specialized web crawlers, incremental web crawlers, and deep web crawlers based on the technology and structure employed.
The need for network information has skyrocketed since the dawn of the big data age. Many firms gather external data from the Internet for a variety of purposes, including researching the competition, summarizing news items, watching market trends, and gathering daily stock prices to construct prediction models. As a result, web crawlers are becoming more vital. Web crawlers explore or collect information from the Internet in accordance with pre-defined criteria.
Web Crawler's Basic Workflow
A standard web crawler's fundamental procedure is as follows:
- Obtain the original URL. The first URL is the web crawler's entrance point, and it points to the web page that needs to be crawled.
- We need to acquire the HTML content of the web page while crawling it, then parse it to retrieve the URLs of all the sites connected to it.
- Organize these URLs in a queue;
- Loop through the queue, reading the URLs one by one from the queue, crawling the associated web page for each URL, then repeating the crawling process;
- Check to see whether the halt condition has been satisfied. If no stop condition is specified, the crawler will continue crawling until it is unable to get a new URL.
Prepare The Environment For Web Crawling Application
- Ascertain that a browser, such as Chrome, Internet Explorer, or another, is installed in the environment.
- Python should be downloaded and installed.
- Obtain an appropriate IDL.
- Visual Studio Code is used in this article.
- Install the Python packages that are necessary.
Pip is a tool for managing Python packages. It has search, download, install, and uninstall options for Python packages. When you download and install Python, this utility will be included. As a result, we may use 'pip install' to install the libraries we need.
- install beautifulsoup4 with pip
- asks for pip install
- install lxml with pip
• BeautifulSoup is a library for parsing HTML and XML data quickly and effortlessly.
• lxml is a library for speeding up the parsing of XML files.
• requests is a library that may be used to make HTTP requests (such as GET and POST). We'll mostly utilize it to get to the source code of any website.
Source Code for Web Crawler Python Project
import requests
import lxml
from bs4
import BeautifulSoup
from xlwt
import *
workbook = Workbook(encoding = 'utf-8')
table = workbook.add_sheet('data')
table.write(0, 0, 'Number')
table.write(0, 1, 'movie_url')
table.write(0, 2, 'movie_name')
table.write(0, 3, 'movie_introduction')
line = 1
url = "https://www.rottentomatoes.com/top/bestofrt/"
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 QIHU 360SE'
}
f = requests.get(url, headers = headers)
movies_lst = []
soup = BeautifulSoup(f.content, 'lxml')
movies = soup.find('table', {
'class': 'table'
})
.find_all('a')
num = 0
for anchor in movies:
urls = 'https://www.rottentomatoes.com' + anchor['href']
movies_lst.append(urls)
num += 1
movie_url = urls
movie_f = requests.get(movie_url, headers = headers)
movie_soup = BeautifulSoup(movie_f.content, 'lxml')
movie_content = movie_soup.find('div', {
'class': 'movie_synopsis clamp clamp-6 js-clamp'
})
print(num, urls, '\n', 'Movie:' + anchor.string.strip())
print('Movie info:' + movie_content.string.strip())
table.write(line, 0, num)
table.write(line, 1, urls)
table.write(line, 2, anchor.string.strip())
table.write(line, 3, movie_content.string.strip())
line += 1
workbook.save('movies_top100.xls')
The output is:
