Track Prices Automatically: Build Your Own Web Scraper in Python

Introduction

Welcome to PythonSage! In this post, we will explore how to build a web scraper to monitor and track product prices on e-commerce sites using Python, BeautifulSoup, and requests. This project is an excellent way to automate price tracking and get notified when prices drop to your desired level. Let's dive in!

Prerequisites

Before we begin, ensure you have the following Python packages installed:

pip install requests beautifulsoup4

Step 1: Setting Up the Web Scraper

First, we will import the necessary libraries and set up the URL of the product page we want to track.

import requests

from bs4 import BeautifulSoup

import time

import smtplib

from email.mime.text import MIMEText

 

# URL of the product page

url = 'https://www.example.com/product-page'

 

# Set up the headers to simulate a request from a web browser

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}

 

# Desired price threshold

threshold_price = 50.00

Step 2: Fetching and Parsing the Webpage

We will create a loop to continuously check the price of the product. If the price drops below our threshold, we'll send an email notification.

# Main loop to keep checking the price

while True:

    # Fetch the webpage

    response = requests.get(url, headers=headers)

   

    if response.status_code == 200:

        print("Successfully fetched the webpage")

    else:

        print("Failed to fetch the webpage")

        time.sleep(3600)  # Wait for an hour before trying again

        continue

   

    # Parse the HTML content

    soup = BeautifulSoup(response.content, 'html.parser')

   

    # Extract the price from the HTML

    price_tag = soup.find('span', {'class': 'g9WBQb'})

    if price_tag:

        price_text = price_tag.text.strip().replace('SAR', '').replace('+ tax', '').replace(',', '').strip()

        price = float(price_text)

        print(f"The current price of the product is {price}")

       

        # Check if the price is below the threshold

        if price < threshold_price:

            send_email(price)

    else:

        print("Price tag not found on the webpage")

   

    # Wait for an hour before checking the price again

    time.sleep(3600)

Step 3: Sending Email Notifications

We will define a function to send email notifications when the price drops below our threshold. Make sure to replace the placeholders with your actual email credentials.

# Function to send email alert

def send_email(price):

    from_email = 'your_email@gmail.com'  # Replace with your email

    to_email = 'recipient_email@gmail.com'  # Replace with recipient's email

    subject = 'Price Alert!'

    body = f'The price of the product has dropped to {price}. Check it out here: {url}'

   

    msg = MIMEText(body)

    msg['Subject'] = subject

    msg['From'] = from_email

    msg['To'] = to_email

   

    server = smtplib.SMTP('smtp.gmail.com', 587)

    server.starttls()

    server.login(from_email, 'your_app_password')  # Replace 'your_app_password' with your actual app password

    server.sendmail(from_email, to_email, msg.as_string())

    server.quit()

Full Script

Here is the complete script combining all the parts together:

import requests

from bs4 import BeautifulSoup

import time

import smtplib

from email.mime.text import MIMEText

 

# Function to send email alert

def send_email(price):

    from_email = 'your_email@gmail.com'  # Replace with your email

    to_email = 'recipient_email@gmail.com'  # Replace with recipient's email

    subject = 'Price Alert!'

    body = f'The price of the product has dropped to {price}. Check it out here: {url}'

   

    msg = MIMEText(body)

    msg['Subject'] = subject

    msg['From'] = from_email

    msg['To'] = to_email

   

    server = smtplib.SMTP('smtp.gmail.com', 587)

    server.starttls()

    server.login(from_email, 'your_app_password')  # Replace 'your_app_password' with your actual app password

    server.sendmail(from_email, to_email, msg.as_string())

    server.quit()

 

# URL of the product page

url = 'https://www.example.com/product-page'

 

# Set up the headers to simulate a request from a web browser

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}

 

# Desired price threshold

threshold_price = 50.00

 

# Main loop to keep checking the price

while True:

    # Fetch the webpage

    response = requests.get(url, headers=headers)

   

    if response.status_code == 200:

        print("Successfully fetched the webpage")

    else:

        print("Failed to fetch the webpage")

        time.sleep(3600)  # Wait for an hour before trying again

        continue

   

    # Parse the HTML content

    soup = BeautifulSoup(response.content, 'html.parser')

   

    # Extract the price from the HTML

    price_tag = soup.find('span', {'class': 'g9WBQb'})

    if price_tag:

        price_text = price_tag.text.strip().replace('SAR', '').replace('+ tax', '').replace(',', '').strip()

        price = float(price_text)

        print(f"The current price of the product is {price}")

       

        # Check if the price is below the threshold

        if price < threshold_price:

            send_email(price)

    else:

        print("Price tag not found on the webpage")

   

    # Wait for an hour before checking the price again

    time.sleep(3600)

Conclusion

In this tutorial, we demonstrated how to create a web scraper using Python to track product prices on e-commerce sites. With BeautifulSoup and requests, you can easily fetch and parse webpage content, extract product prices, and send email notifications when prices drop. This automation saves you time and ensures you never miss a good deal.

External Links to Learn More:

BeautifulSoup Documentation

Requests Documentation

For more Python tips, tutorials, and projects, stay tuned to PythonSage.

 Happy coding!