Recently I started an activity to answer stack overflow question tagged with Selenium. To my surprise, most of the question were coming for scrapping website contents with the help of selenium and python.

And most of the question were having the same problem, that was that they were struggling with pagination. I answered few questions and below is my one code snippet from ebay site scrapping for apple products.

Tip: Always check how your URL is constructing when you are talking about pagination. Most of the time, your URL will be appending page no which will help you to loop.

from time import sleep

from selenium.webdriver.common.by import By

from selenium import webdriver

PATH = r" Your chromdrive exe path"

driver = webdriver.Chrome(PATH)

url_first_part = "https://www.ebay.com/b/Apple/bn_21819543"

for i in range(5):

    i = i + 1

    url = url_first_part + "?_pgn=" + str(i)

    driver.get(url)

    sleep(3)

    driver.maximize_window()

    products = driver.find_element(By.XPATH, '//*[@id="s0-27-9-0-1[0]-0-1"]/ul')

    print(products.text)

driver.quit()

Output: Now it will navigate to each page (till 5) and will scrape the content.