Selenium Scrolling a Web Page

Selenium is an open-source program that automates web browsers. It offers a single interface that enables us to create test scripts in several different programming languages, including Ruby, Java, NodeJS, PHP, Perl, Python, and C#. It is also compatible with different Operating Systems like Windows, Mac, Linux, etc.

Selenium ecosystem consists of four major tools namely - Selenium WebDriver, Selenium Grid, Selenium IDE, and Selenium RC. Selenium Webdriver is mainly used to execute the scripts according to the browser we are using.

In this article, we will see various methods by which scrolling a webpage is possible using Selenium while we test our applications.

Prerequisites:

• Install the Selenium Python package using the following command:

pip install selenium

• Install Chrome Webdriver to use Selenium Click here.

Steps to Scroll a Webpage using Selenium:

Selenium has many different methods for scrolling a webpage. There are several options for how we want to scroll across a webpage: we can scroll through the webpage slowly with a defined speed, go up or down to a specific element, or go straight to the top or bottom of the page. For all the ways of scrolling, the initial setup and imports remain the same and are mentioned below.

1. Import the required packages.

The webdriver module will be used to run the Webdriver Implementations for Chrome browser. Time package will be used for functions like sleep for clear visibility of what is happening on the webpage.

from selenium import webdriver

import time

2. Define the necessary string constants.

PATH variable is the location of the chrome driver we have installed and will be used in initializing the instance of webdriver. URL is the example webpage used for the demo.

PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"

URL = "https://vrii14.github.io/"

3. Initializing and defining Chrome Options.

These options ensure we do not get any unnecessary warnings while running Selenium due to some Chromedriver issues.

options = webdriver.ChromeOptions()

options.add_experimental_option('excludeSwitches', ['enable-logging'])

4. Creating an instance of the webdriver and opening the URL webpage.

browser = webdriver.Chrome(PATH, options=options)

browser.get(URL);

browser.maximize_window();

Here, we create an instance of the Chrome webdriver in the first line using the PATH variable and the options. In the second line, we open the webpage of the URL provided. Third line maximizes the Chrome window.

Scroll the Webpage:

1. Scrolling through the webpage slowly (infinite till end of page):

To scroll through the webpage slowly, we create a while loop and use window.scrollBy function to scroll down by that many pixels down and followed by time.sleep function for the scrolling to be slow and visible. We also use a count variable "stopScrolling" for the while loop to end after reaching the end of webpage. The number "120" will according to the length of the webpage and can be determined by trial and error or printing the counter and checking at value has the webpage ended.

Below is the Python program to scroll through the webpage slowly:

# Python program to scroll through
# the webpage slowly
from selenium import webdriver
import time

# string constants
PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"
URL = "https://vrii14.github.io/"

# chrome options
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])

try:
    browser = webdriver.Chrome(PATH, options=options)
    browser.get(URL)
    browser.maximize_window()

    # sleep for visibility of change
    time.sleep(2)

# scrolling down slowly
    stopScrolling = 0
    while True:
        stopScrolling += 1
        browser.execute_script("window.scrollBy(0,40)")
        time.sleep(0.5)
        if stopScrolling > 120:
            break
    time.sleep(3)
finally:
    browser.quit()

Output:

2. Scroll to the bottom of the webpage:

For scrolling to the bottom of the page we use the windows.scrollTo function and specify the end of webpage pixel value with document.body.scrollHeight which is defined already.

Method:

window.scrollTo(0,document.body.scrollHeight)

Code:

from selenium import webdriver
import time

# string constants
PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"
URL = "https://vrii14.github.io/"

# chrome options
options = webdriver.ChromeOptions();
options.add_experimental_option('excludeSwitches', ['enable-logging']);

try:
    browser = webdriver.Chrome(PATH, options=options);
    browser.get(URL);
    browser.maximize_window();
    # sleep for visibility of change
    time.sleep(2);
    # scroll to the bottom 
    browser.execute_script("window.scrollTo(0,document.body.scrollHeight)");
    time.sleep(3);
finally:
    browser.quit()

Output:

3. Scroll based on the visibility of the Web element on the page:

For scrolling to the particular element, we need to first find the element by any specific characteristic like id, class, xpath, link text, etc and the value can be found by inspecting the webpage. Here, we use the id of the element to find it with this function browser.find_element_by_id(). After getting the element, we can scroll to the particular element using the scrollItoView function and providing the element as the argument.

Method:

scrollIntoView();

Code:

from selenium import webdriver
import time

# string constants
PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"
URL = "https://vrii14.github.io/"

# chrome options
options = webdriver.ChromeOptions();
options.add_experimental_option('excludeSwitches', ['enable-logging']);

try:
    browser = webdriver.Chrome(PATH, options=options);
    browser.get(URL);
    browser.maximize_window();
    # sleep for visibility of change
    time.sleep(2);
    # scroll to the top
    element = browser.find_element_by_id("journey");
    browser.execute_script('arguments[0].scrollIntoView(true)', element);
    time.sleep(3);
finally:
    browser.quit()

Output:

4. Scroll down a page by specified pixels:

For scrolling by a specified number of pixels, we use the function window.scrollBy(0,x) and specify the number of pixels in place of x. If we need to scroll down by the number of pixels, then the value of x is positive and if we need to scroll up with same number of pixels we can negate the value of x.

Method to be used:

scrollBy();

Code:

from selenium import webdriver
import time

# string constants
PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"
URL = "https://vrii14.github.io/"

# chrome options
options = webdriver.ChromeOptions();
options.add_experimental_option('excludeSwitches', ['enable-logging']);

try:
    browser = webdriver.Chrome(PATH, options=options);
    browser.get(URL);
    browser.maximize_window();
    # sleep for visibility of change
    time.sleep(2);
    browser.execute_script("window.scrollBy(0,850)");
    time.sleep(3);
finally:
    browser.quit()

Output:

5. Scroll to the top of the page:

For scrolling to top of the webpage, we need to set the pixel values to coordinates of either (0,0) or (document.body.scrollHeight, 0). For visibility of scrolling to the top of the page, we need to first scroll down and then to the up again.

Method to be used:

scrollTo(0,0);

Code:

from selenium import webdriver
import time

# string constants
PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"
URL = "https://vrii14.github.io/"

# chrome options
options = webdriver.ChromeOptions();
options.add_experimental_option('excludeSwitches', ['enable-logging']);

try:
    browser = webdriver.Chrome(PATH, options=options);
    browser.get(URL);
    browser.maximize_window();
    # sleep for visibility of change
    time.sleep(2);
    # scroll to the top
    browser.execute_script("window.scrollTo(0, 0)");
    time.sleep(3);
finally:
    browser.quit()

Output:

6. Scroll a webpage horizontally:

For scrolling a page horizontally, we just have to change the x-coordinate in the windows.scrollBy() function instead of y-coordinate. The value of x coordinate determines the pixels by how much it will scroll horizontally.

Method:

window.scrollBy(850, 0);

Code:

from selenium import webdriver
import time

# string constants
PATH = r"C:\Users\VRINDA\Desktop\chromedriver_win32\chromedriver.exe"
URL = "https://vrii14.github.io/"

# chrome options
options = webdriver.ChromeOptions();
options.add_experimental_option('excludeSwitches', ['enable-logging']);

try:
    browser = webdriver.Chrome(PATH, options=options);
    browser.get(URL);
    browser.maximize_window();
    # sleep for visibility of change
    time.sleep(2);
    #scroll the page horizontally
    browser.execute_script("window.scrollBy(850, 0)");
    time.sleep(3);
finally:
    browser.quit()

Output:

Close the browser:

browser.quit()

This will finally close the browser.

Conclusion:

We have seen how we can use selenium scrolling in multiple ways to help test our applications.