Browser Automation with Python and Selenium — 5: Waits

Make sure it is there

In the previous post, we looked at how to locate elements before interacting with them. We will explore how waits work in Selenium to make sure that elements are presented in the DOM before interacting with them in this post.

The document.readyState property of an HTML document describes the loading state of the current document. By default, a driver.get request returns to the caller when the ready state becomes complete.

There are 3 kinds of page loading strategies for Selenium WebDriver.

normal

Wait for the entire page is loaded that is determined with the load event.

eager

Wait until the initial HTML document has been completely loaded and parsed without stylesheets, images, and subframes specified by the DOMContentLoaded event.

none

If the page loading strategy is set to none, Selenium WebDriver only waits until the initial page is downloaded.

When Selenium WebDriver loads a page, it uses the normal strategy by default.

Different elements on the page can be loaded at different times. This may cause failures to locate elements. If an element is not present in the DOM structure, we get an exception when we try to access and operate on it. These issues can be solved with waits in Selenium.

Elements added after the document has completed loading cannot be found by the find_* methods. Our automation code will raise NoSuchElementException in this case. Waiting allows the automated task execution to elapse a certain amount of time before continuing with the next step.

There are 3 kinds of waits:

1. Implicit
2. Explicit
3. Fluent

Implicit Wait

With implicit wait, WebDriver polls the DOM for the specified amount of time before raising an exception. The default setting is 0. Once it is set, it is valid for the entire life of the WebDriver instance.

In the following example, the id is given wrong deliberately to the find_element_by_id method. Since we set the implicit wait to 10 seconds, we will get the exception after 10 seconds of waiting.

# output after 10 seconds
Message: Unable to locate element: [id="typer"]

Implicit waits are easy to use but since the wait interval is valid for all the elements and during the lifetime of the WebDriver instance, they increase the overall execution time.

Selenium provides explicit waits to avoid the drawbacks of these hardcoded global waits.

Explicit Wait

Unlike hardcoded implicit wait, an explicit wait waits for a specific condition to occur. It also applies only to a specific element given rather than all elements globally.

The condition is called with a certain frequency(0.5 seconds by default) until the timeout of the wait is elapsed. If the condition fails, TimeoutException is raised.

There are some methods provided by the Selenium Python API for the common cases.

They are defined in the expected_conditions module under selenium.webdriver.support package.

List of the expected conditions

* alert_is_present
* element_located_selection_state_to_be
* element_located_to_be_selected
* element_selection_state_to_be
* element_to_be_clickable
* element_to_be_selected
* frame_to_be_available_and_switch_to_it
* invisibility_of_element
* invisibility_of_element_located
* new_window_is_opened
* number_of_windows_to_be
* presence_of_all_elements_located
* presence_of_element_located
* staleness_of
* text_to_be_present_in_element
* text_to_be_present_in_element_value
* title_contains
* title_is
* url_changes
* url_contains
* url_matches
* url_to_be
* visibility_of
* visibility_of_all_elements_located
* visibility_of_any_elements_located
* visibility_of_element_located

In the following example, the end of the text is checked for an animated element. As soon as the condition is met, wait returns true and the full text can be printed.

# output
Full text typed: Ask Doctor Python to fix your code...

Custom Wait Conditions

You can also create your custom wait conditions if the previous methods don’t respond to your requirements. A custom wait condition can be created using a class with __call__ method which returns False when the condition doesn’t match.

The following example shows a custom wait condition implementation that checks if an element has a certain css class.

Since implicit waits are applied globally, it is recommended that not to mix implicit and explicit waits.

Fluent Wait

FluentWait defines the maximum amount of time to wait for a condition, as well as the frequency with which to check the condition. Python API does not have a FluentWait class, but it kind of supports it with keyworded arguments.

I suggest you read the Fluent Interface post by Martin Fowler.

Things to Remember

• When Selenium WebDriver loads a page, it uses the normal page loading strategy by default that waits for the load event to return to the caller from the driver.get request.
• With implicit wait, WebDriver polls the DOM for the specified amount of time before raising an exception. Once it is set, it is valid for the entire life of the WebDriver instance and for all the elements on the page.
• An explicit wait waits for a certain condition to occur for a specific element.

In the next post, I will write about navigation among URLs, windows, and frames.

References

1. https://www.selenium.dev/documentation/en/webdriver/waits/
2. https://www.softwaretestingo.com/selenium-wait-commands/
3. https://selenium-python.readthedocs.io/waits.html
4. https://github.com/SeleniumHQ/selenium/blob/trunk/py/selenium/webdriver/support/wait.py

Thank you for your time.