Selenium on Python - navigation issues

In

Marking checkboxes

Issues steps

  1. Locate the element using its Xpath.
  2. Use the click method to click on it.
    • Selenium may not recognize it as clickable, so it raises and exception.

Solution

  1. Locate the element.
  2. Use an ActionChains object to move the cursor to the check box.
  3. Click on it.
1
2
3
4
5
check_box = driver.find_element_by_xpath('Xpath')

actions = webdriver.ActionChains(driver)
actions.move_to_element_with_offset(check_box, -5, 5).perform()
actions.click().perform()

Define the move_to_element_with_offset parameters

move_to_element_with_offset: moves the mouse by an offset of a specific element on the page, relative to its top-left corner.

  • Calculate the distance you want the cursor to move away from the top-left corner, in order to to have the cursor around the middle of the check box.
  • Find the distance for the move, execute the code with the size attribute of the element before running the complete code. 
    1
    2
    3
    4
    check_box = driver.find_element_by_xpath('Xpath')
    print(check_box.size)

    # output: {'height': 10, 'width': 10}

Handling frames

Issue

  1. Find an element on the webpage, but get errors.
  2. Retry using the Xpath, the class name… , but still getting errors.

Solution

Selenium may not find certain element since it is on another HTML frames. You need to switch to the correct frame before trying to interact with the page again.

  • If you know the name of the frame.

    1
    driver.switch_to.frame('mainIframe')

  • If you know how many frames are there on the page.

    1
    driver.switch_to.frame(0)

  • If you have no idea, find all the frames.

    1
    2
    3
    4
    frames = driver.find_elements_by_tag_name('iframe')
    print(len(frames))
    for frame in frames:
       print(frame.get_attribute('name'))

Switching tabs

Issue

  1. A button automatically opens a new tag
  2. You need to move to the other tab

Solution

Navigating through tabs is similar to navigating frames.

  • Rustic approach: iteration to a different tab

    1
    2
    3
    4
    5
    6
    current_tab = driver.current_window_handle

    all_tabs = driver.window_handles
    for tab in all_tabs:
       if tab!= current_tab:
          driver.switch_to.window(tab)

  • Elegant approach: keep rack of the order of the tabs you opened, keeping indexes.

    1
    driver.switch_to.window(all_tabs[i])

  • Hardcore approach: iteration through all tabs

    1
    2
    3
    all_tabs = driver.window_handles
    for tab in all_tabs:
       driver.switch_to.window(tab)

Traffic warnings

Be aware that you’re making considerably more requests to the website than usual. This is because for each link, you’re opening two or three new tabs.

  • Insert some random pauses in your code in order not to overload the server.
  • Take advantage of a proxy provider to make sure your code will keep running as long as there are pages left to scrape.