Scraping recalcitrant web sites with Python & Selenium
6 likes9,081 views
This document discusses using Selenium and Python to scrape websites that are difficult to scrape due to their use of dynamic content, JavaScript, and other obfuscation techniques. It describes how Selenium can automate a browser to interact with such sites as a user would, allowing all content to load properly before scraping. The document provides a general recipe for using Selenium with Python, an example Selenium test case exported to Python, and discusses using PIL and MD5 hashes to scrape an image-based keypad on one such difficult site.
1 of 13
Downloaded 51 times
![Example from Selenium IDE
class Ingdirect2(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait( 30)
self.base_url = "https://www.ingdirect.com.au"
self.verificationErrors = []
def test_ingdirect2(self):
driver = self.driver
But what about
driver.get( self.base_url + "/client/index.aspx")
that dang
driver.switch_to_frame( 'body') # Had to add this keypad? ...
driver.find_element_by_id( "txtCIF").clear()
driver.find_element_by_id( "txtCIF").send_keys( "12345678")
driver.find_element_by_id( "objKeypad_B1").click()
driver.find_element_by_id( "objKeypad_B2").click()
driver.find_element_by_id( "objKeypad_B3").click()
driver.find_element_by_id( "objKeypad_B4").click()
driver.find_element_by_id( "btnLogin").click()
self.assertTrue( self.is_element_present(By.ID, "ctl2_lblBalance"))](https://image.slidesharecdn.com/scrapingrecalcitrantwebsiteswithpythonselenium-120705192849-phpapp02/85/Scraping-recalcitrant-web-sites-with-Python-Selenium-10-320.jpg)
![PIL saves the day
# Get screenshot for extraction of button images
screenshot = driver.get_screenshot_as_base64()
im = Image.open(StringIO.StringIO(base64.decodestring(screenshot)))
table = driver.find_element_by_xpath( '//*[@id="objKeypad_divShowAll"]/table')
all_buttons = table.find_elements_by_tag_name( "input")
# Determine md5sum of each button by cropping based on element positions
for button in all_buttons:
button_image = im.crop(getcropbox(button))
hexid = hashlib.md5(button_image.tostring()).hexdigest()
button_mapping[hexid] = button.get_attribute( "id")
# Now we know which button is which ( based on previous lookup), enter the PIN
for char in self.pin:
driver.find_element_by_id(button_mapping[hex_mapping[char]]).click()
driver.find_element_by_id( "btnLogin").click()
# We're in!!!11one](https://image.slidesharecdn.com/scrapingrecalcitrantwebsiteswithpythonselenium-120705192849-phpapp02/85/Scraping-recalcitrant-web-sites-with-Python-Selenium-11-320.jpg)
Ad
Recommended
Javascript Test Automation Workshop (21.08.2014)
Javascript Test Automation Workshop (21.08.2014)Deutsche Post The document summarizes a workshop on JS test automation using NightwatchJS and DalekJS. It introduces the presenters and provides an overview of the workshop goals, which are to learn rapid test development from scratch using NightwatchJS and DalekJS and compare them on real-world examples from an ING-DiBa Bank site. The document also covers NightwatchJS and DalekJS setup, examples of tests, assertions and commands, and two case studies to be developed using NightwatchJS.
JavaScript + Jenkins = Winning!
JavaScript + Jenkins = Winning!Eric Wendelin This document discusses tools and techniques for automating JavaScript development tasks to improve code quality. It recommends using tools like JSHint for static analysis, PhantomJS for testing, Closure Compiler for optimization, and Gradle for building. It also recommends setting up continuous integration with Jenkins to run tests and metrics on every code change and prevent defects. The document stresses that automation and conventions help enforce best practices and make code more understandable for future engineers.
Making the most of your Test Suite
Making the most of your Test Suiteericholscher 1. The document discusses defining tests, running tests on every commit using continuous integration (CI) tools like Hudson, and monitoring code quality and deployment with tools like coverage.py, Pony Build, and Django Kong.
2. It emphasizes documenting how to run tests, setting up CI to run tests on each commit, and using tools like coverage reports and test data to measure and improve code quality over time.
3. Key recommendations include setting up a CI server, writing tests that run on commit, and using tools to capture test data and monitor code quality and site functionality over time.
Introduction to Selenium and Ruby
Introduction to Selenium and RubyYnon Perek This document provides an introduction to Selenium, an open source tool for automating web application testing. Selenium allows testing of web applications across different browsers without needing to modify the source code. It works by issuing commands to browser drivers to simulate user interactions such as clicking links, filling forms, and verifying page content. The document outlines how to set up and run automated tests using the Selenium Ruby bindings and RSpec testing framework.
探討Web ui自動化測試工具
探討Web ui自動化測試工具政億 林 This document discusses various UI testing tools, including Selenium and Sikuli. It provides overviews of Selenium IDE, RC, and WebDriver, explaining how they work and their capabilities. Live demos are shown of Selenium IDE and Sikuli. The document also discusses why UI testing is important, best practices for testing, and concludes that good automated testing can reduce bugs and improve code quality.
Like a Genie from a Lamp: Headless JavaScript Unit Testing with Jasmine and P...
Like a Genie from a Lamp: Headless JavaScript Unit Testing with Jasmine and P...Rob Friesel The document is a presentation by Rob Friesel on headless JavaScript testing using Jasmine and PhantomJS. It covers the benefits of unit testing, discusses automation challenges, and provides a demo of querying post titles from a webpage. The session concludes with resources for further exploration and an invitation for questions.
Test-Driven JavaScript Development (JavaZone 2010)
Test-Driven JavaScript Development (JavaZone 2010)Christian Johansen This document discusses test-driven development for JavaScript using various testing frameworks like YUI Test, JSpec, and JsTestDriver. It covers how to set up unit testing for JavaScript, challenges like testing asynchronous code and events, and strategies for addressing those challenges using tools like Sinon.js for stubs and mocks. The document also provides examples of testing a chat client application and integrating JavaScript testing into a continuous integration workflow using Hudson.
前端網頁自動測試
前端網頁自動測試 政億 林 This document discusses using Selenium to test websites. It introduces Selenium as a browser-based tool for automated web application testing. It covers the different parts of Selenium including Selenium IDE for recording tests in Firefox, Selenium RC for remote control, and Selenium WebDriver for driving the browser. It provides examples of using Selenium IDE to record a test and Selenium WebDriver in Java to automate testing with FirefoxDriver. It also discusses the benefits of automated testing for reducing bugs and improving code quality.
Web driver training
Web driver trainingDipesh Bhatewara This document provides an overview of WebDriver and its APIs. WebDriver is a browser automation tool that directly controls the browser using native browser capabilities rather than relying on JavaScript. It has object-oriented APIs for Java, C#, Python, Ruby and supports mobile browsers. The document outlines WebDriver's history and architecture, describes common APIs for controlling the browser, finding elements, and interacting with elements. It also provides examples of implicit and explicit waits, handling popups/alerts, and backward compatibility with Selenium 1.
Selenium Automation Using Ruby
Selenium Automation Using RubyKumari Warsha Goel Selenium is a tool for testing web applications. It allows tests to run directly in browsers like Internet Explorer, Firefox, and Chrome across different operating systems. Selenium tests can check browser compatibility and verify application functionality. Tests are written using Selenium's API in languages like Java, Ruby, and Python. The tests automate user interactions like clicking buttons and entering text. Selenium includes features for recording, editing, and debugging tests. It can run tests from outside the browser or directly within it.
Selenium webdriver
Selenium webdriversean_todd Selenium-Webdriver is a tool for writing automated tests of websites. It provides Ruby bindings that allow testing a website's functionality across multiple browsers. It supports browsers like Internet Explorer, Firefox, Safari, Opera, and Chrome. Selenium-Webdriver allows finding elements on a page by class name, css selector, id, link text, name, partial link text, tag name, or xpath. It also supports manipulating the browser window by resizing or maximizing it. Javascript execution and switching between frames are also capabilities. Waiting for elements to load is supported through explicit and implicit waits.
Automated Testing With Watir
Automated Testing With WatirTimothy Fisher Watir is an open-source framework for browser-based automated testing in Ruby, designed for functional and integration testing. It enables the creation of test scripts that leverage Ruby's capabilities and can automate tedious manual processes. The document includes examples of Watir scripts and discusses its features, testimonials, and alternatives like Selenium.
Detecting headless browsers
Detecting headless browsersSergey Shekyan The document discusses headless browsers, particularly PhantomJS, detailing their legitimate uses such as web application testing and SEO, alongside potential malicious applications like botnets and content scraping. It outlines the differences between headless and regular browsers, including detection techniques and risks associated with misuse. The authors also provide guidelines for safely using headless browsers and recommendations for web admins to combat unauthorized access.
Webdriver.io
Webdriver.io LinkMe Srl This document discusses the webdriver.io framework for automated browser testing. The author needed a framework for blackbox testing of a web interface like a user would. Webdriver.io provides JavaScript bindings for Selenium that allow writing tests in a synchronous style using the browser object. Tests can run across multiple browsers and platforms. The framework is easy to set up and use, supports plugins, and allows custom commands. Under the hood, it communicates with Selenium using the WebDriver protocol to automate actual browsers.
Session on Selenium 4 : What’s coming our way? by Hitesh Prajapati
Session on Selenium 4 : What’s coming our way? by Hitesh PrajapatiAgile Testing Alliance The document provides an overview of the new features in Selenium 4, including relative locators, element screenshot capturing, window/tab handling, and integration with Chrome DevTools. It outlines the transition from Selenium 3.x to 4.x, detailing changes in architecture and the implementation of the W3C WebDriver protocol. Additionally, it includes references and resources for further exploration of Selenium 4 capabilities.
Introduction to Protractor
Introduction to ProtractorJie-Wei Wu The document discusses setting up and running end-to-end tests with Protractor. It includes instructions on installing Protractor and related packages, configuring the test framework, writing tests using Mocha and Chai, and tips for running specific tests and managing test data across examples. Code snippets show examples of defining tests to validate page content and links on page load.
Jenkins and Groovy
Jenkins and GroovyKiyotaka Oku This document discusses using Groovy with Jenkins for continuous integration. It describes how to install Groovy on Jenkins, use Groovy scripts for initialization and through the Jenkins CLI, and leverage Groovy features like Grape for dependencies. Examples are given of initializing Jenkins, running scripts against it from the CLI or Groovy shell, and developing Groovy-based plugins. Benefits highlighted include Groovy's Java interoperability and how it enables dynamic Jenkins configurations and automation through scripts.
Zombiejs
ZombiejsВиктор Ткаченко Zombie.js is a lightweight framework for testing client-side JavaScript code in a simulated environment. It allows automation of tasks like verifying functional requirements, load testing, and checking databases. Zombie.js runs quickly, is easy to use and understand, supports promises and mainstream test frameworks, and does not require an IDE. It provides APIs for managing the DOM, forms, browser state, and events to simulate user interactions and test JavaScript code in a headless browser environment.
Introduction To Ruby Watir (Web Application Testing In Ruby)
Introduction To Ruby Watir (Web Application Testing In Ruby)Mindfire Solutions Ruby-Watir is an open-source library used for web application testing in Ruby, allowing users to interact with browsers like Firefox and Chrome, and manipulate web page objects. It provides a sophisticated alternative to professional testing tools with a supportive community and additional resources for learning. The document covers various topics including the setup, usage of IDEs, and basic scripting techniques for effective test automation.
High Performance JavaScript 2011
High Performance JavaScript 2011Nicholas Zakas The document discusses the importance of JavaScript performance in enhancing user experience, highlighting the interaction between JavaScript execution and the UI thread's responsiveness. It outlines various techniques to optimize JavaScript performance, including script loading methods, execution timing, and minimizing long-running scripts. The document also covers common performance pitfalls such as repaint and reflow and emphasizes that effective management of JavaScript execution can significantly improve application responsiveness.
Code ceptioninstallation
Code ceptioninstallationAndrii Lagovskiy Codeception is a modern PHP testing framework inspired by behavior-driven development. It allows writing acceptance, functional, and unit tests. The document provides instructions on installing Codeception via Composer, generating tests, and writing a basic test to check that a site's front page contains the word "Home". It recommends Codeception for testing every project and emphasizes the importance of testing.
An introduction to PhantomJS: A headless browser for automation test.
An introduction to PhantomJS: A headless browser for automation test.BugRaptors PhantomJS is a headless browser designed for automation testing, providing support for various web standards and allowing webpage capturing, manipulation, and file system access via a JavaScript API. It facilitates continuous integration and boasts features like screen capture, network monitoring, and page automation, making it suitable for smoke testing and running multiple test cases simultaneously. However, it has limitations such as pending issues and challenges in replicating user behavior in UI and end-to-end testing, leading to potential inconsistencies in results.
Protractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applicationsLudmila Nesvitiy The document outlines strategies for developing stable end-to-end (e2e) tests for Angular applications using the Protractor framework. It discusses prioritizing test cases, managing performance, stability, timeouts, and the importance of proper configuration while providing examples of code snippets for various testing scenarios. Additionally, it covers practical tips, the use of data providers, and integration with external tools like XLS for data validation.
Session on Launching Selenium Grid and Running tests using docker compose and...
Session on Launching Selenium Grid and Running tests using docker compose and...Agile Testing Alliance Aditya Garg's session demonstrates launching a Selenium Grid and running tests using Docker Compose and Maven, covering setup for Chrome and Firefox debug grids. It includes practical demos for creating and managing Selenium tests in a cloud environment with specified configurations. The session also details integration with Maven for dynamic grid creation and teardown during test execution.
Workshop: Functional testing made easy with PHPUnit & Selenium (phpCE Poland,...
Workshop: Functional testing made easy with PHPUnit & Selenium (phpCE Poland,...Ondřej Machulda The document outlines a workshop led by Ondřej Machulda on functional testing using PHPUnit and Selenium, covering topics such as automating routine tests, the importance of functional tests, and strategies for writing maintainable test cases. Key concepts include the test pyramid, the significance of swift debugging through automated tests, and practical examples using tools like the Steward test runner. Participants learn to implement tests effectively within a continuous integration environment, emphasizing best practices and various integration options in PHP.
Integrační testy - Selenium
Integrační testy - SeleniumKeyup The document discusses the integration testing framework using Selenium, which includes Selenium IDE, WebDriver, and Grid for automating web browser testing. It outlines the components, commands, and capabilities of Selenium IDE for creating tests, as well as the role of WebDriver and Grid in controlling browsers and running tests across multiple machines. Additionally, it touches upon deploying tests in Java and PHP projects through CI tools like Jenkins.
Protractor Tutorial Quality in Agile 2015
Protractor Tutorial Quality in Agile 2015Andrew Eisenberg Andrew Eisenberg gives an overview of UI testing with Protractor. He discusses installing and running Protractor, writing basic tests, more complex test scenarios using page objects and promises, debugging tips, and considerations around when and how to use Protractor for UI testing AngularJS applications. While Protractor is a powerful tool, he notes it has limitations and maintenance costs, and manual testing may be better for unstable UIs and use cases in flux.
淺談 Groovy 與 AWS 雲端應用開發整合
淺談 Groovy 與 AWS 雲端應用開發整合Kyle Lin The document outlines Groovy as a programming language for Java developers, highlighting its features such as minimal learning curve, productivity enhancements, and tools like Gradle and Geb for building and testing applications. It includes instructions for setting up Groovy using the Groovy Environment Manager (GVM) and examples of Groovy syntax, including dynamic typing and metaprogramming. The document also discusses integration with AWS services like S3 and Elastic Beanstalk through Groovy scripts.
Web UI test automation instruments
Web UI test automation instrumentsArtem Nagornyi The document discusses web UI test automation tools, focusing on Selenium WebDriver, Sikuli X, and TestNG. It covers functionalities, features, and examples of automated interactions, user navigation, and working with page objects and tables. Additionally, it mentions the use of Apache Ant and Jenkins for test execution and continuous integration in the testing framework.
Testing web application with Python
Testing web application with PythonJachym Cepicky Testing web mapping applications and services using Python provides concise summaries in 3 sentences or less:
The document discusses using Python for testing web mapping applications and services, including unit, integration, and system testing. It provides an example of using Selenium to test a web map application, writing tests to interact with elements and assert expected behavior. Implementing tests in Python makes the process easy and allows new programmers to get involved, helping to catch bugs and improve software quality.
More Related Content
What's hot (20)
Web driver training
Web driver trainingDipesh Bhatewara This document provides an overview of WebDriver and its APIs. WebDriver is a browser automation tool that directly controls the browser using native browser capabilities rather than relying on JavaScript. It has object-oriented APIs for Java, C#, Python, Ruby and supports mobile browsers. The document outlines WebDriver's history and architecture, describes common APIs for controlling the browser, finding elements, and interacting with elements. It also provides examples of implicit and explicit waits, handling popups/alerts, and backward compatibility with Selenium 1.
Selenium Automation Using Ruby
Selenium Automation Using RubyKumari Warsha Goel Selenium is a tool for testing web applications. It allows tests to run directly in browsers like Internet Explorer, Firefox, and Chrome across different operating systems. Selenium tests can check browser compatibility and verify application functionality. Tests are written using Selenium's API in languages like Java, Ruby, and Python. The tests automate user interactions like clicking buttons and entering text. Selenium includes features for recording, editing, and debugging tests. It can run tests from outside the browser or directly within it.
Selenium webdriver
Selenium webdriversean_todd Selenium-Webdriver is a tool for writing automated tests of websites. It provides Ruby bindings that allow testing a website's functionality across multiple browsers. It supports browsers like Internet Explorer, Firefox, Safari, Opera, and Chrome. Selenium-Webdriver allows finding elements on a page by class name, css selector, id, link text, name, partial link text, tag name, or xpath. It also supports manipulating the browser window by resizing or maximizing it. Javascript execution and switching between frames are also capabilities. Waiting for elements to load is supported through explicit and implicit waits.
Automated Testing With Watir
Automated Testing With WatirTimothy Fisher Watir is an open-source framework for browser-based automated testing in Ruby, designed for functional and integration testing. It enables the creation of test scripts that leverage Ruby's capabilities and can automate tedious manual processes. The document includes examples of Watir scripts and discusses its features, testimonials, and alternatives like Selenium.
Detecting headless browsers
Detecting headless browsersSergey Shekyan The document discusses headless browsers, particularly PhantomJS, detailing their legitimate uses such as web application testing and SEO, alongside potential malicious applications like botnets and content scraping. It outlines the differences between headless and regular browsers, including detection techniques and risks associated with misuse. The authors also provide guidelines for safely using headless browsers and recommendations for web admins to combat unauthorized access.
Webdriver.io
Webdriver.io LinkMe Srl This document discusses the webdriver.io framework for automated browser testing. The author needed a framework for blackbox testing of a web interface like a user would. Webdriver.io provides JavaScript bindings for Selenium that allow writing tests in a synchronous style using the browser object. Tests can run across multiple browsers and platforms. The framework is easy to set up and use, supports plugins, and allows custom commands. Under the hood, it communicates with Selenium using the WebDriver protocol to automate actual browsers.
Session on Selenium 4 : What’s coming our way? by Hitesh Prajapati
Session on Selenium 4 : What’s coming our way? by Hitesh PrajapatiAgile Testing Alliance The document provides an overview of the new features in Selenium 4, including relative locators, element screenshot capturing, window/tab handling, and integration with Chrome DevTools. It outlines the transition from Selenium 3.x to 4.x, detailing changes in architecture and the implementation of the W3C WebDriver protocol. Additionally, it includes references and resources for further exploration of Selenium 4 capabilities.
Introduction to Protractor
Introduction to ProtractorJie-Wei Wu The document discusses setting up and running end-to-end tests with Protractor. It includes instructions on installing Protractor and related packages, configuring the test framework, writing tests using Mocha and Chai, and tips for running specific tests and managing test data across examples. Code snippets show examples of defining tests to validate page content and links on page load.
Jenkins and Groovy
Jenkins and GroovyKiyotaka Oku This document discusses using Groovy with Jenkins for continuous integration. It describes how to install Groovy on Jenkins, use Groovy scripts for initialization and through the Jenkins CLI, and leverage Groovy features like Grape for dependencies. Examples are given of initializing Jenkins, running scripts against it from the CLI or Groovy shell, and developing Groovy-based plugins. Benefits highlighted include Groovy's Java interoperability and how it enables dynamic Jenkins configurations and automation through scripts.
Zombiejs
ZombiejsВиктор Ткаченко Zombie.js is a lightweight framework for testing client-side JavaScript code in a simulated environment. It allows automation of tasks like verifying functional requirements, load testing, and checking databases. Zombie.js runs quickly, is easy to use and understand, supports promises and mainstream test frameworks, and does not require an IDE. It provides APIs for managing the DOM, forms, browser state, and events to simulate user interactions and test JavaScript code in a headless browser environment.
Introduction To Ruby Watir (Web Application Testing In Ruby)
Introduction To Ruby Watir (Web Application Testing In Ruby)Mindfire Solutions Ruby-Watir is an open-source library used for web application testing in Ruby, allowing users to interact with browsers like Firefox and Chrome, and manipulate web page objects. It provides a sophisticated alternative to professional testing tools with a supportive community and additional resources for learning. The document covers various topics including the setup, usage of IDEs, and basic scripting techniques for effective test automation.
High Performance JavaScript 2011
High Performance JavaScript 2011Nicholas Zakas The document discusses the importance of JavaScript performance in enhancing user experience, highlighting the interaction between JavaScript execution and the UI thread's responsiveness. It outlines various techniques to optimize JavaScript performance, including script loading methods, execution timing, and minimizing long-running scripts. The document also covers common performance pitfalls such as repaint and reflow and emphasizes that effective management of JavaScript execution can significantly improve application responsiveness.
Code ceptioninstallation
Code ceptioninstallationAndrii Lagovskiy Codeception is a modern PHP testing framework inspired by behavior-driven development. It allows writing acceptance, functional, and unit tests. The document provides instructions on installing Codeception via Composer, generating tests, and writing a basic test to check that a site's front page contains the word "Home". It recommends Codeception for testing every project and emphasizes the importance of testing.
An introduction to PhantomJS: A headless browser for automation test.
An introduction to PhantomJS: A headless browser for automation test.BugRaptors PhantomJS is a headless browser designed for automation testing, providing support for various web standards and allowing webpage capturing, manipulation, and file system access via a JavaScript API. It facilitates continuous integration and boasts features like screen capture, network monitoring, and page automation, making it suitable for smoke testing and running multiple test cases simultaneously. However, it has limitations such as pending issues and challenges in replicating user behavior in UI and end-to-end testing, leading to potential inconsistencies in results.
Protractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applicationsLudmila Nesvitiy The document outlines strategies for developing stable end-to-end (e2e) tests for Angular applications using the Protractor framework. It discusses prioritizing test cases, managing performance, stability, timeouts, and the importance of proper configuration while providing examples of code snippets for various testing scenarios. Additionally, it covers practical tips, the use of data providers, and integration with external tools like XLS for data validation.
Session on Launching Selenium Grid and Running tests using docker compose and...
Session on Launching Selenium Grid and Running tests using docker compose and...Agile Testing Alliance Aditya Garg's session demonstrates launching a Selenium Grid and running tests using Docker Compose and Maven, covering setup for Chrome and Firefox debug grids. It includes practical demos for creating and managing Selenium tests in a cloud environment with specified configurations. The session also details integration with Maven for dynamic grid creation and teardown during test execution.
Workshop: Functional testing made easy with PHPUnit & Selenium (phpCE Poland,...
Workshop: Functional testing made easy with PHPUnit & Selenium (phpCE Poland,...Ondřej Machulda The document outlines a workshop led by Ondřej Machulda on functional testing using PHPUnit and Selenium, covering topics such as automating routine tests, the importance of functional tests, and strategies for writing maintainable test cases. Key concepts include the test pyramid, the significance of swift debugging through automated tests, and practical examples using tools like the Steward test runner. Participants learn to implement tests effectively within a continuous integration environment, emphasizing best practices and various integration options in PHP.
Integrační testy - Selenium
Integrační testy - SeleniumKeyup The document discusses the integration testing framework using Selenium, which includes Selenium IDE, WebDriver, and Grid for automating web browser testing. It outlines the components, commands, and capabilities of Selenium IDE for creating tests, as well as the role of WebDriver and Grid in controlling browsers and running tests across multiple machines. Additionally, it touches upon deploying tests in Java and PHP projects through CI tools like Jenkins.
Protractor Tutorial Quality in Agile 2015
Protractor Tutorial Quality in Agile 2015Andrew Eisenberg Andrew Eisenberg gives an overview of UI testing with Protractor. He discusses installing and running Protractor, writing basic tests, more complex test scenarios using page objects and promises, debugging tips, and considerations around when and how to use Protractor for UI testing AngularJS applications. While Protractor is a powerful tool, he notes it has limitations and maintenance costs, and manual testing may be better for unstable UIs and use cases in flux.
淺談 Groovy 與 AWS 雲端應用開發整合
淺談 Groovy 與 AWS 雲端應用開發整合Kyle Lin The document outlines Groovy as a programming language for Java developers, highlighting its features such as minimal learning curve, productivity enhancements, and tools like Gradle and Geb for building and testing applications. It includes instructions for setting up Groovy using the Groovy Environment Manager (GVM) and examples of Groovy syntax, including dynamic typing and metaprogramming. The document also discusses integration with AWS services like S3 and Elastic Beanstalk through Groovy scripts.
Session on Launching Selenium Grid and Running tests using docker compose and...
Session on Launching Selenium Grid and Running tests using docker compose and...Agile Testing Alliance
Similar to Scraping recalcitrant web sites with Python & Selenium (20)
Web UI test automation instruments
Web UI test automation instrumentsArtem Nagornyi The document discusses web UI test automation tools, focusing on Selenium WebDriver, Sikuli X, and TestNG. It covers functionalities, features, and examples of automated interactions, user navigation, and working with page objects and tables. Additionally, it mentions the use of Apache Ant and Jenkins for test execution and continuous integration in the testing framework.
Testing web application with Python
Testing web application with PythonJachym Cepicky Testing web mapping applications and services using Python provides concise summaries in 3 sentences or less:
The document discusses using Python for testing web mapping applications and services, including unit, integration, and system testing. It provides an example of using Selenium to test a web map application, writing tests to interact with elements and assert expected behavior. Implementing tests in Python makes the process easy and allows new programmers to get involved, helping to catch bugs and improve software quality.
iOS Automation Primitives
iOS Automation PrimitivesSynack Mikhail Sosonkin discusses iOS automation primitives and tools for blackbox testing of iOS applications without developer access. He outlines static and dynamic analysis techniques using tools like IDAPro, Cycript, and Frida. Sosonkin then introduces CHAOTICMARCH, an open source framework he developed that uses Lua scripts to simulate user interactions, find UI elements, and automate testing of iOS apps through a lightweight injected module. The goal is to build a library of automated testing logic that can be run against any iOS application.
Owasp orlando, april 13, 2016
Owasp orlando, april 13, 2016Mikhail Sosonkin This document discusses iOS automation primitives and tools for automating iOS applications without developer control. It begins by introducing the author and their background in security research and iOS emulation. It then covers jailbreaking iOS devices, using tools like Cycript and Frida for dynamic analysis, and generating touch events to simulate user interactions. The document proposes an open-source framework called CHAOTICMARCH that uses Lua scripts to drive the iOS UI in an automated and repeatable way for blackbox testing purposes. It describes the framework's architecture and provides examples of functions for finding UI elements and representing them to facilitate automation logic.
How to execute Automation Testing using Selenium
How to execute Automation Testing using Seleniumvaluebound The document provides an introduction to Selenium, an automation testing tool developed by Jason Huggins in 2004. It covers essential features such as WebDriver, locators, interacting with elements, handling windows, taking screenshots, and reading data from Excel. Additionally, it introduces TestNG, a testing framework compatible with core Java and .NET, including useful annotations.
Top100summit 谷歌-scott-improve your automated web application testing
Top100summit 谷歌-scott-improve your automated web application testingdrewz lin This document discusses improving automated web application testing. It begins with an introduction and agenda. Some key points:
- Testing at Google expects test code quality to match production code quality. Tests must be maintainable and meet engineering standards.
- The present focuses on Selenium/WebDriver automation best practices and things to avoid. Base classes and locator organization can help improve test robustness and maintainability.
- The future may include more exploratory testing tools like WebTestingExplorer for automated exploratory testing.
Overall the document provides guidance on using Selenium/WebDriver for web application testing in a best practices way, focusing on test code quality, maintainability, and anticipating future changes. It emphasizes organizing tests through
Automating Django Functional Tests Using Selenium on Cloud
Automating Django Functional Tests Using Selenium on CloudJonghyun Park This document discusses automating Django functional tests using Selenium on AWS EC2. It begins with an overview of testing login functionality on Django using Selenium. It then covers running Selenium tests on the AWS cloud to allow automated testing. Tips are provided for writing efficient functional tests, including using explicit waits, page object models, and test organization techniques. The document aims to help Python beginner-intermediates implement automated functional testing.
Nightwatch 101 - Salvador Molina
Nightwatch 101 - Salvador MolinaSalvador Molina (Slv_) The document provides an overview of Nightwatch.js, a testing framework for Node.js, highlighting its core features such as built-in test running capabilities, documentation, and reporting. It discusses the flexibility of command types and the concept of page objects for encapsulation, as well as practical examples of usage, including syntax support for CSS and XPath selectors. Additionally, it emphasizes advanced functionalities like parallel test execution and CI integration.
Javascript Everywhere
Javascript EverywherePascal Rettig This document covers various aspects of JavaScript, including debugging, optimizing, local and offline storage, and server-side JavaScript with web sockets. It presents tools, frameworks, and best practices for debugging (like Firebug), automated testing (e.g., QUnit, Jasmine), performance optimization techniques, and the use of local storage and session storage. It also discusses server-side JavaScript with Node.js and how to implement real-time messaging using web sockets.
UI Testing Best Practices - An Expected Journey
UI Testing Best Practices - An Expected JourneyOren Farhi The document outlines the importance of UI testing in software development, discussing its purpose, best practices, and tools like Jasmine and Karma. It emphasizes the necessity of automating testing processes to improve efficiency and code quality, particularly in Angular applications. The document provides examples of testing strategies, including unit tests and end-to-end tests, along with practical application through a demo app called Echoes Player.
Server Side JavaScript - You ain't seen nothing yet
Server Side JavaScript - You ain't seen nothing yetTom Croucher This document discusses using JavaScript on the server side with Node.js and the YUI framework. It begins by explaining why server-side JavaScript is useful and discusses JavaScript runtimes like V8, SpiderMonkey, and Rhino. It then covers Node.js, CommonJS frameworks, and how to use YUI modules on the server by enabling YUI's module loader. Examples are provided for accessing remote data, rendering HTML on the server, and implementing progressive enhancement.
End-to-end testing with geb
End-to-end testing with gebJesús L. Domínguez Muriel The document discusses end-to-end testing in software development, highlighting its importance for maintaining large applications over time. It provides a narrative about a library that evolved into a complex software with numerous dependencies, detailing the journey of implementing tests using tools like Selenium and Geb. Ultimately, it encourages the use of end-to-end tests alongside Groovy and Geb for robust testing practices.
Introduction to jQuery
Introduction to jQueryAlek Davis This document provides an introduction to jQuery, including:
- A brief overview of what jQuery is and its key facts
- Details on learning jQuery through websites, tutorials, and books
- Explanations of jQuery syntax, selectors, animations, common operations, and events
- Descriptions of related tools like jQuery UI, plugins, and browser developer tools
Writing automation tests with python selenium behave pageobjects
Writing automation tests with python selenium behave pageobjectsLeticia Rss This document discusses writing automated tests using Python and Selenium. It covers best practices like using the Behavior Driven Development (BDD) and page object patterns. Gherkin is introduced as a way to describe test scenarios in natural language. Selenium is presented as a tool for browser automation, and Cucumber, Behave, and Godog are compared as tools for implementing tests from Gherkin scenarios. The document also provides tips on project structure, running tests, and taking screenshots.
OpenCms Days 2014 - User Generated Content in OpenCms 9.5
OpenCms Days 2014 - User Generated Content in OpenCms 9.5Alkacon Software GmbH & Co. KG The document discusses implementing user-generated content (UGC) in OpenCMS 9.5, emphasizing the new UGC API that automates XML content generation with reduced server-side coding. It provides an overview of creating and configuring UGC forms, including JavaScript interactions for form validation and submission, alongside best practices for session management and error handling. Additionally, the document highlights security considerations and configuration options to control guest content creation and editing capabilities.
Testing ASP.NET - Progressive.NET
Testing ASP.NET - Progressive.NETBen Hall This document discusses automated acceptance testing for ASP.NET applications. It begins with a demonstration of using Selenium WebDriver to automate a Google search. The document then covers topics like the testing pipeline, challenges with real-world testing, focusing tests on solving specific pain points, and designing UI for testability. It emphasizes that UI tests should validate scenarios rather than individual actions and should focus on the most critical tests. The document also discusses tools, techniques, and strategies for testing databases, legacy systems, and more.
JSMVCOMFG - To sternly look at JavaScript MVC and Templating Frameworks
JSMVCOMFG - To sternly look at JavaScript MVC and Templating FrameworksMario Heiderich The document discusses JavaScript MVC and templating frameworks and security issues found during penetration testing. Several frameworks were found to execute arbitrary JavaScript from markup in dangerous ways due to overuse of eval-like functions and lack of separation between code and content. This could lead to bypassing of content security policies. Metrics are proposed to evaluate frameworks on security practices like sandboxing and preventing injection into templates. While challenges exist, following best practices like strict separation of code and content could help frameworks improve security.
StHack 2014 - Mario "@0x6D6172696F" Heiderich - JSMVCOMFG
StHack 2014 - Mario "@0x6D6172696F" Heiderich - JSMVCOMFGStHack JSMVCOMFG is a presentation by Mario Heiderich on looking at JavaScript MVC and templating frameworks. These frameworks are becoming popular and complex, often maintained by large corporations. They extend and abstract the DOM using custom elements, script templates, and data bindings. However, they also introduce security risks by allowing JavaScript execution from data attributes and encoded templates, bypassing XSS filters. Frameworks need to implement better sandboxing and separation of code and content to address these issues.
Automation - web testing with selenium
Automation - web testing with seleniumTzirla Rozental The document provides a comprehensive guide on QA automation tests using Selenium, covering the basics of automation, project setup, coding, and design. It includes instructions on installing necessary tools, creating a project using Maven, and writing automation tests with various Selenium selectors. Additionally, it discusses best practices for handling test failures and includes details about different testing frameworks and methods for element interaction.
Catch a spider monkey
Catch a spider monkeyChengHui Weng The document provides an overview of the SpiderMonkey JavaScript engine, including its codebase reference, debugging techniques, and self-hosting functions. It discusses how to build and use the engine effectively, demonstrating features such as the disassembly of functions and the implementation of ES6 array methods. Additionally, it addresses memory safety and performance considerations while debugging JavaScript via GDB.
Ad
More from Roger Barnes (6)
The life of a web request - techniques for measuring and improving Django app...
The life of a web request - techniques for measuring and improving Django app...Roger Barnes The document presents a talk on measuring and improving Django application performance, highlighting the significance of web application performance on user engagement, bounce rate, conversion rates, and revenue. It discusses various front-end and back-end performance measurement tools and strategies, such as caching, resource optimization, and deferred processing. The conclusion emphasizes the importance of continuous performance measurement and staying updated with technological advancements.
Building data flows with Celery and SQLAlchemy
Building data flows with Celery and SQLAlchemyRoger Barnes The document presents a talk by Roger Barnes on building data flows utilizing Celery and SQLAlchemy, highlighting their application in data warehousing. It covers the importance of timely and accurate reporting, introduces key concepts of SQLAlchemy and Celery for processing and managing data, and illustrates creating and managing data workflows. Additionally, it discusses how to structure and monitor tasks to enhance data processing efficiency.
Introduction to SQL Alchemy - SyPy June 2013
Introduction to SQL Alchemy - SyPy June 2013Roger Barnes The document summarizes a presentation on SQLAlchemy by Roger Barnes at the PyCon conference in June 2013, highlighting its features as a Python SQL toolkit and object-relational mapper. It compares SQLAlchemy with Django's ORM, noting that while both have their merits, SQLAlchemy is more flexible and suitable for a broader range of applications. The presentation also outlines the core structure and functionality of SQLAlchemy, including session management and mapping between domain objects and database rows.
Poker, packets, pipes and Python
Poker, packets, pipes and PythonRoger Barnes The document discusses using Python to analyze online poker gameplay data in real-time. It describes capturing packet data from an Android device or router, processing the data with Python using pipes and stdin, and analyzing the poker hands with a library. IPython notebooks are used for visualization and documentation. The example code connects an Ngrep process on a router to a Python script via pipes to parse live poker game updates and rank hands.
Towards Continuous Deployment with Django
Towards Continuous Deployment with DjangoRoger Barnes The document presents a comprehensive guide to implementing continuous deployment using Django, focusing on strategies and tools such as Vagrant, Fabric, and Jenkins. Key concepts include automating the build and deployment process, leveraging virtual environments for consistent development, and utilizing monitoring tools for performance measurement. The talk emphasizes the importance of agile methodologies, testing, and version control in achieving a streamlined continuous delivery pipeline.
Intro to Pinax: Kickstarting Your Django Apps
Intro to Pinax: Kickstarting Your Django AppsRoger Barnes This document provides an overview and introduction to Pinax, an open source framework built on Django that aims to provide common functionality needed for web applications out of the box. It discusses what Pinax is, when it should be used, its key features like reusable apps and starter projects, alternatives to Pinax, and tips for development using Pinax like testing, deployment and source control best practices.
Ad
Recently uploaded (20)
WebdriverIO & JavaScript: The Perfect Duo for Web Automation
WebdriverIO & JavaScript: The Perfect Duo for Web Automationdigitaljignect In today’s dynamic digital landscape, ensuring the quality and dependability of web applications is essential. While Selenium has been a longstanding solution for automating browser tasks, the integration of WebdriverIO (WDIO) with Selenium and JavaScript marks a significant advancement in automation testing. WDIO enhances the testing process by offering a robust interface that improves test creation, execution, and management. This amalgamation capitalizes on the strengths of both tools, leveraging Selenium’s broad browser support and WDIO’s modern, efficient approach to test automation. As automation testing becomes increasingly vital for faster development cycles and superior software releases, WDIO emerges as a versatile framework, particularly potent when paired with JavaScript, making it a preferred choice for contemporary testing teams.
Techniques for Automatic Device Identification and Network Assignment.pdf
Techniques for Automatic Device Identification and Network Assignment.pdfPriyanka Aash Techniques for Automatic Device Identification and Network Assignment
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdfEarley Information Science As AI reshapes expectations in retail and B2B commerce, organizations are recognizing a critical reality: meaningful AI outcomes depend on well-structured, adaptable infrastructure. In this session, Seth Earley is joined by Phil Ryan - AI strategist, search technologist, and founder of Glass Leopard Technologies - for a candid conversation on what it truly means to engineer systems for scale, agility, and intelligence.
Phil draws on more than two decades of experience leading search and AI initiatives for enterprise organizations. Together, he and Seth explore the challenges businesses face when legacy architectures limit personalization, agility, and real-time decisioning - and what needs to change to support agentic technologies and next-best-action capabilities.
Key themes from the webinar include:
Composability as a prerequisite for AI - Why modular, loosely coupled systems are essential for adapting to rapid innovation and evolving business needs
Search and relevance as foundational to AI - How techniques honed-in enterprise search have laid the groundwork for more responsive and intelligent customer experiences
From MDM and CDP to agentic systems - How data platforms are evolving to support richer customer context and dynamic orchestration
Engineering for business alignment - Why successful AI programs require architectural decisions grounded in measurable outcomes
The conversation is practical and forward-looking, connecting deep technical understanding with real-world business needs. Whether you’re modernizing your commerce stack or exploring how AI can enhance product discovery, personalization, or customer journeys, this session provides a clear-eyed view of the capabilities, constraints, and priorities that matter most.
AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc AI VIDEO MAGAZINE - r/aivideo community newsletter – Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model LibraryPetraVukmirovic Threat Model Library Launch at OWASP Barcelona 2025
https://owasp.org/www-project-threat-model-library/
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025pcprocore 👉𝗡𝗼𝘁𝗲:𝗖𝗼𝗽𝘆 𝗹𝗶𝗻𝗸 & 𝗽𝗮𝘀𝘁𝗲 𝗶𝗻𝘁𝗼 𝗚𝗼𝗼𝗴𝗹𝗲 𝗻𝗲𝘄 𝘁𝗮𝗯> https://pcprocore.com/ 👈◀
CapCut Pro Crack is a powerful tool that has taken the digital world by storm, offering users a fully unlocked experience that unleashes their creativity. With its user-friendly interface and advanced features, it’s no wonder why aspiring videographers are turning to this software for their projects.
2025_06_18 - OpenMetadata Community Meeting.pdf
2025_06_18 - OpenMetadata Community Meeting.pdfOpenMetadata The community meetup was held Wednesday June 18, 2025 @ 9:00 AM PST.
Catch the next OpenMetadata Community Meetup @ https://www.meetup.com/openmetadata-meetup-group/
In this month's OpenMetadata Community Meetup, "Enforcing Quality & SLAs with OpenMetadata Data Contracts," we covered data contracts, why they matter, and how to implement them in OpenMetadata to increase the quality of your data assets!
Agenda Highlights:
👋 Introducing Data Contracts: An agreement between data producers and consumers
📝 Data Contracts key components: Understanding a contract and its purpose
🧑🎨 Writing your first contract: How to create your own contracts in OpenMetadata
🦾 An OpenMetadata MCP Server update!
➕ And More!
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdfPriyanka Aash GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...Fwdays We will explore how e-commerce projects prepare for the busiest time of the year, which key aspects to focus on, and what to expect. We’ll share our experience in setting up auto-scaling, load balancing, and discuss the loads that Silpo handles, as well as the solutions that help us navigate this season without failures.
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdfyosra Saidani Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and SwecoSafe Software Managing airport and airspace data is no small task, especially when you’re expected to deliver it in AIXM format without spending a fortune on specialized tools. But what if there was a smarter, more affordable way?
Join us for a behind-the-scenes look at how Sweco partnered with Swedavia, the Swedish airport operator, to solve this challenge using FME and Esri.
Learn how they built automated workflows to manage periodic updates, merge airspace data, and support data extracts – all while meeting strict government reporting requirements to the Civil Aviation Administration of Sweden.
Even better? Swedavia built custom services and applications that use the FME Flow REST API to trigger jobs and retrieve results – streamlining tasks like securing the quality of new surveyor data, creating permdelta and baseline representations in the AIS schema, and generating AIXM extracts from their AIS data.
To conclude, FME expert Dean Hintz will walk through a GeoBorders reading workflow and highlight recent enhancements to FME’s AIXM (Aeronautical Information Exchange Model) processing and interpretation capabilities.
Discover how airports like Swedavia are harnessing the power of FME to simplify aviation data management, and how you can too.
The Future of Product Management in AI ERA.pdf
The Future of Product Management in AI ERA.pdfAlyona Owens Hi, I’m Aly Owens, I have a special pleasure to stand here as over a decade ago I graduated from CityU as an international student with an MBA program. I enjoyed the diversity of the school, ability to work and study, the network that came with being here, and of course the price tag for students here has always been more affordable than most around.
Since then I have worked for major corporations like T-Mobile and Microsoft and many more, and I have founded a startup. I've also been teaching product management to ensure my students save time and money to get to the same level as me faster avoiding popular mistakes. Today as I’ve transitioned to teaching and focusing on the startup, I hear everybody being concerned about Ai stealing their jobs… We’ll talk about it shortly.
But before that, I want to take you back to 1997. One of my favorite movies is “Fifth Element”. It wowed me with futuristic predictions when I was a kid and I’m impressed by the number of these predictions that have already come true. Self-driving cars, video calls and smart TV, personalized ads and identity scanning. Sci-fi movies and books gave us many ideas and some are being implemented as we speak. But we often get ahead of ourselves:
Flying cars,Colonized planets, Human-like AI: not yet, Time travel, Mind-machine neural interfaces for everyone: Only in experimental stages (e.g. Neuralink).
Cyberpunk dystopias: Some vibes (neon signs + inequality + surveillance), but not total dystopia (thankfully).
On the bright side, we predict that the working hours should drop as Ai becomes our helper and there shouldn’t be a need to work 8 hours/day. Nobody knows for sure but we can require that from legislation. Instead of waiting to see what the government and billionaires come up with, I say we should design our own future.
So, we as humans, when we don’t know something - fear takes over. The same thing happened during the industrial revolution. In the Industrial Era, machines didn’t steal jobs—they transformed them but people were scared about their jobs. The AI era is making similar changes except it feels like robots will take the center stage instead of a human. First off, even when it comes to the hottest space in the military - drones, Ai does a fraction of work. AI algorithms enable real-time decision-making, obstacle avoidance, and mission optimization making drones far more autonomous and capable than traditional remote-controlled aircraft. Key technologies include computer vision for object detection, GPS-enhanced navigation, and neural networks for learning and adaptation. But guess what? There are only 2 companies right now that utilize Ai in drones to make autonomous decisions - Skydio and DJI.
AI Agents and FME: A How-to Guide on Generating Synthetic Metadata
AI Agents and FME: A How-to Guide on Generating Synthetic MetadataSafe Software In the world of AI agents, semantics is king. Good metadata is thus essential in an organization's AI readiness checklist. But how do we keep up with the massive influx of new data? In this talk we go over the tips and tricks in generating synthetic metadata for the consumption of human users and AI agents alike.
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
Using the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software The SQLExecutor is one of FME’s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation –Validating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.
Mastering AI Workflows with FME by Mark Döring
Mastering AI Workflows with FME by Mark DöringSafe Software Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FME’s powerful tools for training, optimization, and seamless integration
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's SeatSafe Software Finding a specific car online can be a time-consuming task, especially when checking multiple dealer websites. A few years ago, I faced this exact problem while searching for a particular vehicle in New Zealand. The local classified platform, Trade Me (similar to eBay), wasn’t yielding any results, so I expanded my search to second-hand dealer sites—only to realise that periodically checking each one was going to be tedious. That’s when I noticed something interesting: many of these websites used the same platform to manage their inventories. Recognising this, I reverse-engineered the platform’s structure and built an FME workspace that automated the search process for me. By integrating API calls and setting up periodic checks, I received real-time email alerts when matching cars were listed. In this presentation, I’ll walk through how I used FME to save hours of manual searching by creating a custom car-finding automation system. While FME can’t buy a car for you—yet—it can certainly help you find the one you’re after!
Scraping recalcitrant web sites with Python & Selenium
- 1. Scraping recalcitrant web sites with Python & Selenium Roger Barnes SyPy July 2012
- 3. Some sites suck - "for your own good" For security reasons, each button is an image, dynamically generated by a hash wrapped in a mess of javascript, randomly placed
- 4. ...but they work in a web browser! Let's use the web browser to scrape them
- 5. Enter Selenium Selenium automates browsers That's it
- 6. Selenium can... ● navigate (windows, frames, links) ● find elements and parse attributes ● interact and trigger events (click, type, ...) ● capture screenshots ● run javascript ● let the browser take care of the hard stuff (cookies, javascript, sessions, profiles, DOM) Comes with various components and bindings ... including python
- 7. General Recipe Ingredients: ● firefox (or chrome) ● firebug (or chrome dev tools) ● Selenium IDE ○ record a session, write less code ● python and its batteries ● python-selenium ● xvfb and pyvirtualdisplay (optional) ● other libraries to taste ○ eg image manipulation, database access, DOM parsing, OCR
- 8. General Recipe Method: ● Install requirements (apt-get, pip etc) ○ sudo apt-get install xvfb firefox ○ pip install selenium pyvirtualdisplay ● Start up Firefox and Selenium IDE ● Record a "test" run through site ○ Add in some assertions along the way ● Export test as Python script ● Hack from there ○ Loops ○ Image/data extraction ○ Wrangling data into a database
- 10. Example from Selenium IDE class Ingdirect2(unittest.TestCase): def setUp(self): self.driver = webdriver.Firefox() self.driver.implicitly_wait( 30) self.base_url = "https://www.ingdirect.com.au" self.verificationErrors = [] def test_ingdirect2(self): driver = self.driver But what about driver.get( self.base_url + "/client/index.aspx") that dang driver.switch_to_frame( 'body') # Had to add this keypad? ... driver.find_element_by_id( "txtCIF").clear() driver.find_element_by_id( "txtCIF").send_keys( "12345678") driver.find_element_by_id( "objKeypad_B1").click() driver.find_element_by_id( "objKeypad_B2").click() driver.find_element_by_id( "objKeypad_B3").click() driver.find_element_by_id( "objKeypad_B4").click() driver.find_element_by_id( "btnLogin").click() self.assertTrue( self.is_element_present(By.ID, "ctl2_lblBalance"))
- 11. PIL saves the day # Get screenshot for extraction of button images screenshot = driver.get_screenshot_as_base64() im = Image.open(StringIO.StringIO(base64.decodestring(screenshot))) table = driver.find_element_by_xpath( '//*[@id="objKeypad_divShowAll"]/table') all_buttons = table.find_elements_by_tag_name( "input") # Determine md5sum of each button by cropping based on element positions for button in all_buttons: button_image = im.crop(getcropbox(button)) hexid = hashlib.md5(button_image.tostring()).hexdigest() button_mapping[hexid] = button.get_attribute( "id") # Now we know which button is which ( based on previous lookup), enter the PIN for char in self.pin: driver.find_element_by_id(button_mapping[hex_mapping[char]]).click() driver.find_element_by_id( "btnLogin").click() # We're in!!!11one
- 12. But why do all this? It's my data! ... and I'll graph if i want to * Actual results may vary. Graph indicates open inodes, not high-roller gambling problem
- 13. That's all folks Slides ● http://bit.ly/scrapium Code ● https://gist.github.com/3015852 Me ● https://twitter.com/mindsocket ● https://github.com/mindsocket ● [email protected]