warbo-utilities: f5221eda4df7062ec3fe6d3b33f49270d5e7669b

     1: From: Chris Warburton
     2: Date: Fri, 16 Jun 2017 10:37:46 +0100
     3: State: resolved
     4: Subject: Use Selenium for browser-based scripts
     5: Message-Id: <a8cd987554b03b93-0-artemis@nixos>
     6: resolution: fixed
     7: 
     8: Some of our scripts interact with Web pages. There are three levels to
     9: this:
    10: 
    11:  - Nice sites: these don't rely on crap like Javascript, so we can grab
    12:    their HTML using wget or curl, and do what we like (e.g. XPath, etc.)
    13:  - Annoying sites: the things we want are only available by executing
    14:    Javascript. This might not be too bad, e.g. if it's an "on page load"
    15:    event, we can run it through e.g. phantomjs, wait a few seconds, and
    16:    dump out the resulting page source.
    17:  - Crap sites: these not only require Javascript, but they only seem to
    18:    work in bloatware browsers like Firefox.
    19: 
    20: For the latter, we're currently launching Firefox on an XVFB display,
    21: using xdotool to perform actions (like opening the Web inspector, typing
    22: into the console and copying the source) and xclip to dump out results
    23: from the clipboard.
    24: 
    25: This is horrible.
    26: 
    27: It's time to bite the bullet and try out Selenium, probably via the
    28: Python library. If it's nice enough, we might be able to use it in place
    29: of phantomjs too.