The Sorry State of the Web

Being a programmer, I have a bunch of scripts to automate everyday tasks. Some of those scripts provide commandline interfaces for various Web sites, which I use both directly and from within other scripts.

Recently, one of these sites has changed its pages such that the page’s markup doesn’t occur in the result of a HTTP GET of the page’s URL. Instead the resulting markup contains Javascript, which inserts new elements into the page’s DOM when executed in a browser (e.g. Firefox).

To try and fix my scripts, I tried to replace their previous use of wget, curl, xidel, etc. with PhantomJS, which I’ve used in other projects before.

Unfortunately, even with timeouts, and spoofing the user agent and window size, the elements weren’t being fetched and inserted into the DOM, which I could verify by dumping out screenshots.

Since it works in Firefox, but not in PhantomJS, I tried the same script using SlimerJS, which is backed by the same Gecko rendering engine as Firefox, but that didn’t work either.

Since the only working browser I could find was Firefox, I tried using Selenium, but that seems to be very flaky regarding versions, and I couldn’t get it to run (Firefox would load, but Selenium wouldn’t connect to it).

I made a brief attempt at using SikuliX and xautomation, I tried using Firefox’s “Save Page As” feature, but that only stores the original HTML without the modifications.

Eventually, I cobbled together a script based on the following, which seems to work:

• Use xvfb-run to launch a second script, where all the following take place
• Run firefox in the background, using timeout to kill the process after 30 seconds. Firefox itself is launched in “safe mode” with the desired URL.
• Use xdotool to send keyboard events which close the “safe mode” information message, open the developer console and wait long enough for the DOM to be updated.
• Again, using xdotool, type a jQuery-based expression into the console, to extract the desired text. This is fed into a call to prompt, which causes the text to appear, selected, in an overlay.
• One final call to xdotool presses ctrl+c to copy the text to the X clipboard.
• xsel is used to paste the text to stdout, and echo adds a newline.
• We kill the background firefox process using its process ID, stored from when it was invoked.

Needless to say, this is a very convoluted and fragile approach, and it speaks to the sorry state of the Web as a medium for the exchange of hyperlinked documents between a proliferation of applications on disparate machines.