Web Profiler

Tools for gathering statistics about contents and performance of Web sites. There are two parts:

1. Python "loader" classes that load Web pages (using various backends, like PhantomJS, Chrome, and Firefox) and produce summary statistics, HAR files, and/or screenshots.

2. Scripts that use these loaders to gather particular statistics (e.g., compare the load times of the HTTP and HTTPS versions of a site).

Loaders

The loaders are Python classes that encapsulate the logic for loading Web pages and saving statistics. Each one implements a common interface (specified by the Loader superclass in loader.py) and currently supports the following backends:

• PhantomJS (PhantomJSLoader in phantomjs_loader.py)
• Chrome (ChromeLoader in chrome_loader.py)
• Firefox (FirefoxLoader in firefox_loader.py)
• Python Requests (PythonRequestsLoader in pythonrequests_loader.py)
• Curl (CurlLoader in curl_loader.py)
• NodeJS (NodeJsLoader in nodejs_loader.py)

API documentation available here.

Dependencies:

• PhantomJS

  Needed by the PhantomJSLoader.

  If the loader can't find PhantomJS, try harding-coding the path to PhantomJS at the top of phantomjs_loader.py.

• Xvfb

  Needed to run Firefox or Chrome in headless mode.

• chrome-har-capturer

  Needed by the ChromeLoader.

  Set path to binary at top of chrome_loader.py. To install globally:

    sudo npm install -g chrome-har-capturer
  

  To record HTTP versions, you currently need to use this fork. To install globally:

    git clone git@github.com:dtnaylor/chrome-har-capturer.git
    cd chrome-har-capturer
    sudo npm install -g .
  

• node.js

  Needed by the ChromeLoader. (On Ubuntu, be sure to install nodejs package and not node.)

• Python Requests

  Used to test the availability of sites of HTTP or HTTPS and by the PythonRequestsLoader.

• node-http2

  Needed by the NodeJsLoader and ZombieJsLoader. Location of the module must be hardcoded in webloader/nodejs_loader.py and webloader/zombiejs_loader.py. z