Robotex does not follow redirects #7
Description
Robotex does not follow redirected robots.txt pages, which can result in pages erroneously appearing to be allowed?. Example:
In https://www.yelp.com/robots.txt:
Disallow: /biz_link
> robotex = Robotex.new "My User Agent"
> robotex.allowed?("https://www.yelp.com/biz_link")
false #correct
> robotex = Robotex.new "My User Agent"
> robotex.allowed?("http://www.yelp.com/biz_link")
true #incorrect