Feature request

[bettlebrox]

Would it be possible to put the http response in the error output?

For example I get this output:

onboarding/platform-engineer.md

• https://www.amazon.co.uk/Programming-Scala-3rd-Martin-Odersky/dp/0981531687
• https://www.amazon.co.uk/Programming-Scala-Scalability-Functional-Objects/dp/1491949856

but I don't know why there is an issue with these links. I get a 200 from curl:

curl -v -s https://www.amazon.co.uk/Programming-Scala-Scalability-Functional-Objects/dp/1491949856 1> /dev/null
*   Trying 52.84.94.79...
* TCP_NODELAY set
* Connected to www.amazon.co.uk (52.84.94.79) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4192 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=www.amazon.co.uk
*  start date: Jul 13 00:00:00 2020 GMT
*  expire date: Jul  6 12:00:00 2021 GMT
*  subjectAltName: host "www.amazon.co.uk" matched cert's "www.amazon.co.uk"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global CA G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb1cc810a00)
> GET /Programming-Scala-Scalability-Functional-Objects/dp/1491949856 HTTP/2
> Host: www.amazon.co.uk
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< content-type: text/html;charset=UTF-8
< server: Server
< date: Mon, 28 Sep 2020 14:03:58 GMT
< x-amz-rid: SYCQ4K32AX9HJY9XMMB4
< accept-ch: ect,rtt,downlink
< accept-ch-lifetime: 86400
< p3p: policyref="https://www.amazon.co.uk/w3c/p3p.xml",CP="CAO DSP LAW CUR ADM IVAo IVDo CONo OTPo OUR DELi PUBi OTRi BUS PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA HEA PRE LOC GOV OTC "
< cache-control: no-cache, no-transform
< content-encoding: gzip
< x-xss-protection: 1;
< x-content-type-options: nosniff
< vary: Accept-Encoding,User-Agent,Content-Type,Accept-Encoding,X-Amzn-CDN-Cache,X-Amzn-AX-Treatment,User-Agent
< strict-transport-security: max-age=47474747; includeSubDomains; preload
< x-frame-options: SAMEORIGIN
< x-cache: Miss from cloudfront
< via: 1.1 8cedfb7a16a346fb0119eb355ecdaf4c.cloudfront.net (CloudFront)
< x-amz-cf-pop: LHR62-C4
< x-amz-cf-id: jpO9gEM1j--erqIIpYJQyVsvPKUlKQzGRYtNrw6vWitXLvUtEPqaiQ==
< 
{ [109 bytes data]
* Connection #0 to host www.amazon.co.uk left intact

[diegobernardes]

I'll do that. Probably this error happens because it's being accessed by shared servers and Amazon may be blocking thinking that it's a crawler/bot. If this is the case, there is no way around it, we need to add those links to the ignore list.