Should exit non-zero on DNS or other error

[prenagha]

Was expecting to get a non-zero exit code if reader cannot get the http page, but it always returns zero.

Example shows curl return non-zero on invalid domain name

> curl https://notfound555.cnn.com ; echo $?
curl: (6) Could not resolve host: notfound555.cnn.com
6

But reader seems to always return 0

> reader https://notfound555.cnn.com ; echo $?
Request returned a Syscall Error: lookup notfound555.cnn.com: no such host-> Get
"https://notfound555.cnn.com": dial tcp: lookup notfound555.cnn.com: no such host
0

I would like a reliable way to know if reader worked, so was trying to use the exit code to do that.

[mrusme]

Thank you for reporting!

The bug ultimately comes from an HTTP client library that is being used internally, which differs from the Go default HTTP client in the way it returns errors. In my opinion, the bug is to be solved within the external library, hence I created an issue there. Feel free to subscribe to that issue to get notified about the status of the underlying issue.

However, I added a workaround that might help you for the time being, which is the --no-cycletls parameter. It allows you to switch to the default Go HTTP client, which will return a non-zero status code on error:

▲ reader ./reader --no-cycletls https://notfound555.cnn.com; echo $?                                                       Get "https://notfound555.cnn.com": dial tcp: lookup notfound555.cnn.com on 10.64.0.1:53: no such host
1

Feel free to grab the new version that contains the workaround.

However, it is likely that many sites will block this sort of request, as it doesn't support JA3 fingerprints, hence ymmv.

Note to myself: With issues and PRs apparently piling up in the CycleTLS library and little to no activity happening, submitting a PR for this issue doesn't seem to make a lot of sense and I might need to move to a different library (https://github.com/gospider007/requests) in the future.