Google-it fails when Google returns different HTML formats #95
Description
Environment
- node: v16.15.1
- npm: 8.11.0
- google-it: 1.6.3
Issue
I am suffering from a problem where using google-it fails about 50% of the time.
After investigation, I assume the following results are the root cause.
Look at the following HTML pattern
success:
https://github.com/ryudenx/google-it_fail/blob/master/takeda_success.html
fail:
https://github.com/ryudenx/google-it_fail/blob/master/takeda_fail.html
These have different HTML format/design.
Despite the same URL, same User-Agent, same IP, Google seems to return different HTML.
If possible, could you please fix this problem?
Or could you support the failing HTML format?
Reproduce
Reproduced code:
const { setTimeout } = require('timers/promises');
const googleIt = require('google-it')
async function main() {
while (true) {
googleIt({ 'query': "takeda", 'no-display': true, 'limit': 10 }).then(results => {
//check
if (results.length > 1) {
console.log("success")
} else {
console.log("fail")
}
}).catch(e => {
console.log(e)
})
await setTimeout(5000);
}
}
main()
log (example):
$ node ./test.js
success
success
success
fail
success
fail
fail
Thank you for your cooperation.