SEOstats\Google::getSerps('search_term', 10) returning same domain multiple times (and different results)

[jamesspittal]

When I do a search on my local computer on my local network using Google Chrome for a specific search term (in incognito mode, searching from Melbourne, Australia), I see search results like:

[1]: http://www.company1.edu.au/internal-page-one
[2]: http://www.company2.edu.au/internal-page-one
[3]: http://www.company3.com.au/internal-page-one
...

When calling SEOstats\Google::getSerps($SearchTerm, 10) from my server (running on AWS in Sydney, Australia), I see similar search results, but rather they look like:

[1]: http://www.company1.edu.au/internal-page-one
[2]: http://www.company1.edu.au/internal-page-two
[3]: http://www.company1.edu.au/folder/internal-page-three
[4]: http://www.company1.edu.au/folder/folder/
[5]: http://www.company1.edu.au/folder/
[6]: http://www.company2.edu.au/folder/folder/internal-page
[7]: https://www.company3.com.au/folder/folder/folder/folder/internal-page
...

In other words, the domain "company1.edu.au" (not the actual domain) appears only once (Position 1) when searched from Google Australia on my computer - but for some reason, SEOstats\Google::getSerps($SearchTerm, 10) returns the same domain (with different internal pages) in the first 5 search results.

I have already checked GOOGLE_TLD and HTTP_HEADER_ACCEPT_LANGUAGE in /vendor/seostats/seostats/SEOstats/Config/DefaultSettings.php (setting them to com.au and en-US,en;q=0.8,en-AU;q=0.6 respectively).

Thanks in advance.

[jamesspittal]

For debugging purposes, I added:

curl_setopt($ch, CURLOPT_VERBOSE, true);

To: /SEOstats/blob/master/SEOstats/Services/Google/Search.php

The output:

*   Trying 74.125.68.94...
* Connected to www.google.com.au (74.125.68.94) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* ALPN, server accepted to use http/1.1
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.google.com,O=Google Inc,L=Mountain View,ST=California,C=US
*       start date: Nov 10 15:45:53 2016 GMT
*       expire date: Feb 02 15:31:00 2017 GMT
*       common name: *.google.com
*       issuer: CN=Google Internet Authority G2,O=Google Inc,C=US
> GET /search?q=Search%20Term&filter=0 HTTP/1.1
Host: www.google.com.au
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: ncr
Accept-Language: en-US,en;q=0.8,en-AU;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

< HTTP/1.1 200 OK
< Content-Type: text/html; charset=UTF-8
< Date: Sun, 04 Dec 2016 11:26:37 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< P3P: CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
< Server: gws
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: NID=91=eaooi2Ogl4kfjt1E1U-SHCUBE2Lw2HolV5S94qXcDtJXKsResPnURwq-0QdafnJ3MUmgl6HNG3KOLjIX7lpdvFzLKyDcikjB3ApxhzsJMWJ6wZdJV0ThJ5f_aMzCkQ0-; expires=Mon, 05-Jun-2017 11:26:37 GMT; path=/; domain=.google.com.au; HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="36,35,34"
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
<
* Connection #0 to host www.google.com.au left intact

When compared with a real search using Internet Explorer (via Charles Web Proxy), the request is:

GET /search?q=search+term&bav=on.2,or.&cad=b&fp=1&biw=1200&bih=855&dpr=1&tch=1&ech=1&psi=hf9DWJLtJIOM8gXzw4OICQ.1480851332487.3

[jamesspittal]

In /seostats/seostats/SEOstats/Services/Google/Search.php, changing:

    protected static function getNextSerp ($start, $query)
    {
        return 0 == $start
            ? sprintf('search?q=%s&filter=0', $query)
            : sprintf('search?q=%s&filter=0&start=%s0', $query, $start);
    }

To:

    protected static function getNextSerp ($start, $query)
    {
        return 0 == $start
            ? sprintf('search?q=%s&filter=1', $query)
            : sprintf('search?q=%s&filter=1&start=%s0', $query, $start);
    }

... fixes this problem. More information is provided here: https://support.google.com/gsa/answer/2672256?hl=en. By including filter=0 in the request, similar search results (such as those from the same domain) are displayed explicitly (rather than hidden). Since they are hidden by default for a new Google search, it seems like a good idea to set this to filter=1 by default.

[jamesspittal]

This has been fixed in my Pull Request #143.

[MichaelFriedberg]

@jamesspittal I was getting the same issue as well. By any chance has your fix for this been working for you? Also any luck with getting Mobile rankings?

[jamesspittal]

@jamesspittal I was getting the same issue as well. By any chance has your fix for this been working for you?

@MichaelFriedberg Yep. Fix in Pull Request #143 has been working (in that the same domain no longer shows up in multiple search results).

There's still some discrepancies I'm seeing in rankings on my local desktop versus using SEOstats\Google::getSerps($SearchTerm, 10) from a Sydney based server. I'm presuming this is mostly due to results personalisation based upon IP Address location (i.e.: I need to use a Melbourne-based IP address/server). Yet to test that though.

Haven't experimented with the mobile rankings yet, but in theory, that should just be a matter of setting User-Agent: to that of a known mobile device such as Mozilla/5.0 (iPhone; CPU iPhone OS 10_0 like Mac OS X) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (which would be the User Agent for an iPhone 7 running Safari 602.1)