Google searches on wiki act dumb

blargg · Post by **blargg** » Sun Jun 06, 2010 6:16 pm

I've noticed that Google searches on the Wiki using site:wiki.nesdev.com often act dumb. For example, when I search for MMC3, none of the 21 hits is the main MMC3 page, even though it has MMC3 in the title. Using intitle:MMC3 gives zero hits (even if I add MMC3 again as a normal search string).

Is there something telling Google to skip documents? At least a week or two ago, I was searching for things on nesdevwiki and Google had lots of hits to the now-nonexistent site. Maybe that has something to do with it, like Google thinks the new site is a spam mirror or something, I dunno. I see no robots.txt, so that wouldn't be it (some bad entry in one or something).

Post by **Banshaku** » Sun Jun 06, 2010 6:32 pm

I don't know enough about media wiki to give an answer but could it be related to the latest spam links that we received about essays that could affect google since it must be a common spam link?

Maybe there is a way to check on google to see how nesdev is affected by this but I don't know about that too. I will see if I can give it a look.

Post by **tepples** » Mon Jun 07, 2010 4:52 am

The robots.txt is a 404, and there don't appear to be any robots directives in meta elements. Nor are internal links using nofollow.

blargg · Post by **blargg** » Mon Jun 07, 2010 6:55 pm

I just noticed that Google is also getting hits within the skins/ directory on the Wiki. I'm thinking that should have an empty index.html in it. Otherwise one gets useless hits, even when restricting via a site: as mentioned in an earlier message.

koitsu · Post by **koitsu** » Tue Jun 08, 2010 2:06 am

blargg wrote:I just noticed that Google is also getting hits within the skins/ directory on the Wiki. I'm thinking that should have an empty index.html in it. Otherwise one gets useless hits, even when restricting via a site: as mentioned in an earlier message.

This is because of the secure MPM we use for Apache. A more appropriate fix would be to place an .htaccess file in /home/ndwiki/www/w which contains:

Code: Select all

Options -Indexes

...which disables automatic directory generation listings for any directory therein which lacks an index.php/index.html/etc. document. The end result is the web client receiving an HTTP 403 Forbidden.

I've put said .htaccess in place; verified as working. It may take a few weeks before Google picks up the changes, as their crawler sometimes takes a while to notice such things.