What's Google up to?

JackLewis.net ^ | 2/15/05 | Jack Lewis

[Brian Mosely]

The robots.txt file is supposed to be a tool for keeping search engines away from directories on your web site you don't want spidered or indexed. The major search engines all claim the obey them, but warn that there may be a delay between when a robots.txt file is changed and the spider reads, and follows it. All nice and good in print, but the reality is scary.

To cut down on bandwidth use I recently listed two directories containing seldom used message boards in my robots.txt as disallowed. Almost immediately Google began hitting those directories with the fervor of a teen-age hacker.  The index page alone of one received 692 hits in one day from GoogleBots.

Now add that bit of info to the recent story from Reuters about hackers discovering a “wealth” of information regarding things most people don't want on the internet -- at Google.com. (I mentioned it here.) Could Google be using the robots.txt files to intentionally harvest data people want hidden?

Not scary enough for you? Well, add to that the problems Michelle Malkin, Charles Johnson and other bloggers have had getting their blogs listed on Google News. Apparently Google refused to add Conservative blogs, but has no problem adding Liberal blogs such as Wonkette or the Democrat Underground.

Then it should come as no surprise that as I reported earlier today about the political contributions of Google employees.

Let's add it up: Google a blatantly Liberal entity, is found to have tons of sensitive data archived on its site, and seems to be using the robots.txt files to sniff out where that sensitive information is hidden. Why would they want it, and what do they plan to do with it? The last election was pretty dirty and stuff was being dug up left and right. Could Google be building a “dirt chest” of secrets to unload during the next election?

[Texas_Jarhead]

Interesting. The robots file would definitely give one a leg up in finding non-indexed content.

[JoJo Gunn]

The DU is not a blog, but a bog.

[longtermmemmory]

is there a way to spider proof web pages?

How about just having no meta tags?

[faq]

Getting rid of meta tags will not keep you off the search engines. Don't know if this would be an option for you, but you could password protect your web site. That would keep everything off limits to the bots except for your login page.

[Alia]

Thanks for the post and data. I no longer use Google.

[rface]

I have been using altavista for the past 2 days - but I don't thgink it is as easy to use a google......suggestions?

[Bloody Sam Roberts]

If this is true, Google had better watch their asses.

I don't think it would be too big a stretch to imagine someone getting righteously POed at this and bundling up a few little surprises for the spiders to take back to the Google servers.

I know a few people who have the skills. People I'd be afraid to have ticked off at my computers.

[Alia]

There's so many engines out there; depends upon what you want an engine for. Dogpile is good for deep searches, for example.

Google has pluses, but also some minuses -- it misses, IME, things that I get via other engines. I haven't used altavista in a while. I think I'll try it out.

[Psycho_Bunny]

Considering the number of times Google has returned FR pages from the distant past in my searches, I'm surprised that anyone would have a problem getting a conservative site spidered.

I can look up posts there from years past.  Heck, How To Properly Flame is still listed.

[Glock17]

Dog pile looks like bad news. Read this:
http://securityresponse.symantec.com/avcenter/venc/data/spyware.dogpile.html

I'm no computer expert but it scares me.

[Hegemony Cricket]

Oh.

My.

Gosh!

THAT is SO funny! I don't know if I'm just in a really weird mood tonight, or what, but my sides hurt from trying to laugh without waking the rest of the family up. Especially the "Paul & the ISP"...

[Captain Peter Blood]

Well it just came out that the Google founders have given big sums of money to the Democrats and that is enough for me right there.
If there are problems with having conservative web blogs and sites being on the Google search engine then that is a bias that cannot and should not be tolerated.
I use Google only when I can't seem to find what I am looking for on Yahoo and if not there I go to Dogpile, which I think is very underrated.
If Google is going to have that kind of bias in their listings then they are going to have credibility issues down the line.
I wish I could say lets boycott using them but I fear that will not work with something this big.

[Cultural Jihad]

Microsoft unveiled their new search engine the other week to compete with google: www.msn.com

[Mind-numbed Robot]

All your robots are belong to us.

[cynwoody]

I don't know ... I suspect Google has more than enough to do without deliberately trying to queer their search results. This is not to say there are no people trying to tilt Google rankings (along with those of other search engines). But most of them are pornsters or just folks trying to get businesses off the ground. It's called SEO (search engine optimization). It's its own little industry.

I know plenty of smart people who are very good at what they do (mostly computer related) but have a totally cockeyed view of politics. They are mostly focused on what they do, and they have the classic knee-jerk liberal instincts when it comes to politics. It appears Google attracts a lot of such people.

I just ran a quick test. The search argument 005957160 appears on this Free Republic page from 2001. A Google search on that string turns up a valid hit on a Hillary forum, but nothing on FR. However, Clusty, Yahoo, Altavista, and Accoona all turn up the FR page. MSN draws a blank.

I also tried "site:freerepublic.com". Google claims to have about 2,450,000 pages, Clusty 87,381, Yahoo 745,000, Altavista 756,000, and Accoona 156,366.

[Alia]

Thank you for the heads up. I read the link. But doesn't this mean you'd have to install the toolbar for the spyware to be activated?

[Glock17]

Yes, it does.