Posted on 11/11/2004 1:35:03 PM PST by mhking
2004-11-11
|
 |
| Is MSN Crawling Google? |
 |
My Theory On This Mysterious Microsoft Crawler
The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.
First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.
Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.
So Microsoft's Screen Scraping?
Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.
Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.
Jason's article first appeared on his blog MarketingShift.com.
Bill Gates is no Republican, but Microsoft gave no political contributions and had no Washington lobbiests until the government of Bill Clinton sued them.
msn for "freerepublic" Results 1-15 of about 16,239 containing freerepublic
google for "freerepublic" Results 1-10 of about 2,970,000 for freerepublic
msn Results 1-15 of about 2,142,700 containing "George W. Bush"
google Results 1-10 of about 12,700,000 for "George W. Bush"
You're a bit new to be calling established members of this forum trolls.
Politics isn't EVERYTHING, you know. I eat Ben and Jerry's because I like the ice cream. I use Google because it's an awesome search engine.
Walter Mossberg is a failure as a person and a columnist. He has no more technological knowledge than my dad (which isn't considerably bad, but not quite enough to understand the cutting edge).
I'd be very, VERY surprised if Microsoft took down Google. Heck, I'd be very surprised if they gave them anything of a fight. M$ won the OS war because they were first and best to market. No matter what innovation they have in their new search engine (WinFS was pretty impressive, though - then it got pulled from Longhorn - blech), Google's already established in the market.
Google will win, and M$'s search engine will have disappeared in a year and folded within three, and you can quote me on that.
And, oh yes, I'm willing to bet most universities across the country have "more and better Ph.D's" than FreeRepublic, but are you about to blindly say they are more politically apt than us?
Of course not. Argument defeated.
"I'd be very, VERY surprised if Microsoft took down Google. Heck, I'd be very surprised if they gave them anything of a fight. M$ won the OS war because they were first and best to market. No matter what innovation they have in their new search engine (WinFS was pretty impressive, though - then it got pulled from Longhorn - blech), Google's already established in the market."
Need I say more to respond to this than "Netscape"?
In 1982 I saw a machine that had most of the components of Windoze 3.1
A "Lisa" by Apple/McIntosh
Ah, then you missed the point.
The point I was making was: you do not need to have a Ph.D to understand the material, and having a Ph.D doesn't always mean you do.
The best search tool is still the meta-search application Copernic. (www.copernic.com)
It comes bundled with other nifty tools, like a summarizer that will produce an accurate summary of any web page, a tracker tool to follow changes on specific websites, and an application to search your own files on your desktop, without uploading details of all your documents, like Google's desktop search tool requires you to do.
It costs $100 or so, but it is absolutely worth it.
mhking is a well-known poster around here. You, on the other hand, have been here for little over a month and are accusing him of being a troll because he dared suggest that Microsoft (which isn't exactly right-leaning company) may be doing something slightly underhanded. If I had to pick a troll here, it would be you.
Personal insults get us nowhere around here.
He posts, you decide/discuss. Maybe you should back off the accusatory tone vs mhking, one of the most respected members of this forum.
mhking just posted the piece for debate, not because he wrote it.
Are you a shill for Gates, or just an anti-Google troll out to spread anti-Google FUD, sugah? 
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.