Skip to comments.
Microsoft Crawling Google Results For New Search Engine?
WebProNews ^
| 11.11.04
Posted on 11/11/2004 1:35:03 PM PST by mhking
Microsoft Crawling Google Results For New Search Engine?
Jason Dowdell | Contributing Writer
2004-11-11
I was questioned today by a developer who was watching a particular IP address scan his site. The IP was 65.54.188.86 and is registered to Microsoft Corp. located at One Microsoft Way, Redmond, Washington 98052. This visitor was not sending the normal header information associated with a crawler to the web server such as an http robot name or identifying info or even a browser name.
 |
| Is MSN Crawling Google? |
 |
Is Microsoft "using" Google's search results to populate their index? Discuss Microsoft's behavior at WebProWorld.
The behavior it demonstrated made it look like a crawler, especially since it was spidering urls that were no longer in existence (search engine spiders crawl site segments at regular intervals and often come back when an initial crawl left urls uncrawled) and doing so at the rate of 1 page every 3 - 5 seconds. The visitor started their visit at 7:37 am and was still on the site at 12:00 pm.
Correction, the data was there after all, here's the crawler info... msnbot/0.3 (+http://search.msn.com/msnbot.htm)
Here's the kicker
So now you're saying, so what, big deal. But this really is a big deal. It's a big deal not only because the urls this visitor was making requests to don't exist any longer but because the only place these urls can be found is in Google's search results using site:www.sitename.com. A similar query on MSN Search doesn't show the urls at all, even on the beta version of their new Microsoft search engine. But then within just hours of the visitors exit from the site the new same search at Microsoft's new search engine shows all of the urls in question being fully indexed within its results.
My Theory On This Mysterious Microsoft Crawler
The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.
First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.
Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.
So Microsoft's Screen Scraping?
Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.
Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.
Jason's article first appeared on his blog MarketingShift.com.
TOPICS: Business/Economy; Culture/Society; News/Current Events
KEYWORDS: google; internetexploiter; microsnot; underweartootight
Navigation: use the links below to view more comments.
first previous 1-20, 21-40, 41-60, 61-80 ... 181-200 next last
To: el_chupacabra
You're not. I have a headache now.
To: KwasiOwusu
'In 1994, a Microsoft employee e-mailed company chairman Bill Gates to tell him about her husband's campaign for the Washington state legislature. Stu Jacobson is a liberal Democrat, wrote wife Sandra. He favors abortion rights, wants stricter gun control, and supports the state's 1993 health-reform law (a mini version of ClintonCare).
"It sounds like we agree on many things," replied Gates. "I am willing to endorse Stu." '
Bottom line, William H. Gates III is a left-wing socialist thug who has hypocritically exploited the capitalist system while refusing to embrace free market capitalist priniciples.
He is only a small push from becoming just like George Soros.
To: K1avg
"Personal insults get us nowhere around here"
Look who's talking.
Remember your first post to me?
I'll just quote it back to you:
"Are you always like this, or is it PMS?
Take a midol and settle down, sister"
If that is not an insult what is?
To: JoJo Gunn
Calling you a moron will be an insult to morons.
You still need to grow up a few more years, then maybe you can be Mr. Moron. :)
To: KwasiOwusu
Seems you're confused.
jra posted that to you.
45
posted on
11/11/2004 2:23:14 PM PST
by
K1avg
To: xrp
The original post made certain allegations against Microsoft, suggesting that Microsoft was using nefarious tactics against Google (on very flimsy evidence), on the very day that Microsoft brought out a really good beta of their long awaited search engine.
If that is not anti-Mcrosoft FUD, what is?
To: KwasiOwusu
If that is not anti-Mcrosoft FUD, what is? It's called truth.
To: djf
You got it. There is this cool documentary from PBS that came out years ago about computers, 8080a, apple, MS, ect. It is called "the revenge of the Nerds". One of the best documentarys I have ever seen. It told about all of this.
48
posted on
11/11/2004 2:27:37 PM PST
by
Revel
To: reagandemo
49
posted on
11/11/2004 2:28:54 PM PST
by
Revel
To: Knitebane
Has Bill Gates had liberal leanings? Sure.
But the point is, he backed George Bush in this elections, same as Zell Miller , and Ed Koch did.
You could just as well say Zell Miller and Ed Koch have been liberal most of their lives too.
They still backed President Bush though, as did Bill Gates. That's what matters.
To: KwasiOwusu
Perhaps I was too harsh on a moronic Gates paid newbie who calls established members trolls?
Run along, ShillBoy. There are many letters to the editor from dead people you have to write before the sun goes down. And don't forget to haunt ZDnet and Cnet ... 
51
posted on
11/11/2004 2:29:54 PM PST
by
JoJo Gunn
(More than two lawyers in any Country constitutes a terrorist organization. ©)
To: KwasiOwusu
They still backed President Bush though, as did Bill Gates. That's what matters. He gave a few thousand bucks to the Bush campaign.
He's given millions to pro-abortion groups, anti-gun groups and other leftist organizations.
That's what matters.
To: All
Check out
M$'s new engine and see for yourself. It looks very similar to Google. I dunno...
53
posted on
11/11/2004 2:32:57 PM PST
by
K1avg
To: Knitebane
"It's called truth.
Truth?
Based on flimsy to non-existent evidence and wild allegations on the Internet?
That i not truth.
Its called disinformation.
Has this guy bothered to check up his wild legations with Microsoft Research, who are actually responsible for the search technology?
To: KwasiOwusu
55
posted on
11/11/2004 2:35:19 PM PST
by
proust
To: KwasiOwusu
Has this guy bothered to check up his wild legations with Microsoft Research, who are actually responsible for the search technology? And if they were stealing from Google (like they've stolen from dozens of other companies over the years) they would actually say so, right?
This is less about whether Microsoft is stealing from Google (though the preliminary evidence seems to indicate something to that effect) and more about a continuation of a very old pattern of behavior by Microsoft, namely stealing your competitions products and then either buying them out, forcing them out of the market with dumping or suing them out of existence.
To: KwasiOwusu
Hey, are you a shill for Google or just an anti-Microsoft troll out to spread anti-Microsoft FUD? Lighten up Francis .... who are you to be calling other posters trolls ... NEWBIE
57
posted on
11/11/2004 2:37:40 PM PST
by
Centurion2000
(Truth, Justice and the Texan Way)
To: KwasiOwusu
Right-oh. This is growing increasingly absurd. I refuse to argue any longer. Goodbye.
58
posted on
11/11/2004 2:38:55 PM PST
by
K1avg
To: Knitebane
"He gave a few thousand bucks to the Bush campaign"
Bottom line:
He backed Bush in the presidential campaign.
"He's given millions to pro-abortion groups, anti-gun groups and other leftist organizations"
Heck, we have Republican Senators like Specter who have backed pro-abortion gropus too.
And they are still siting in the US Senate representing the Republican Party.
We have Gulliani, who was twice mayor of New York on the Republcan ticket and is being talked about as a possible Republican Presidential candidate in 2008, he is pro-abortion as well.
You don't have a point.
To: KwasiOwusu
Are you mentally defective?
On second thought, nevermind.
60
posted on
11/11/2004 2:40:50 PM PST
by
jra
Navigation: use the links below to view more comments.
first previous 1-20, 21-40, 41-60, 61-80 ... 181-200 next last
Disclaimer:
Opinions posted on Free Republic are those of the individual
posters and do not necessarily represent the opinion of Free Republic or its
management. All materials posted herein are protected by copyright law and the
exemption for fair use of copyrighted works.
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson
