Free Republic
Browse · Search
News/Activism
Topics · Post Article

Skip to comments.

Microsoft Crawling Google Results For New Search Engine?
WebProNews ^ | 11.11.04

Posted on 11/11/2004 1:35:03 PM PST by mhking

Microsoft Crawling Google Results For New Search Engine?


Jason Dowdell | Contributing Writer

2004-11-11



I was questioned today by a developer who was watching a particular IP address scan his site. The IP was 65.54.188.86 and is registered to Microsoft Corp. located at One Microsoft Way, Redmond, Washington 98052. This visitor was not sending the normal header information associated with a crawler to the web server such as an http robot name or identifying info or even a browser name.

MSN Spiders
Is MSN Crawling Google?

Is Microsoft "using" Google's search results to populate their index? Discuss Microsoft's behavior at WebProWorld.

The behavior it demonstrated made it look like a crawler, especially since it was spidering urls that were no longer in existence (search engine spiders crawl site segments at regular intervals and often come back when an initial crawl left urls uncrawled) and doing so at the rate of 1 page every 3 - 5 seconds. The visitor started their visit at 7:37 am and was still on the site at 12:00 pm.

Correction, the data was there after all, here's the crawler info... msnbot/0.3 (+http://search.msn.com/msnbot.htm)

Here's the kicker

So now you're saying, so what, big deal. But this really is a big deal. It's a big deal not only because the urls this visitor was making requests to don't exist any longer but because the only place these urls can be found is in Google's search results using site:www.sitename.com. A similar query on MSN Search doesn't show the urls at all, even on the beta version of their new Microsoft search engine. But then within just hours of the visitors exit from the site the new same search at Microsoft's new search engine shows all of the urls in question being fully indexed within its results.



My Theory On This Mysterious Microsoft Crawler

The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.

First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.

Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.

So Microsoft's Screen Scraping?

Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.

Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.

Jason's article first appeared on his blog MarketingShift.com.


TOPICS: Business/Economy; Culture/Society; News/Current Events
KEYWORDS: google; internetexploiter; microsnot; underweartootight
Navigation: use the links below to view more comments.
first previous 1-2021-4041-6061-80 ... 181-200 next last
To: el_chupacabra

You're not. I have a headache now.


41 posted on 11/11/2004 2:18:32 PM PST by arizonarachel (Prayer works!)
[ Post Reply | Private Reply | To 6 | View Replies]

To: KwasiOwusu
'In 1994, a Microsoft employee e-mailed company chairman Bill Gates to tell him about her husband's campaign for the Washington state legislature. Stu Jacobson is a liberal Democrat, wrote wife Sandra. He favors abortion rights, wants stricter gun control, and supports the state's 1993 health-reform law (a mini version of ClintonCare). "It sounds like we agree on many things," replied Gates. "I am willing to endorse Stu." '

Bottom line, William H. Gates III is a left-wing socialist thug who has hypocritically exploited the capitalist system while refusing to embrace free market capitalist priniciples.

He is only a small push from becoming just like George Soros.

42 posted on 11/11/2004 2:18:51 PM PST by Knitebane
[ Post Reply | Private Reply | To 27 | View Replies]

To: K1avg
"Personal insults get us nowhere around here"

Look who's talking.

Remember your first post to me?
I'll just quote it back to you:
"Are you always like this, or is it PMS?
Take a midol and settle down, sister"

If that is not an insult what is?
43 posted on 11/11/2004 2:19:36 PM PST by KwasiOwusu
[ Post Reply | Private Reply | To 37 | View Replies]

To: JoJo Gunn

Calling you a moron will be an insult to morons.
You still need to grow up a few more years, then maybe you can be Mr. Moron. :)


44 posted on 11/11/2004 2:21:56 PM PST by KwasiOwusu
[ Post Reply | Private Reply | To 40 | View Replies]

To: KwasiOwusu
Seems you're confused. jra posted that to you.
45 posted on 11/11/2004 2:23:14 PM PST by K1avg
[ Post Reply | Private Reply | To 43 | View Replies]

To: xrp
The original post made certain allegations against Microsoft, suggesting that Microsoft was using nefarious tactics against Google (on very flimsy evidence), on the very day that Microsoft brought out a really good beta of their long awaited search engine.
If that is not anti-Mcrosoft FUD, what is?
46 posted on 11/11/2004 2:25:18 PM PST by KwasiOwusu
[ Post Reply | Private Reply | To 39 | View Replies]

To: KwasiOwusu
If that is not anti-Mcrosoft FUD, what is?

It's called truth.

47 posted on 11/11/2004 2:27:22 PM PST by Knitebane
[ Post Reply | Private Reply | To 46 | View Replies]

To: djf

You got it. There is this cool documentary from PBS that came out years ago about computers, 8080a, apple, MS, ect. It is called "the revenge of the Nerds". One of the best documentarys I have ever seen. It told about all of this.


48 posted on 11/11/2004 2:27:37 PM PST by Revel
[ Post Reply | Private Reply | To 32 | View Replies]

To: reagandemo

No I did not know that.


49 posted on 11/11/2004 2:28:54 PM PST by Revel
[ Post Reply | Private Reply | To 25 | View Replies]

To: Knitebane
Has Bill Gates had liberal leanings? Sure.
But the point is, he backed George Bush in this elections, same as Zell Miller , and Ed Koch did.
You could just as well say Zell Miller and Ed Koch have been liberal most of their lives too.
They still backed President Bush though, as did Bill Gates. That's what matters.
50 posted on 11/11/2004 2:29:33 PM PST by KwasiOwusu
[ Post Reply | Private Reply | To 42 | View Replies]

To: KwasiOwusu
Perhaps I was too harsh on a moronic Gates paid newbie who calls established members trolls?

Run along, ShillBoy. There are many letters to the editor from dead people you have to write before the sun goes down. And don't forget to haunt ZDnet and Cnet ...

51 posted on 11/11/2004 2:29:54 PM PST by JoJo Gunn (More than two lawyers in any Country constitutes a terrorist organization. ©)
[ Post Reply | Private Reply | To 44 | View Replies]

To: KwasiOwusu
They still backed President Bush though, as did Bill Gates. That's what matters.

He gave a few thousand bucks to the Bush campaign.

He's given millions to pro-abortion groups, anti-gun groups and other leftist organizations.

That's what matters.

52 posted on 11/11/2004 2:31:25 PM PST by Knitebane
[ Post Reply | Private Reply | To 50 | View Replies]

To: All
Check out M$'s new engine and see for yourself. It looks very similar to Google. I dunno...
53 posted on 11/11/2004 2:32:57 PM PST by K1avg
[ Post Reply | Private Reply | To 51 | View Replies]

To: Knitebane
"It's called truth.

Truth?
Based on flimsy to non-existent evidence and wild allegations on the Internet?
That i not truth.
Its called disinformation.
Has this guy bothered to check up his wild legations with Microsoft Research, who are actually responsible for the search technology?
54 posted on 11/11/2004 2:33:37 PM PST by KwasiOwusu
[ Post Reply | Private Reply | To 47 | View Replies]

To: KwasiOwusu

Just Damn.


55 posted on 11/11/2004 2:35:19 PM PST by proust
[ Post Reply | Private Reply | To 19 | View Replies]

To: KwasiOwusu
Has this guy bothered to check up his wild legations with Microsoft Research, who are actually responsible for the search technology?

And if they were stealing from Google (like they've stolen from dozens of other companies over the years) they would actually say so, right?

This is less about whether Microsoft is stealing from Google (though the preliminary evidence seems to indicate something to that effect) and more about a continuation of a very old pattern of behavior by Microsoft, namely stealing your competitions products and then either buying them out, forcing them out of the market with dumping or suing them out of existence.

56 posted on 11/11/2004 2:37:20 PM PST by Knitebane
[ Post Reply | Private Reply | To 54 | View Replies]

To: KwasiOwusu
Hey, are you a shill for Google or just an anti-Microsoft troll out to spread anti-Microsoft FUD?

Lighten up Francis .... who are you to be calling other posters trolls ... NEWBIE

57 posted on 11/11/2004 2:37:40 PM PST by Centurion2000 (Truth, Justice and the Texan Way)
[ Post Reply | Private Reply | To 19 | View Replies]

To: KwasiOwusu

Right-oh. This is growing increasingly absurd. I refuse to argue any longer. Goodbye.


58 posted on 11/11/2004 2:38:55 PM PST by K1avg
[ Post Reply | Private Reply | To 44 | View Replies]

To: Knitebane
"He gave a few thousand bucks to the Bush campaign"


Bottom line:
He backed Bush in the presidential campaign.

"He's given millions to pro-abortion groups, anti-gun groups and other leftist organizations"
Heck, we have Republican Senators like Specter who have backed pro-abortion gropus too.
And they are still siting in the US Senate representing the Republican Party.
We have Gulliani, who was twice mayor of New York on the Republcan ticket and is being talked about as a possible Republican Presidential candidate in 2008, he is pro-abortion as well.
You don't have a point.
59 posted on 11/11/2004 2:40:19 PM PST by KwasiOwusu
[ Post Reply | Private Reply | To 52 | View Replies]

To: KwasiOwusu

Are you mentally defective?

On second thought, nevermind.


60 posted on 11/11/2004 2:40:50 PM PST by jra
[ Post Reply | Private Reply | To 43 | View Replies]


Navigation: use the links below to view more comments.
first previous 1-2021-4041-6061-80 ... 181-200 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
News/Activism
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson