Skip to comments.
Microsoft Crawling Google Results For New Search Engine?
WebProNews ^
| 11.11.04
Posted on 11/11/2004 1:35:03 PM PST by mhking
Microsoft Crawling Google Results For New Search Engine?
Jason Dowdell | Contributing Writer
2004-11-11
I was questioned today by a developer who was watching a particular IP address scan his site. The IP was 65.54.188.86 and is registered to Microsoft Corp. located at One Microsoft Way, Redmond, Washington 98052. This visitor was not sending the normal header information associated with a crawler to the web server such as an http robot name or identifying info or even a browser name.
 |
| Is MSN Crawling Google? |
 |
Is Microsoft "using" Google's search results to populate their index? Discuss Microsoft's behavior at WebProWorld.
The behavior it demonstrated made it look like a crawler, especially since it was spidering urls that were no longer in existence (search engine spiders crawl site segments at regular intervals and often come back when an initial crawl left urls uncrawled) and doing so at the rate of 1 page every 3 - 5 seconds. The visitor started their visit at 7:37 am and was still on the site at 12:00 pm.
Correction, the data was there after all, here's the crawler info... msnbot/0.3 (+http://search.msn.com/msnbot.htm)
Here's the kicker
So now you're saying, so what, big deal. But this really is a big deal. It's a big deal not only because the urls this visitor was making requests to don't exist any longer but because the only place these urls can be found is in Google's search results using site:www.sitename.com. A similar query on MSN Search doesn't show the urls at all, even on the beta version of their new Microsoft search engine. But then within just hours of the visitors exit from the site the new same search at Microsoft's new search engine shows all of the urls in question being fully indexed within its results.
My Theory On This Mysterious Microsoft Crawler
The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.
First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.
Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.
So Microsoft's Screen Scraping?
Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.
Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.
Jason's article first appeared on his blog MarketingShift.com.
TOPICS: Business/Economy; Culture/Society; News/Current Events
KEYWORDS: google; internetexploiter; microsnot; underweartootight
Navigation: use the links below to view more comments.
first previous 1-20 ... 121-140, 141-160, 161-180, 181-200 next last
To: JoJo Gunn
"Yes, it's also our responsibility to do the legwork for the writer of any piece. "
I am sure if this story has anything to it, its going to get spread all over the Internet and onto the Old Media, and Microsoft is going to have to issue a statement sooner or later, like they almost always o these days when rumors like this crop up.
I am sure the truth is going to come out sooner or later. After all, it is a pretty serious allegation.
To: KwasiOwusu
After all, it is a pretty serious allegation.Actually, it's not. The author noted that there's likely nothing legally wrong with it.
It's the principle of the thing. Being a conservative (I'm assuming here), you can understand that, right?
We never have figured out: do you own M$ stock?
142
posted on
11/11/2004 3:52:36 PM PST
by
K1avg
To: KwasiOwusu
Microsoft has denied it and Google has more or less acknowledged it didn't happen.
In other words: Both companies deny it.
143
posted on
11/11/2004 3:54:16 PM PST
by
invoman
To: Knitebane
"Shall we ask Spyglass Software or Stac Software about what they think of Microsoft's marketing strategy?
Namely, lie to get inside, steal someone else's product and then sell it as your own.
When caught, sue the victim out of existence
"
#1, Weren't Spyglass the ones that sold the original browser technology to Microsoft?
When did Microsoft turn round and steal from Spyglass again?
#2, wasn't it Stack that sued Microsoft, instead of the other way round?
And #3, didn't Stac and Microsoft end up signing cross licensing agreements and become good buddies?
To: K1avg
"We never have figured out: do you own M$ stock?"
Yep.
A very proud owner of Microsoft stock.
Doesn't change the fact that we still need to hear both sides of any such allegations.
Both you and I know there is too much hatred towards Gates and Microsoft on the Internet. There are sites out there dedicatd to hating Gates or Microsoft
And we have more Microsoft conspiracy nuts out there than even DU nuts.
So when see any such stories (which appear all the time), I usually have to see it confirmed from either or both sides, before I start taking it to seriously.
To: KwasiOwusu
I personally don't think it's nowhere near being as big of a deal as their employees writing letters to editors and signing with names of the dead, or the fact that MS seems more worried about their security than mine, (such as their focus on licensing, how they want me to get user authorization from yesterdays fry cooks for something that I paid for, and ignoring so many holes in their code yet howling when others reveal to the world the latest security holes).
But that's not my point. You apparently are calling for censorship, for us to not post certain things in order to protect Big Brother Gates. Another irony, since as I just said, Gates doesn't care much about protecting us.
146
posted on
11/11/2004 4:03:40 PM PST
by
JoJo Gunn
(More than two lawyers in any Country constitutes a terrorist organization. ©)
To: invoman
"Microsoft has denied it and Google has more or less acknowledged it didn't happen. In other words: Both companies deny it"
I'll be glad if your info is crrect of course.
But do you have any link for that news, please?
To: KwasiOwusu
Plus Gogle topped themselves by coming up with President Bush's name whenever anyone typed in the words "Pathetic failure" in Google search.
This was due to the concerted efforts of many thousands of bloggers, who created page after page with "miserable failure" hrefs pointing to President Bush's site. It was an exploitation of one of Google pre-existing methods of associating search keywords with urls, based on volume of like-named links pointing to a given url target. The staff of Google had no part in this. In fact, the same type of tactic was been employed by the good guys against Michael Moore's site, though I don't remember which derogatory search phrase was used (as if anything could be considered derogotory against scum like him).
148
posted on
11/11/2004 4:08:21 PM PST
by
GETMAIN
To: KwasiOwusu
#1 Spyglass agreed to give Microsoft the source code to their Internet Browser. In return, Microsoft would give Spyglass a percentage of every copy sold.
Numbers were bandied about, but the final number was zero, since after Microsoft had the code in hand they gave away the browser, leaving Spyglass with nothing.
2) Microsoft entered into negotiations with Stac to include Stacker drive compression software with the next version of MSDOS, version 6. As part of the negotiation process, Microsoft was allowed to borrow the Stac code in order to "ensure integration."
When Stac wanted too much money, Microsoft broke off negotiations. When MSDOS 6 shipped it included disk compression software. Suspicious, Stac developers reverse engineered MSDOS 6 to determine whether the code being used was theirs.
It was. Stac sued.
Microsoft counter-sued. What for? Because Stac caught them. Stac was sued for the reverse engineering that they did to determine that Microsoft had stolen from them.
And reverse engineering is legal. Maybe it had something to do with the fact that Microsoft dragged the legal proceeding out until Stac was bankrupt. When Stac couldn't pay their attorneys anymore, Microsoft proposed a settlement. Microsoft paid Stac enough to cover their outstanding debts.
Both companies have since recovered somewhat, though neither has made much of an entry into any market. The talented engineers and managers that helped those companies create innovative software have since moved on. The remains of those companies, under different management (and others that have had similar experiences with Microsoft) have since made some minor licensing deals with Microsoft.
To: KwasiOwusu
150
posted on
11/11/2004 4:12:15 PM PST
by
xrp
(Executing assigned posting duties flawlessly -- ZERO mistakes)
To: KwasiOwusu
There are sites out there dedicatd to hating Gates or Microsoft Perhaps if Gates and Microsoft didn't do such hateful and evil things, those sites wouldn't be there.
Being hated is the price one pays for being nasty, vicious and evil.
To: KwasiOwusu
152
posted on
11/11/2004 4:12:37 PM PST
by
invoman
To: JoJo Gunn
"But that's not my point. You apparently are calling for censorship, for us to not post certain things in order to protect Big Brother Gates"
A big NO to censorship of any kind.
As for Gates, has he done some pretty nasty things in his business and personal life? You bet.
I read "Hard Drive" and at least another 20 books on Microsoft and ates.
He has been pretty ruthless when it suited him.
But I keep asking myself: Are most successful businessmen like that? and the answer I keep getting increasingly is yeah.
Just look at Larry Ellison of Oracle, or Ron Perelman of Revlon who even as we speak is screwing some restaurateur here in New York City or Micheal Eisner of Disney, and literally hundreds of these top executives.
Most of them are pretty ruthless men.
I just sometimes wonder why Gates in particular is the subject of so much venom and hatred, when more ruthless men like Ellison get a pass.
To: GETMAIN
"miserable failure" was Google-bombed to GWB, but also Jimmy Carter and Michael Moore. You can see the full bit
here. Hillary Clinton and Dick Gephardt were at one point Google-bombed as "miserable failures," but they seem to have fallen a bit in the rankings lately.
In sum, what you say is correct. Google or its owners had no hand in making the miserable failure link.
154
posted on
11/11/2004 4:16:36 PM PST
by
K1avg
To: KwasiOwusu
Yahoo! IS showing the bio for Bush as a #1 result for "miserable failure".
For that matter, so is MSN.
155
posted on
11/11/2004 4:19:50 PM PST
by
invoman
To: invoman
To: KwasiOwusu; mhking
I suspect Gates deserves a lot of venom from Freepers, seeing as how every single one of them are
computer users!
I suppose I could come out of the closet and quibble about Revlon if mhking would have the audacity to post a thread about them. 
157
posted on
11/11/2004 4:22:31 PM PST
by
JoJo Gunn
(More than two lawyers in any Country constitutes a terrorist organization. ©)
To: invoman
To: JoJo Gunn
"I suppose I could come out of the closet and quibble about Revlon if mhking would have the audacity to post a thread about them. "
You don't want to do that. :)
Your gf/wife may not like it. LOL!
To: KwasiOwusu
wow I haven't seen this much crossfire since I defended Jeff Gordon on a Nascar thread......
Navigation: use the links below to view more comments.
first previous 1-20 ... 121-140, 141-160, 161-180, 181-200 next last
Disclaimer:
Opinions posted on Free Republic are those of the individual
posters and do not necessarily represent the opinion of Free Republic or its
management. All materials posted herein are protected by copyright law and the
exemption for fair use of copyrighted works.
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson
