Posted on 11/11/2004 1:35:03 PM PST by mhking
2004-11-11
|
 |
| Is MSN Crawling Google? |
 |
My Theory On This Mysterious Microsoft Crawler
The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.
First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.
Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.
So Microsoft's Screen Scraping?
Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.
Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.
Jason's article first appeared on his blog MarketingShift.com.
"Lighten up Francis .... who are you to be calling other posters trolls ... NEWBIE"
Go suck on a lollipop OLD TIMER!
I AM THE DARKLORD GATES.
MHKING IS NOT MY UNDERLING.
HE MERELY LIKES THE NICE TOYS SLICK INTERFACE AND OTHER TWEAKS HE DOES TOO THE SYSTEM.
I SHALL FEED UPON THE UNHAPPINESS OF MY ENEMIES.
OBEY THE CAT.
*cough*
/ end of 'voice of the ultimate unnngh'.
You're lucky there's not a limit on getting b!tch-slapped in a single thread, cuz you'd be over the limit.
Welcome to FR, and better luck next time on biting off more than you can chew.
He wrote a check for an amount that would be equivilant to you or me digging some change out of the couch cushions. That is not equal to "support." That is equal to an insult or perhaps a cheap way to set up a legion of trolls to go out and astroturf for him now that Bush has been re-elected.
And here you are.
Heck, we have Republican Senators like Specter who have backed pro-abortion gropus too. And they are still siting in the US Senate representing the Republican Party. We have Gulliani, who was twice mayor of New York on the Republcan ticket and is being talked about as a possible Republican Presidential candidate in 2008, he is pro-abortion as well.
And none of those people have billions of dollars in the bank or send millions of dollars to left-wing causes. And none of them are part of this discussion, so let's keep to the subject of WHG3 and his politics, shall we?
You don't have a point.
I think the point is whether you believe a recently paroled car thief has stopped boosting cars because he drops a fiver in the collection plate or whether you continue to keep an eye on him, set the alarm on your car and wait to see if he's really changed.
And if your neighbor's car gets swiped, who do you suspect first?
Bill Gates has a long history of being a lefty. One public political contribution does not make that vanish, no matter how badly you want it to.
You might want to rephrase that comment newbie.
Also, I regrettably won't be able to skewer you for the next couple of hours...I'm going to dinner.
Love,
jra
Do you know anything about ping acknowledge packets, ip addies, and internet routers?
LOL!
And it doesn't make the 100 million he gave to the UN vanish either.
Knock off the personal attacks.
Umm.. you MIGHT want to look at the replies you've generated.
YOU DEFINITELY ARE GETTING B!TCH SLAPPED, buddy.
While trying to keep track of all the other leftist causes he supports I let that one slip my mind.
Nope.
Quality, not quantity.
Naah.
Not about to explode.
Just about to call down the lightning on a smart mouthed newbie who doesn't know when to shut up.
Then paving operations begin.
I notice he hasn't answered this question.
Good grief! You sure are a smart-aleck.
Why don't you calm down a bit, dude ?
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.