Posted on 11/11/2004 1:35:03 PM PST by mhking
2004-11-11
|
 |
| Is MSN Crawling Google? |
 |
My Theory On This Mysterious Microsoft Crawler
The old msn required a fee to be crawled by its spider. But a few months back MSN dropped the fee and said they were going to begin crawling the entire web and doing it without charge. However, that's no easy task. So I believe MSN is using the results from Google and possibly even Yahoo to get all of the pages they've indexed on sites that have a relatively low page count in the current msn search engine.
First off, that's the fastest way to get the relevant pages from a web site. Sure they could just go to the site directly and start crawling but in doing so they're going to get tons of duplicate urls and urls that seem different but point to the same content. Crawling Google's results will eliminate the bandwidth to some extent but will not completely take care of the duplicate content issue their spider will encounter.
Secondly, crawling Google's results can act as a qualitative measure for their new search engine. By creating a baseline number of pages per site when the new Microsoft Search is launched and running a comparison on a regular interval for the next 6 months, they'll be able to determine internally if their engine is finding and indexing the same links and as many links as Google. Call it competitive analysis or whatever you want.
So Microsoft's Screen Scraping?
Obviously my conclusion should be taken as a grain of salt but it's a definite possibility. Microsoft very well could be screen scraping Google (or maybe even using their API, LOL) and crawling the urls it finds. It makes sense from a business case but I wonder if there are any legal issues there. I doubt it. It's like putting garbage out to the curb. Once it's out there it's fair game but I bet Google's lawyers would have more to say than that on the case.
Has anyone out there seen similar behavior on their own sites? Please comment with your qualitative/objective data if so.
Jason's article first appeared on his blog MarketingShift.com.
Isn't it odd, though, that my site ends up in exactly the same position with the same search terms as it does in Google?
I won't tell you the site URL, but I've worked very hard to get it to the third position on the first page of Google for that search term. I don't mind that it shows up on the Microsoft engine in the same position, but it's interesting.
Looks to me like MSFT is just mining Google's database.
Well, let's hope he's learning. He went to eat 2 hours ago. Haven't heard a peep since.
*fingers crossed*
He lost me at "I."
You're right on both statements. Some think that one of the reasons that Microsoft got sued was because it didn't have any lobbyists (or only a couple, if any) and the other tech firms had quite a few.
That was before he was married and had 3 children. Everyone changes (BTW his wife is a Catholic, and I believe he attends Mass with her, at least on occassion).
But hey, that's only what the US Courts, Circuit and Appelate, thought after reviewing all of the evidence, even the evidence that Microsoft's lawyers got caught forging.
I wonder what the response would have been if it had been some company like, say, CBS that got caught faking evidence....
Oh, wait....
bttt
Well, if not, kitties might be ready to pounce, LOL.
But since he hasn't done any of that, I still call him a lefty.
Bill Clinton went to church once or twice too. That doesn't make him any more of a moral person.
It would be nice if he's changed. I await the evidence that he has. But so far, all of the extant evidence shows that he funds and supports far left causes.
You know more than I about what left causes he supports. I do know his foundation gives millions to help people, vaccinating children in Africa was just one of the causes.
IMHO, it never would have gotten to the courts if there hadn't been so many lobbyists from the other side pushing it through the Dem (aka Reno) Justice Dept (who had some problems of their own they were trying to divert attention to). If these other companies could compete on their own they wouldn't be such cry babies. When I was little, and would go in whining to my mom that someone was picking on me, she send me back out and say go fight your own battles.
The other companies could have 'fought' Microsoft by inventing better products and selling them at better prices, but they didn't. The states and the gov't saw $$s - kind of like the tobacco suits.
GOOGLE CAN GO TO HELL. THEY REFUSED TO ALTER THEIR LOGO TODAY IN HONOR OF VETERANS DAY, EVEN THOUGH THEY ROUTINELY CHANGE IT FOR NON-HOLIDAYS LIKE FRANCE'S BASTILE DAY AND "EARTH DAY". LET THEM KNOW YOU ARE PISSED OFF AT THEIR UTTER CONTEMPT FOR AMERICA'S BRAVE VETERANS!!!
Blah blah blah Google. Blah blah blah Microsoft. Blah blah blah Bill Gates. Blah blah blah blah blah. Blah.
There that all I have to say about that. Let's play nice ; )
Apparently we have a flu vaccine shortage in this country. Where is Bill Gates? If he's such a wonderful American, you'd think that his charity would start at home.
Rather, he sends his money overseas to help improve the lives of third-worlders, who send us UN reps that bad-mouth us.
Thanks Bill.
When I was little, and would go in whining to my mom that someone was picking on me, she send me back out and say go fight your own battles.
Really? And if that someone had a gun, would she still send you back out to fighy your own battle?
In our litiginous society, sending a company with a $10 million market cap out to do legal battle with Microsoft is akin to sending a kid out against a mass murderer with a gun.
We have courts and legal processes that are designed to level the playing field so that everyone gets justice.
I'm sorry for you that you feel so badly that Microsoft was found to be a monopoly, that they had illegally used that monopoly and that they did, indeed, do unethical, immoral and illegal things.
I'm less sorry about the final judgement. Microsoft got a slap on the wrist. For much fewer evil things, both Standard Oil and AT&T were broken up.
The other companies that 'fought' Microsoft did so because Microsoft wouldn't compete against them directly. Microsoft used illegal tactics to shape the market such that no one but Microsoft could get their products to market.
And since the government didn't get a dime from Microsoft I'm not sure where your consipiracy theory about the government seeing $$$ comes from.
But then I tend to suspect either the sanity, judgement or agenda of anyone who still supports Microsoft or Bill Gates after having seen the evidence.
Excuse me, but THAT statement is utterly FALSE. The United States is FAR more supportive of abortion than the EU. The dabates in the EU over the last decade or so have ALL revolved around the "first trimester" - NO ONE EU Country allows abortion after the first trimester.
We however (as a country) allow abortion until the very end - as well as the atrocity of partial birth abortion.
"But then I tend to suspect either the sanity, judgement or agenda of anyone who still supports Microsoft or Bill Gates after having seen the evidence."
Thank you, but I'm perfectly sane, have very good judgement (as many have told me), and have no agendas. As anyone can see, you have very strong opinions on this. I have facts, too. I could go on more but I won't as I don't want this thread to become another repeat of last night. It's rather obvious that you don't like Microsoft and don't like Bill Gates and I doubt that anything anyone could said, would get you to change your mind (if you would even listen). Let's just respectfully agree to disagree on this one.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.