Free Republic
Browse · Search
General/Chat
Topics · Post Article

Skip to comments.

Online publishers face a dilemma: Allow AI scraping from Google or lose search visibility Blocking the company’s AI overviews also blocks its web crawler.
Engadget ^ | Thu, Aug 15, 2024 | Will Shanklin

Posted on 08/16/2024 9:59:42 AM PDT by dennisw

As the US government weighs its options following a landmark “monopolist” ruling against Google last week, online publications increasingly face a bleak future. (And this time, it’s not just because of severely diminished ad revenue.) Bloomberg reports that their choice now boils down to allowing Google to use their published content to produce inline AI-generated search “answers” or losing visibility in the company’s search engine.

The crux of the problem lies in the Googlebot, the crawler that scours and indexes the live web to produce the results you see when you enter search terms. If publishers block Google from using their content for the AI-produced answers you now see littered at the top of many search results, they also lose the privilege of including their web pages in the standard web results.

The catch-22 has led publications, rival search engines and AI startups to pin their hopes on the Justice Department. On Tuesday, The New York Times reported that the DOJ is considering asking a federal judge to break up parts of the company (spinning off sections like Chrome or Android). Other options it’s reportedly weighing include forcing Google to share search data with competitors or relinquishing its default search-engine deals, like the $18 billion one it inked with Apple.

Google uses a separate crawler for its Gemini (formerly Bard) chatbot. But its main crawler covers both AI Overviews and standard searches, leaving web publishers with little (if any) leverage.

If you let Google scrape your content for AI Overview answers, readers may consider that the end of the matter without bothering to visit your site (meaning zero revenue from those potential readers). But if you block the Googlebot, you lose search visibility, which likely means significantly less short-term income and a colossal loss of long-term competitive standing.


TOPICS: Books/Literature; Business/Economy; Computers/Internet
KEYWORDS: ai; googlebot; monopoly; webcrawler

1 posted on 08/16/2024 9:59:42 AM PDT by dennisw
[ Post Reply | Private Reply | View Replies]

To: dennisw

That sounds like an illegal expectation, for a “search engine.”


2 posted on 08/16/2024 10:01:21 AM PDT by ConservativeMind (Trump: Befuddling Democrats, Republicans, and the Media for the benefit of the US and all mankind.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: dennisw

Anti Trust is needed here. Google = scum of the galaxy, when it comes anti-competition, monopolization, money grubbing. They have no shame.

Meanwhile, use the best. Perplexity AI search. https://www.perplexity.ai/

On one browser, I am logged into Perplexity. This means they keep a query library I can refer to. HOWEVER, if I forward a Perplexity link to someone, it comes up blank.

So...... on another browser, I am not logged into Perplexity. Then when I send a link to my Perplexity query, it is readable-useable by my friend.


3 posted on 08/16/2024 10:08:15 AM PDT by dennisw
[ Post Reply | Private Reply | To 1 | View Replies]

To: ConservativeMind

Another monopoly tool, controlled by biased per its imput .


4 posted on 08/16/2024 10:09:15 AM PDT by Freeleesy
[ Post Reply | Private Reply | To 2 | View Replies]

To: dennisw

Immediately block it in mass to make them adjust their their attitude “So what? We can live with out you Google.”

Google is not the only the only one crawling the web and there are tons of SEO tools to replace Google tools. And tons of alternative analytic tools

It is way past time for domain owners to mass boycott Google. “You have to have Google” is an absolute ignorant myth. Dump them like a hot rock and do not go back. Their services can be replaced without any problem at all nowdays. They are running a bluff and folks are falling for their scams.


5 posted on 08/16/2024 10:28:59 AM PDT by Openurmind (The ultimate test of a moral society is the kind of world it leaves to its children. ~ D. Bonhoeffer)
[ Post Reply | Private Reply | To 1 | View Replies]

To: dennisw
I've been told by Libertarians that anti-trust laws are statist. That private companies can only grow large by offering a superior product. And that in a free market, any small competitor could topple even a huge company.

I don't buy a lot of that anymore.

6 posted on 08/16/2024 10:29:22 AM PDT by Angelino97
[ Post Reply | Private Reply | To 3 | View Replies]

To: dennisw

It’s getting hard to tell who the customer is and who the product is.


7 posted on 08/16/2024 10:42:46 AM PDT by radmanptn
[ Post Reply | Private Reply | To 1 | View Replies]

To: radmanptn

Google has always viewed YOU the user as their product.


8 posted on 08/16/2024 10:47:47 AM PDT by bigbob
[ Post Reply | Private Reply | To 7 | View Replies]

To: Angelino97
And that in a free market, any small competitor could topple even a huge company.

I don't buy a lot of that anymore.

It's still true - when government doesn't put its thumb on the scale. Google has always been so closely integrated with the intelligence community as to be indistinguishable from a Federal agency, itself. There is no way the IC is going to allow its #1 information source to be disrupted by some upstart anti-trust judge.

9 posted on 08/16/2024 10:53:01 AM PDT by Mr. Jeeves ([CTRL]-[GALT]-[DELETE])
[ Post Reply | Private Reply | To 6 | View Replies]

To: bigbob

I agree, but now it appears that the entities that you were present to as the product are also the product. If only the top AI excerpt is read who is their customer?


10 posted on 08/16/2024 11:00:03 AM PDT by radmanptn
[ Post Reply | Private Reply | To 8 | View Replies]

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
General/Chat
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson