ChatGPT gets access to website content to learn from it. This is how to block your content from becoming AI training data.
There is concern about the lack of an easy way to opt out of having one’s content used to train large language models (LLMs) like ChatGPT. There is a way to do it, but it’s neither straightforward nor guaranteed to work.
How AIs Learn From Your Content
Large Language Models (LLMs) are trained on data that originates from multiple sources. Many of these datasets are open source and are freely used for training AIs.
Some of the sources used are:
- Government court records
- Crawled websites
Portals and websites offer datasets that give away vast amounts of information.
One of the portals is hosted by Amazon, offering thousands of datasets at the Registry of Open Data on AWS.
The Amazon portal with thousands of datasets is just one out of many others that contain more.
Wikipedia lists 28 portals for downloading datasets, including the Google Dataset and the Hugging Face portals for finding thousands of datasets.
Datasets of Web Content
A popular dataset of web content is called OpenWebText. OpenWebText consists of URLs found on Reddit posts that had at least three upvotes.
The idea is that these URLs are trustworthy and will contain quality content. I couldn’t find information about a user agent for their crawler, maybe it’s just identified as Python, I’m not sure.
Nevertheless, we do know that if your site is linked from Reddit with at least three upvotes then there’s a good chance that your site is in the OpenWebText dataset.
More information about OpenWebText is here.
One of the most commonly used datasets for Internet content is offered by a non-profit organization called Common Crawl.
Common Crawl data comes from a bot that crawls the entire Internet.
The data is downloaded by organizations wishing to use the data and then cleaned of spammy sites, etc.
The name of the Common Crawl bot is, CCBot.
CCBot obeys the robots.txt protocol so it is possible to block Common Crawl with Robots.txt and prevent your website data from making it into another dataset.
However, if your site has already been crawled then it’s likely already included in multiple datasets.
Nevertheless, by blocking Common Crawl it’s possible to opt out your website content from being included in new datasets sourced from newer Common Crawl data.
The CCBot User-Agent string is:
Add the following to your robots.txt file to block the Common Crawl bot:
User-agent: CCBot Disallow: /
An additional way to confirm if a CCBot user agent is legit is that it crawls from Amazon AWS IP addresses.
CCBot also obeys the nofollow robots meta tag directives.
Use this in your robots meta tag:
Blocking AI From Using Your Content
Search engines allow websites to opt out of being crawled by AI bots such as ChatGPT. Common Crawl also allows opting out. But there is currently no way to remove one’s website content from existing datasets.
Furthermore, research scientists don’t seem to offer website publishers a way to opt out of being crawled.
The article, Is ChatGPT Use Of Web Content Fair? explores the topic of whether it’s even ethical to use website data without permission or a way to opt out.
Many publishers may appreciate it if in the near future, they are given more say on how their content is used, especially by AI products like ChatGPT.
Whether that will happen is unknown at this time.
Related: [PHISHING ALERT] “Hey Did You See That Fake AI Porn Movie Of Yourself?”
Article by Roger Montii
Roger Montti is a search marketer with over 20 years of experience. I offer site audits, phone consultations, and content and link strategy assistance.
I am a Head Judge of the 2021 U.S. Search Awards. I am also a publisher of award-winning websites. I was a Head Judge for the 2020 U.S. Search awards and have presented at the following search marketing conferences:
SES, SMX East, SMX West, SMX Advanced Seattle, Affiliate Summit NYC, Affiliate Summit West, PubCon Las Vegas, PubCon Dallas, PubCon New Orleans, PubCon Orlando, PubCon Austin
I write about WordPress, Facebook, Google, SEO, and search marketing.
Contact me at Martinibuster.com
Leave A Comment
You must be logged in to post a comment.