1
SEO / Re: Blocking Unwanted Spider Crawls
« on: April 23, 2016, 02:19:08 PM »
Add the following code in your robots txt
# Disallow Following crawlers access to site.
User-agent: TurnitinBot/2.1
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: AboutUsBot
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: robtexbot
Disallow: /
# Disallow Following crawlers access to site.
User-agent: TurnitinBot/2.1
Disallow: /
User-agent: TurnitinBot
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: EmailWolf
Disallow: /
User-agent: EmailSiphon
Disallow: /
User-agent: EmailCollector
Disallow: /
User-agent: AboutUsBot
Disallow: /
User-agent: SurveyBot
Disallow: /
User-agent: robtexbot
Disallow: /