General Topics > SEO
Blocking Unwanted Spider Crawls
ezeeozee:
Specifically> baiduspider from baidu.com
So annnoying.
Apparently it originates from China or thereabouts and it is bombarding my website with thousands of crawls which are eating up my bandwidth.
I want to block it altogether but can't find an easy way to do this. Anyone know an easy way?
On a previous cart I used to use, there was a page within Admin which listed all the spiders/bots/search engines, which had access to the site and you could allow/disallow access with a click.
Is there anything like this within Abantecart? Or an Extension for this?
Is tinkering with the htaccess file the only option?
Any answers to any of the above would be appreciated even if it means pointing me to where this query has already been answered somewhere.
Thank you.
Basara:
Hello.
Try robot.txt file https://support.google.com/webmasters/answer/6062608?hl=en
ezeeozee:
Brilliant, thank you!
ezeeozee:
Doesn't work.
Baiduspider is resistant to attempts to block it through robot.txt and also .htaccess.
Modifying both files has made diddly squat difference to the volume of crawls from Baiduspider.
As a last resort I am trying to block the IP - they use multiple IPs but they all start the same so I have added this to the .htaccess file:
Deny from 180.76.15.
Hopefully, this will work.
yonghan:
Hi, please take a look here and test it. Who knows it works for you.
http://webmasters.stackexchange.com/questions/31837/how-to-block-baidu-spiders
Navigation
[0] Message Index
[#] Next page
Go to full version