General Topics > SEO

Robots.txt

<< < (3/3)

DavidLIR:
Great Thanks

edwin324:
Thanks for sharing, if any one still confused with robots.txt rules then refer below URL and upload the .txt file to your root folder.
 
robotstxt.org/robotstxt.html

paulsimmons:
You don't have to enter the allow commands. By default it is allowed for Google to crawl and index whole website. You only need to mention the disallow code to stop crawlers from indexing unwanted URLs like customer accounts, admin urls etc.

Advisor:
Don't block important directories like IMG / image dirs... or your products will not show up in Google Images

Natashawilliams:
nice post! thank you for sharing information about robots.txt

Navigation

[0] Message Index

[*] Previous page