Robots.txt

[gordontaylor]

I know this is old school, but do you realize SE's still look for this file?

robots.txt

I have proven it gets better rankings by publishing two identical sites with similar URL's and the one with the robots.txt consistently ranked higher then the one without it.

Here is a good example to use:

----text begins---

Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: *
Allow: /all
Allow: /

User-agent: Mediapartners-Google
Allow: /all
Allow: /

User-agent: Googlebot
Allow: /all
Allow: /

User-agent: Adsbot-Google
Allow: /all
Allow: /

User-agent: Googlebot-Mobile
Allow: /all
Allow: /

---end text---

Simply copy that to notepad, and FTP it to your website...it does help!

[abolabo]

good post!
Thank you!

[llegrand]

Hi Gordon,

Considering the various levels of knowledge of all of the users,  please add to your instructions where they need to add the robot.txt file.

Thanks
Lee

[gordontaylor]

Lee, I just use what I posted. Since Google is the most important Search Engine and the other index Google, I target Google.

Copy that text to a notepad, save it as robots.txt and upload it to your root directory. i.e. http://yourwebsite.com/robots.txt

All the bots look for that file, no link is necessary on the home page or in the site map.

Also see this post http://forum.abantecart.com/index.php/topic,1662.msg6494.html#msg6494 for Facebook SEO tips and tricks

[DavidLIR]

Gordon,
I just found this post...I am trying this in robots.txt file to see if it makes a difference.  I just want to make sure I am doing it in the right place...since my website is forwarded to mywebsite.com/store...is the robots.txt file still in the root directory?

[gordontaylor]

Yes, David...root directory

[DavidLIR]

Gordon,
I did the robots.txt the way you said to do it...I also found there some other stuff which I will post here...should it be there as well?

User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

Thanks,
David

[gordontaylor]

Yes David, those are a good idea.

[DavidLIR]

Gordon,

Does it matter where this is put? meaning above or below the other text you suggested to be there?

and what am I telling it to do with this?
User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

David

[gordontaylor]

No difference David, anywhere you like.

[DavidLIR]

Great Thanks

[edwin324]

Thanks for sharing, if any one still confused with robots.txt rules then refer below URL and upload the .txt file to your root folder.
 
robotstxt.org/robotstxt.html

[paulsimmons]

You don't have to enter the allow commands. By default it is allowed for Google to crawl and index whole website. You only need to mention the disallow code to stop crawlers from indexing unwanted URLs like customer accounts, admin urls etc.

[Advisor]

Don't block important directories like IMG / image dirs... or your products will not show up in Google Images

[Natashawilliams]

nice post! thank you for sharing information about robots.txt