News:

AbanteCart v1.4.2 is released.

Main Menu

Robots.txt

Started by gordontaylor, January 23, 2014, 10:30:27 PM

Previous topic - Next topic

gordontaylor

I know this is old school, but do you realize SE's still look for this file?

robots.txt

I have proven it gets better rankings by publishing two identical sites with similar URL's and the one with the robots.txt consistently ranked higher then the one without it.

Here is a good example to use:

----text begins---

Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: *
Allow: /all
Allow: /

User-agent: Mediapartners-Google
Allow: /all
Allow: /

User-agent: Googlebot
Allow: /all
Allow: /

User-agent: Adsbot-Google
Allow: /all
Allow: /

User-agent: Googlebot-Mobile
Allow: /all
Allow: /

---end text---

Simply copy that to notepad, and FTP it to your website...it does help!

abolabo

"No one is useless in this world who lightens the burdens of another."
― Charles Dickens

llegrand

Hi Gordon,

Considering the various levels of knowledge of all of the users,  please add to your instructions where they need to add the robot.txt file.

Thanks
Lee

gordontaylor

Lee, I just use what I posted. Since Google is the most important Search Engine and the other index Google, I target Google.

Copy that text to a notepad, save it as robots.txt and upload it to your root directory. i.e. http://yourwebsite.com/robots.txt

All the bots look for that file, no link is necessary on the home page or in the site map.

Also see this post http://forum.abantecart.com/index.php/topic,1662.msg6494.html#msg6494 for Facebook SEO tips and tricks

DavidLIR

Gordon,
I just found this post...I am trying this in robots.txt file to see if it makes a difference.  I just want to make sure I am doing it in the right place...since my website is forwarded to mywebsite.com/store...is the robots.txt file still in the root directory?
Love is Real....All Else is Illusion

gordontaylor

Yes, David...root directory

DavidLIR

Gordon,
I did the robots.txt the way you said to do it...I also found there some other stuff which I will post here...should it be there as well?

User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

Thanks,
David
Love is Real....All Else is Illusion

gordontaylor

Yes David, those are a good idea.

DavidLIR

Gordon,

Does it matter where this is put? meaning above or below the other text you suggested to be there?

and what am I telling it to do with this?
User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

David
Love is Real....All Else is Illusion

gordontaylor

No difference David, anywhere you like.

DavidLIR

Love is Real....All Else is Illusion

edwin324

Thanks for sharing, if any one still confused with robots.txt rules then refer below URL and upload the .txt file to your root folder.

robotstxt.org/robotstxt.html

paulsimmons

You don't have to enter the allow commands. By default it is allowed for Google to crawl and index whole website. You only need to mention the disallow code to stop crawlers from indexing unwanted URLs like customer accounts, admin urls etc.

Advisor

Don't block important directories like IMG / image dirs... or your products will not show up in Google Images

Natashawilliams

nice post! thank you for sharing information about robots.txt

Forum Rules Code of conduct
AbanteCart.com 2010 -