AbanteCart Community

General Topics => SEO => Topic started by: gordontaylor on January 23, 2014, 10:30:27 PM

Title: Robots.txt
Post by: gordontaylor on January 23, 2014, 10:30:27 PM
I know this is old school, but do you realize SE's still look for this file?

robots.txt

I have proven it gets better rankings by publishing two identical sites with similar URL's and the one with the robots.txt consistently ranked higher then the one without it.

Here is a good example to use:

----text begins---

Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: *
Allow: /all
Allow: /

User-agent: Mediapartners-Google
Allow: /all
Allow: /

User-agent: Googlebot
Allow: /all
Allow: /

User-agent: Adsbot-Google
Allow: /all
Allow: /

User-agent: Googlebot-Mobile
Allow: /all
Allow: /

---end text---

Simply copy that to notepad, and FTP it to your website...it does help!
Title: Re: Robots.txt
Post by: abolabo on January 24, 2014, 03:45:26 AM
good post!
Thank you!
Title: Re: Robots.txt
Post by: llegrand on January 24, 2014, 09:14:43 AM
Hi Gordon,

Considering the various levels of knowledge of all of the users,  please add to your instructions where they need to add the robot.txt file.

Thanks
Lee
Title: Re: Robots.txt
Post by: gordontaylor on January 24, 2014, 12:12:32 PM
Lee, I just use what I posted. Since Google is the most important Search Engine and the other index Google, I target Google.

Copy that text to a notepad, save it as robots.txt and upload it to your root directory. i.e. http://yourwebsite.com/robots.txt

All the bots look for that file, no link is necessary on the home page or in the site map.

Also see this post http://forum.abantecart.com/index.php/topic,1662.msg6494.html#msg6494 (http://forum.abantecart.com/index.php/topic,1662.msg6494.html#msg6494) for Facebook SEO tips and tricks
Title: Re: Robots.txt
Post by: DavidLIR on March 23, 2014, 04:56:27 PM
Gordon,
I just found this post...I am trying this in robots.txt file to see if it makes a difference.  I just want to make sure I am doing it in the right place...since my website is forwarded to mywebsite.com/store...is the robots.txt file still in the root directory?
Title: Re: Robots.txt
Post by: gordontaylor on March 23, 2014, 05:47:04 PM
Yes, David...root directory
Title: Re: Robots.txt
Post by: DavidLIR on March 29, 2014, 02:08:30 PM
Gordon,
I did the robots.txt the way you said to do it...I also found there some other stuff which I will post here...should it be there as well?

User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

Thanks,
David
Title: Re: Robots.txt
Post by: gordontaylor on March 29, 2014, 03:12:49 PM
Yes David, those are a good idea.
Title: Re: Robots.txt
Post by: DavidLIR on March 29, 2014, 05:06:19 PM
Gordon,

Does it matter where this is put? meaning above or below the other text you suggested to be there?

and what am I telling it to do with this?
User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

David
Title: Re: Robots.txt
Post by: gordontaylor on March 29, 2014, 05:23:04 PM
No difference David, anywhere you like.
Title: Re: Robots.txt
Post by: DavidLIR on March 30, 2014, 03:23:26 PM
Great Thanks
Title: Re: Robots.txt
Post by: edwin324 on November 13, 2014, 09:19:42 AM
Thanks for sharing, if any one still confused with robots.txt rules then refer below URL and upload the .txt file to your root folder.
 
robotstxt.org/robotstxt.html
Title: Re: Robots.txt
Post by: paulsimmons on December 10, 2014, 05:16:32 AM
You don't have to enter the allow commands. By default it is allowed for Google to crawl and index whole website. You only need to mention the disallow code to stop crawlers from indexing unwanted URLs like customer accounts, admin urls etc.
Title: Re: Robots.txt
Post by: Advisor on December 13, 2014, 10:49:10 PM
Don't block important directories like IMG / image dirs... or your products will not show up in Google Images
Title: Re: Robots.txt
Post by: Natashawilliams on September 24, 2016, 03:02:06 AM
nice post! thank you for sharing information about robots.txt