Author Topic: Robots.txt  (Read 11556 times)

gordontaylor

  • Guest
Robots.txt
« on: January 23, 2014, 10:30:27 PM »
I know this is old school, but do you realize SE's still look for this file?

robots.txt

I have proven it gets better rankings by publishing two identical sites with similar URL's and the one with the robots.txt consistently ranked higher then the one without it.

Here is a good example to use:

----text begins---

Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: *
Allow: /all
Allow: /

User-agent: Mediapartners-Google
Allow: /all
Allow: /

User-agent: Googlebot
Allow: /all
Allow: /

User-agent: Adsbot-Google
Allow: /all
Allow: /

User-agent: Googlebot-Mobile
Allow: /all
Allow: /

---end text---

Simply copy that to notepad, and FTP it to your website...it does help!

Offline abolabo

  • core-developer
  • Administrator
  • Hero Member
  • *****
  • Posts: 1711
  • Karma: +227/-10
  • web for all, all for web!
    • View Profile
    • AbanteCart
Re: Robots.txt
« Reply #1 on: January 24, 2014, 03:45:26 AM »
good post!
Thank you!
“No one is useless in this world who lightens the burdens of another.”
― Charles Dickens

Offline llegrand

  • Hero Member
  • *****
  • Posts: 1313
  • Karma: +332/-2
    • View Profile
Re: Robots.txt
« Reply #2 on: January 24, 2014, 09:14:43 AM »
Hi Gordon,

Considering the various levels of knowledge of all of the users,  please add to your instructions where they need to add the robot.txt file.

Thanks
Lee

gordontaylor

  • Guest
Re: Robots.txt
« Reply #3 on: January 24, 2014, 12:12:32 PM »
Lee, I just use what I posted. Since Google is the most important Search Engine and the other index Google, I target Google.

Copy that text to a notepad, save it as robots.txt and upload it to your root directory. i.e. http://yourwebsite.com/robots.txt

All the bots look for that file, no link is necessary on the home page or in the site map.

Also see this post http://forum.abantecart.com/index.php/topic,1662.msg6494.html#msg6494 for Facebook SEO tips and tricks

Offline DavidLIR

  • Jr. Member
  • **
  • Posts: 86
  • Karma: +7/-0
    • View Profile
    • Love is Real
Re: Robots.txt
« Reply #4 on: March 23, 2014, 04:56:27 PM »
Gordon,
I just found this post...I am trying this in robots.txt file to see if it makes a difference.  I just want to make sure I am doing it in the right place...since my website is forwarded to mywebsite.com/store...is the robots.txt file still in the root directory?
Love is Real....All Else is Illusion

gordontaylor

  • Guest
Re: Robots.txt
« Reply #5 on: March 23, 2014, 05:47:04 PM »
Yes, David...root directory

Offline DavidLIR

  • Jr. Member
  • **
  • Posts: 86
  • Karma: +7/-0
    • View Profile
    • Love is Real
Re: Robots.txt
« Reply #6 on: March 29, 2014, 02:08:30 PM »
Gordon,
I did the robots.txt the way you said to do it...I also found there some other stuff which I will post here...should it be there as well?

User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

Thanks,
David
Love is Real....All Else is Illusion

gordontaylor

  • Guest
Re: Robots.txt
« Reply #7 on: March 29, 2014, 03:12:49 PM »
Yes David, those are a good idea.

Offline DavidLIR

  • Jr. Member
  • **
  • Posts: 86
  • Karma: +7/-0
    • View Profile
    • Love is Real
Re: Robots.txt
« Reply #8 on: March 29, 2014, 05:06:19 PM »
Gordon,

Does it matter where this is put? meaning above or below the other text you suggested to be there?

and what am I telling it to do with this?
User-agent: *
Disallow: /admin/
Disallow: /core/
Disallow: /download/
Disallow: /system/

David
Love is Real....All Else is Illusion

gordontaylor

  • Guest
Re: Robots.txt
« Reply #9 on: March 29, 2014, 05:23:04 PM »
No difference David, anywhere you like.

Offline DavidLIR

  • Jr. Member
  • **
  • Posts: 86
  • Karma: +7/-0
    • View Profile
    • Love is Real
Re: Robots.txt
« Reply #10 on: March 30, 2014, 03:23:26 PM »
Great Thanks
Love is Real....All Else is Illusion

Offline edwin324

  • Newbie
  • *
  • Posts: 1
  • Karma: +1/-0
  • hello world
    • View Profile
    • Uni-eMerchant | Uniecommerce
Re: Robots.txt
« Reply #11 on: November 13, 2014, 09:19:42 AM »
Thanks for sharing, if any one still confused with robots.txt rules then refer below URL and upload the .txt file to your root folder.
 
robotstxt.org/robotstxt.html

Offline paulsimmons

  • Newbie
  • *
  • Posts: 2
  • Karma: +0/-0
  • Professional PHP Developer
    • View Profile
    • FmeAddons
Re: Robots.txt
« Reply #12 on: December 10, 2014, 05:16:32 AM »
You don't have to enter the allow commands. By default it is allowed for Google to crawl and index whole website. You only need to mention the disallow code to stop crawlers from indexing unwanted URLs like customer accounts, admin urls etc.

Offline Advisor

  • Jr. Member
  • **
  • Posts: 53
  • Karma: +7/-4
    • View Profile
Re: Robots.txt
« Reply #13 on: December 13, 2014, 10:49:10 PM »
Don't block important directories like IMG / image dirs... or your products will not show up in Google Images

Offline Natashawilliams

  • Newbie
  • *
  • Posts: 13
  • Karma: +1/-0
  • I am an HR Professional
    • View Profile
    • Bulk SMS Gateway
Re: Robots.txt
« Reply #14 on: September 24, 2016, 03:02:06 AM »
nice post! thank you for sharing information about robots.txt

 

LoliPromDress.com Prom dresses 2018