General Topics > SEO

Robots.txt

(1/3) > >>

gordontaylor:
I know this is old school, but do you realize SE's still look for this file?

robots.txt

I have proven it gets better rankings by publishing two identical sites with similar URL's and the one with the robots.txt consistently ranked higher then the one without it.

Here is a good example to use:

----text begins---

Disallow: /cgi-bin/
Disallow: /tmp/

User-agent: *
Allow: /all
Allow: /

User-agent: Mediapartners-Google
Allow: /all
Allow: /

User-agent: Googlebot
Allow: /all
Allow: /

User-agent: Adsbot-Google
Allow: /all
Allow: /

User-agent: Googlebot-Mobile
Allow: /all
Allow: /

---end text---

Simply copy that to notepad, and FTP it to your website...it does help!

abolabo:
good post!
Thank you!

llegrand:
Hi Gordon,

Considering the various levels of knowledge of all of the users,  please add to your instructions where they need to add the robot.txt file.

Thanks
Lee

gordontaylor:
Lee, I just use what I posted. Since Google is the most important Search Engine and the other index Google, I target Google.

Copy that text to a notepad, save it as robots.txt and upload it to your root directory. i.e. http://yourwebsite.com/robots.txt

All the bots look for that file, no link is necessary on the home page or in the site map.

Also see this post http://forum.abantecart.com/index.php/topic,1662.msg6494.html#msg6494 for Facebook SEO tips and tricks

DavidLIR:
Gordon,
I just found this post...I am trying this in robots.txt file to see if it makes a difference.  I just want to make sure I am doing it in the right place...since my website is forwarded to mywebsite.com/store...is the robots.txt file still in the root directory?

Navigation

[0] Message Index

[#] Next page