How to setup Robots.txt file in Magento

robotx.txt

Magento is a great open source e-Commerce to improve the online business. But we need to do some basic SEO steps in order to improve the business to reach the audience.

The robots.txt file is helps to index your site by the web crawlers like google bot, bing bot, etc..

we can allow/disallow the specific section of your website in order for better indexing the site by the web robots.

How robots.txt Helps for SEO:

It helps to prevent indexing the error logs, reports, cvs/svn files. also it helps to indexing the duplicated content in your web site. so it helps to increase your google page rank.

The robots.txt file is works only for one domain name. if you run multiple magento store with single instance, then you need to set up different robots.txt for each domain.

Sample robots.txt file for your Magento store.

## Enable robots.txt rules for all crawlers
User-agent: *

## Don’t crawl development files and folders
Disallow: CVS
Disallow: .svn

## Don’t crawl Magento admin page
Disallow: /admin/

## Don’t crawl common Magento files
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
Disallow: /STATUS.txt

## Don’t crawl common Magento folders
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

## Don’t crawl the checkout and user account pages
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/

## Don’t crawl common server folders / files
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php

## Don’t allow Google and Bing to index your images
# User-agent: Googlebot-Image
# Disallow: /
# User-agent: msnbot-media
# Disallow: /
Sitemap: http://www.yourmagentostore.com/sitemap.xml

This is only the sample robot.txt file. you need to change the robot.txt file for your needs. you need to upload your robots.txt file inside your root directory of magento store.

If you are using google feeder, in order to access your product images you need to allow media folder to crawl your images.

in the next article we will see what is site map and how it helps to improve your SEO.

5 thoughts on “How to setup Robots.txt file in Magento

  1. Hi,

    Where to put the robots.txt file in Magento CE1.9? If i put in in public_html then when trying to access/show it like http://www.yourwebsitenamehere.com/robots.txt magento says: Service Temporarily Unavailable
    The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

    File persmissions are 755 on the server

    Baffles me…

Leave a Reply

Your email address will not be published. Required fields are marked *


× 3 = 3

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>