web spider

Magento Web Crawler

web spider

What is Web Crawler?

A Web Crawler is a Automated script which browses the web in automated manner. we can call it as web crawler or web spider or bot.

Why Web Crawler?

The Web crawler visits the web sites and fetch the pages and other information to create the entries for the search index. web crawling is specially used to discover publically available web pages.

google bot is the most well know web crawler. google bot will read line by line in your web page and follow the links on those pages. it will reads each and every links and fetch in to the google servers.

Crawling the Web:

The crawler begins to crawl the web site based on the past crawl and also based on the site map provided by the store/site owner. store owners have many choices to handle the crawler for their sites. for google bot, we can able to handle the crawling by the help of google web masters and robots.txt file.

 What is Robots.txt File?

Store/site owner can have an option to restrict the crawling from the robots.txt file. robots.txt is just a simple txt file exists in web root folder. web robots reads the robots.txt file and view the content of each web page and links. with the help of robots.txt file, we can control what are the files we need to crawl so it used to prevent the directives being crawled by the crawler. so it helps to avoid indexing the duplicated page for the purpose of SEO. we can configure the delay of each crawling in robots.txt file. it helps to avoid keep on indexing the page and also it reduces the bandwidth of the web server. in robots.txt file we can allow or disallow the specific bots. so it helps to avoid crawling the site by the bad/spam crawler.

in the next post we will see how to generate the robots.txt file and how to handle the robots.txt file to restrict specific crawlers.


  1. I read a lot of interesting articles here. Probably you
    spend a lot of time writing.

  2. Hey there I am so delighted I found your web site, I
    really found you by accident, while I was researching on Aol
    for something else, Regardless I am here now and would just like to say thanks a lot for a
    tremendous post and a all round entertaining blog (I
    also love the theme/design), I don’t have time to read through it all
    at the moment but I have saved it and also added your RSS feeds, so when I have time I will be back to
    read more, Please do keep up the great work.

    1. Mohan Natarajan October 17, 2014 at 5:18 pm

      Thank you Arthur for your valuable feedback.

  3. Wonderful items from you, man. I’ve keep in mind your stuff prior
    to and you are just extremely magnificent. I really like what you’ve acquired
    here, certainly like what you’re saying and the
    way in which by which you assert it. You make it entertaining and you still take care of to
    stay it sensible. I can not wait to read far more from you.

    This is really a terrific web site.

  4. I’m now not certain where you’re getting your info, however
    great topic. I must spend a while studying much more or working out
    more. Thank you for excellent information I was in search of this information for my mission.

  5. Hello there, I believe your web site may be having web browser compatibility issues.
    When I look at your blog in Safari, it looks fine however, if opening in I.E., it has some overlapping issues.
    I merely wanted to give you a quick heads up! Other than that,
    excellent blog!

    1. Mohan Natarajan October 20, 2014 at 1:09 pm

      Thank you for your notification. I will check it in IE and fix it. Thank you again.

  6. I just like the valuable information you
    supply to your articles. I’ll bookmark your blog and test again here
    regularly. I am quite certain I’ll learn plenty of new stuff right right here!
    Best of luck for the following!

  7. Hello there!I could have sworn I’ve visited this site before but after looking at some of the articles I realized it’s new to me.
    Anyhow, I’m certainly happy I discovered it and I’ll be book-marking it and checking back regularly!

  8. It’s actually a great and useful piece of information. I’m glad that you shared this helpful info with us. Please keep us up to date like this. Thanks for sharing.

  9. Greetings! Very useful advice in this particular article!
    It is the little changes that produce the biggest changes.
    Many thanks for sharing!

  10. Hi there would you mind sharing which blog platform
    you’re working with? I’m looking to start my own blog soon but I’m having a difficult
    time deciding between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your design and style seems different then most blogs and I’m looking for something completely unique.

    P.S Sorry for being off-topic but I had to ask!

Leave a Comment

Your email address will not be published. Required fields are marked *

− 1 = 4