How can I make a robot so search engines can crawl some areas in my web page that have a security filter?

November 9, 2009

I want google and yahoo to index all the content on my site. The problem is I have a security filter (members only), but I still want search engines to index this pages.

Category Categories: Electronic Toys  |  Tag Tags: areas, crawl, Engines, filter, Page, Robot, Search, Security, Some  |  Comments 1 Comment »

1 comment

  1. onlyfor7dollar says:
    November 9, 2009 at 9:19 am

    try this:
    http://www.1-hit.com/all-in-one/tool-robots.txt-generator.htm

    http://www.mcanerin.com/EN/search-engine/robots-txt.asp

    you can just replace folder/file names and use it.
    http://www.nytimes.com/robots.txt

Leave a Reply