Oxygencreative

robotstxt file for domain httpwwwTHIS DOMAINcomUseragentDisallow cgibinDisallow logstatsDisallow mailphpDisallow imagesDisallow scriptsDisallow stylesDisallow javajsDisallow cartjsDisallow validatejsDisallow emailCheckjsDisallow ccheckjsDisallow mapjsDisallow setupjsDisallow navbarsjsDisallow layerjsDisallow mailphpDisallow orderphpDisallow searchincDisallow layersincDisallow linksinc Following text taken from wwwrobotstxtorg Modified by howardfreetimerscomNote that you need a separate Disallow line for every URL prefix you want to exclude you cannot say Disallow cgibin tmpNote also that only 1 robotstxt may exist on a server and it must be in the root directory eg httpwwwthisdomaincomrobotstxt not httpwwwthisdomaincomsomedirrobotstxtAlso you may not have blank lines in a record as they are used to delimit multiple records Note also that regular expression are not supported in either the Useragent or Disallow linesThe in the Useragent field is a special value meaning any robotEG You cannot have lines like Disallow tmp or Disallow gif IMPORTANTDo not put sensitive docs in robotstxtAnyone can request robotstxt and find out which files we dont want people seeingKind of defeats the object slightly but lets face it search engines only follow linksIf its that sensitive whats it doing onlineAlso do not disallow robotstxt in either robotstxt or htaccessRemember as well that robots do not have to obey robotstxt most eg google altavista etc do butif youre being spidered by an email bot its gonna rip through everything it canEverything not explicitly disallowed is considered fair game to retrieve Here follow some examples To exclude all robots from the entire serverUseragent Disallow To allow all robots complete accessOr just dont have a robotstxt file will generate 404s thoughUseragent DisallowTo exclude all robots from part of the serverUseragent Disallow cgibinDisallow tmpDisallow privateTo exclude a single robotUseragent BadBotDisallow To allow only a single robotUseragent WebCrawlerDisallowUseragent Disallow To exclude all files except oneThis is currently a bit awkward as there is no Allow fieldThe easy way is to put all files to be disallowed into a separate directory say joedocsand leave the one file in the level above this directory Useragent Disallow joedocsAlternatively you can explicitly disallow all disallowed pages Useragent Disallow joeprivatehtmlDisallow joefoohtmlDisallow joebarhtml