Robots.txt common mistakesThere is on the internet some robots who doesn't care about your content and website but care about the emails in your webpages. If you have a page called "contact.html" and all the emails are gathered arround inside of this page, may be those robots will find the nice amounts of emails and tomorow you'll receive some news on how you can brush your teeth with a beaver! Those robots are the root of spams. They take your emails, make a nice list everyday. And after that the user spammer take the list and spam with stupids advertisments into your mail box. Common Robots.txt Mistakes1. The most common mistakes is backwards syntax:User-agent: * User-agent: apple 2. One directory at a time:User-agent: Googlebot This will not work at all and you should correct the syntax into: User-agent: Googlebot Be careful with the User-agent also, do not align plenty of robots in the User-agent command. 3. Avoid listing your private/secret directories:Everybody can access your robots.txt file and not only the spiders, you can check on many websites such as google, symantec etc... If you list a directory in a robots.txt file attracts the attention to the directory. And some spiders like i said previously can find out those directories. And give the list to the spammers. 4. Do not create many robots.txt file in different folder:One of my friends have a website and asked me to check his website for mistakes in seo and design. He told me he has a robots.txt file in all of his folders to lead the spiders. I told him no way, you've to use just one robots.txt, you just can't use many at the same time. This would be extremly confusing for the robots. 5.a text file will be fine. robots.txt. No extensions such as .doc, .xls etc... |
|
MenuHomeNews Blog Free phpLD Templates Free Web Tools Site Map Add Your Site Resources Spammers List Seo Guide Seo Glossary Web DirectoriesIndexDMOZ Yahoo! Looksmart Best Paid Web Directories Best Web Directories Search EnginesIndexAOL Ask Jeeves Excite Hotbot Altavista Lycos Teoma SEO TutorialsGoogle OptimizationYahoo! Optimization Meta Tags Optimization Keywords Optimization Link Popularity Tutorials Domain Name TLDs Create A Robots.txt file |
Network: Home | Web Tools | Link to Us | Seo Blog | Site Map | Resources | Web Design Services |
Irkawebpromotions 2005 - 2011 © - All Rights Reserved. |