How to Create a Robots.txt file tutorial
Definition of a Robots.txt file:
A robot index websites, check link popularity on websites, and judge websites whether it contains relevant content with targeted keywords or not. A robot is also called a Crawler or a Spider, they have the same meaning.
A robot is a computer program operated by a search engine, a research organization, a University or an individual. Everytime a search engines robot look into your root domain a file named "robots.txt" usually at the address: 'http://www.mydomain.com/robots.txt'.
This robots.txt file tell robots what files or folders they have to index on your website. We call that system the "Robots Exclusion Standard".
Here is a list of common mistakes used in a robots.txt file.
The robots.txt file looks like that:
# mydomain.com robots.txt:
User-agent is the name of the robots, spiders, crawlers that visit your website.
User-agent: googlebot The rules only apply to googlebot spider
the disallow command keep out robots from certain files or folders.
Disallow: /content/ Robots cannot index the folder content
There is no Allow command as you understood it! If it was so it means you have to write down all the folders and files you want the robots to index.
The comments tag
You can also make comments in a robots.txt file like the following examples:
Here is another robots.txt example:
# go away
This example is the best, all robots are disallowed to crawl the website.
Let's make some more examples and explain about it
+ Here the targeted robot is inktomislurp and is not allowed to index the folder cgi-bin and forum and also not allowed to index the file taxes.asp from the folder data.
This rules only apply to the robot Inktomislurp.
+ All robots are disallowed to enter the folder 4gettobuybread and common. And are disallowed to access the files 404.jsp, tips.jsp, index.jsp, site_index.jsp.
+ It restricts googlebot from indexing all PDF, JEPG and EXE files.
Next page: Robots.txt common mistakes
Free phpLD Templates
Free Web Tools
Add Your Site
Best Paid Web Directories
Best Web Directories
SEO TutorialsGoogle Optimization
Meta Tags Optimization
Link Popularity Tutorials
Domain Name TLDs
Create A Robots.txt file
|Network: Home | Web Tools | Link to Us | Seo Blog | Site Map | Resources | Web Design Services|
|Irkawebpromotions 2005 - 2011 © - All Rights Reserved.|