|
|
|
Creating Sites That Work For You
Let me preface this article by saying that I am not an Internet millionaire, but I have learned enough about the Internet to make a living doing what I like to do, that is creating and running successful web sites. There are a few principles that...
How to choose a Hosting Web Package Cheap
If you're looking for a hosting web package cheap there are many things to consider. To find a hosting web package cheap you really need to do your homework and search the internet for the right host for you. But beware, just because a company might...
Is your web site hosting cost effective?
Choosing the right web hosting company is an important aspect in cutting your web hosting costs down. The web hosting industry, boosting several thousand web hosts, is still young and growing at a remarkable pace; hence finding the right host is not...
Nine Indisputable Rules for Creating A Killer Website
Nine Indisputable Rules for Creating A Killer Website by Caterina Christakos 1) Keep it simple. Movies, audio and animation may be cool but if it takes your user more than 10 seconds to download they are gone. 2) Make it easy to navigate. If...
Testing and Tracking Your Marketing Strategies
Developing and implementing a strategic marketing plan is an essential part of your success. However, unless you're testing and tracking your strategies, you may be losing a great deal of time and money. You can dramatically increase your sales...
|
|
| |
|
|
|
|
|
|
The proper way to use the robots.txt file
When optimizing your web site most webmasters don’t consider using the robots.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder or content that they can not index.
Here is a list of variables that you can include in a robots.txt file and there meaning:
1)User-agent: In this field you can specify a specific robot to describe access policy for or a “*” for all robots more explained in example.
2)Disallow: In the field you specify the files and folders not to include in the crawl.
3)# the number sign represents comments
Here are some examples
of a robots.txt file for redball.com
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another example
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.
About the Author
Jimmy Whisenhunt is the owner of VIP Enterprises
|
|
|
|
|
|