Update: 06th Oct. 2014
No one want to blog without a optimized Robot.txt file. In fact, every WordPress blogger should know a thing about the file and why it’s so important for every blog’s SEO.
- What is robots.txt file? & Why is it important?
Robots.txt is a special text file that stay in the root directory of your blog. It helps to block search engine bots to index and crawl important part of our blog. If you configured wrong Robots.txt file then your blog completely go away from search engines. So, it’s important when you make changes in your robots.txt file, it should be well optimized and should not block access to important part of your blog.
So, Here Bot is a special program used by search engines to find and add web pages to their search indices (databases). Google’s main crawler is called Googlebot and Bing’s crawler is called MSNbot, which is used to create the search index for both Bing and Yahoo.
- How to Add Robots.txt to your WordPress site?
Robots.txt file helps search engine robots to direct which part to crawl and which part to avoid. When Search bot or spider of Search Engine comes to your site and wants to index your site, they follow Robots.txt file first. So, you want to write a Robot.txt file that tell search engines bots to crawl and index your posts and pages, but not your WP files and directories. You also want to make sure that feeds and trackbacks aren’t included in the search results. And make sure that you should declare a sitemap. For add a Robot.txt file first go to Setting>Reading>Additional robots.txt rules.
Put this Optimize txt file. Just copy it & paste in your Additional robots.txt rules box.
Note: You can saw your Robot.txt file just by type ” http://www.YOURSITEURL.com/robots.txt
change YOUR SITEURL to your blog url.
Note: Change “YOUR URL” with your site url.
You can use online tools for check your robot.txt file. After check, if you found any error then you have to correct. Some online tools like:
- Frobee Robots.txt Cheacker
- Motoricerca Robot.txt Checker
It is important to check your robots.txt file. It should be analyzer before you add to your blog. This will ensure that you have not entered any lines incorrectly. You can check Matt Catts blog, he explain all in details.
By this Robot.txt file you will tell bots that Not index all your posts, pages, catagories, tags & aduio, vidio, image uploads.
We, hope you like the post, If you like plz share and like our post then share in social network so that it can viral & other people may also benefit from this post. Thanks!!!
Hope this Post is Help you! Be sure Check Our SITEMAP & SUBSCRIBE our Blog for Never miss any Update!!