This post is very important, it belongs to search engine optimization. I am discussing about robots txt as mentioned on the post title. Here we will know what robots txt is and how to use it in website and blog. I will tell you what robots txt is and how to use it in WordPress means in website and blogger means in blog.
Robots txt is a text file. It is used to add your indexing information. Using this, you can tell to Google, Which separate URLs you want to share with Google in Google search results. And you can add sitemaps here to provide more detail and URL information to Google to index your website or blog in Google search results. I am using Google because it is a top search engine. But all other search engine has the same condition about robots txt as mentioned in this post with the name of Google.
Contents we will cover in this post.
Let’s see the robots.txt file and understand it.
User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /readme.html Sitemap: https://www.jekeshkumaroad.com/post-sitemap.xml Sitemap: https://www.jekeshkumaroad.com/page-sitemap.xml Sitemap: https://www.jekeshkumaroad.com/category-sitemap.xml
It connects you to search engine crawler. If you to make read your robots txt to a particular search engine crawler then you can change it. But * is used to make robots txt readable for all search engine crawlers.
- It is best to use User-Agent: *
- You site’s robots txt file will be readable by all search engines.
- Other User-Agents are extremely not recommended.
User-Agent: Googlebot User-Agent: Googlebot-Image User-Agent: Googlebot-News User-Agent: Yahoo! Slurp User-Agent: Bingbot
You can get further User-Agent at User Agent String.Com
This line is used then when we want to describe which directories, we want to essentially use. In Allow section we tell to Google, all this directory any how. No matter it is present in sitemap or not. Google will add that to search engine results without checking it in sitemap.xml file. So beware of using it.
This line is used to hide a particular directory from Google. If you put any URL or any directory address in Disallow section, it will never display in Google search results. Normally, we should not use it. Only expert and a man, who know this in detail, can use it. Misuse of it can destroy your website.
Here we put our sitemaps. Many of you know what sitemap is. If you don’t know. You can read our article about sitemap.
Sitemap is a file which has set of URLs of your website, according to your permission. It’s up to you. Which URLs you want to add in sitemap file.
Sitemap file extension/format .xml usually.
Add Robots txt to WordPress
Adding robots txt to WordPress is very easy and simple. It is same as in websites. You can use same robots txt for WordPress and php/html website. Before placing the robots.txt file. It is good to add robots txt file in WordPress. It add more exposure to your website and accessible by many search engines. Using this many search engine crawlers crawl your website and add it to their search engine results without submitting sitemap. It means my website and it’s sitemaps is only submitted to Google search engine manually by myself. But my website showing in different search engines. Because they read my robots.txt file.
Here I will define two methods.
- Add robots txt using yoast
- Add robots txt manually
Add robots txt using Yoast
Yoast is WordPress plugin, which is currently the best plugin for Search Engine Optimization. Know more about it at our Review about Yoast.
- Install Yoast SEO plugin
- Go to SEO >> Tools >> File Editor
- Create robots.txt file, paste the above code and save.
Must replace your domain name.
Add robots txt manually
- Open notepad.
- Paste the above robots txt and save as robots.txt .
- Upload it to domain root directory using cpanel.
- It should look like this.
How to add Robots txt to Blogger
- Go to Blogger.com
- Sign in with your gmail account.
- Select Blog, in which you want to add robots txt.
- Go to Settings >> Search Preferences
- Edit Custom robots.txt (Currently Disabled)
- Enable that by clicking Yes.
- Paste the following code.
- Replace the blog address and save.
User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: https://alptia.blogspot.com/feeds/posts/default?orderby=UPDATED Sitemap: https://alptia.blogspot.com/sitemap.xml
If you still feel any difficulty, you can ask in comments.
If you are a blogger user. Then this article belongs to you. Free Blogger Templates Providers with Premium Support and Features.
Learn more about robots txt at:
More from jekesh kumar oad: