Robots txt (robots.txt) – How to add in WordPress and Blogger

Robots txt

This post is very important, it belongs to search engine optimization. I am discussing about robots txt as mentioned on the post title. Here we will know what robots txt is and how to use it in website and blog. I will tell you what robots txt is and how to use it in WordPress means in website and blogger means in blog.

Robots txt

Robots txt is a text file. It is used to add your indexing information. Using this, you can tell to Google, Which separate URLs you want to share with Google in Google search results. And you can add sitemaps here to provide more detail and URL information to Google to index your website or blog in Google search results. I am using Google because it is a top search engine. But all other search engine has the same condition about robots txt as mentioned in this post with the name of Google.

Understand Robots.txt

Let’s see the robots.txt file and understand it.

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /readme.html

Sitemap: https://www.jekeshkumaroad.com/post-sitemap.xml
Sitemap: https://www.jekeshkumaroad.com/page-sitemap.xml
Sitemap: https://www.jekeshkumaroad.com/category-sitemap.xml

Understand

User-Agent: *

It connects you to search engine crawler. If you to make read your robots txt to a particular search engine crawler then you can change it. But * is used to make robots txt readable for all search engine crawlers.

  1. It is best to use User-Agent: *
  2. You site’s robots txt file will be readable by all search engines.
  3. Other User-Agents are extremely not recommended.

Example

User-Agent: Googlebot
User-Agent: Googlebot-Image
User-Agent: Googlebot-News
User-Agent: Yahoo! Slurp
User-Agent: Bingbot

You can get further User-Agent at User Agent String.Com

Allow: /wp-content/uploads/

This line is used then when we want to describe which directories, we want to essentially use. In Allow section we tell to Google, all this directory any how. No matter it is present in sitemap or not. Google will add that to search engine results without checking it in sitemap.xml file. So beware of using it.

Disallow: /wp-content/plugins/

This line is used to hide a particular directory from Google. If you put any URL or any directory address in Disallow section, it will never display in Google search results. Normally, we should not use it. Only expert and a man, who know this in detail, can use it. Misuse of it can destroy your website.

Sitemap: https://www.jekeshkumaroad.com/post-sitemap.xml

Here we put our sitemaps. Many of you know what sitemap is. If you don’t know. You can read our article about sitemap.

Sitemap is a file which has set of URLs of your website, according to your permission. It’s up to you. Which URLs you want to add in sitemap file.

Sitemap file extension/format .xml usually.

Add Robots txt to WordPress

WordPress
WordPress

Adding robots txt to WordPress is very easy and simple. It is same as in websites. You can use same robots txt for WordPress and php/html website. Before placing the robots.txt file. It is good to add robots txt file in WordPress. It add more exposure to your website and accessible by many search engines. Using this many search engine crawlers crawl your website and add it to their search engine results without submitting sitemap. It means my website and it’s sitemaps is only submitted to Google search engine manually by myself. But my website showing in different search engines. Because they read my robots.txt file.

Here I will define two methods.

  1. Add robots txt using yoast
  2. Add robots txt manually

Add robots txt using Yoast

Yoast
Yoast

Yoast is WordPress plugin, which is currently the best plugin for Search Engine Optimization. Know more about it at our Review about Yoast.

  1. Install Yoast SEO plugin
  2. Go to SEO >> Tools >> File Editor
  3. Create robots.txt file, paste the above code and save.

Must replace your domain name.

Add robots txt manually

  1. Open notepad.
  2. Paste the above robots txt and save as robots.txt .
  3. Upload it to domain root directory using cpanel.
  4. It should look like this.

https://www.jekeshkumaroad.com/robots.txt

How to add Robots txt to Blogger

Blogger
Blogger
  1. Go to Blogger.com
  2. Sign in with your gmail account.
  3. Select Blog, in which you want to add robots txt.
  4. Go to Settings >> Search Preferences
  5. Edit Custom robots.txt (Currently Disabled)
  6. Enable that by clicking Yes.
  7. Paste the following code.
  8. Replace the blog address and save.
User-agent: Mediapartners-Google

Disallow:

User-agent: *

Disallow: /search

Allow: /

Sitemap: https://alptia.blogspot.com/feeds/posts/default?orderby=UPDATED
Sitemap: https://alptia.blogspot.com/sitemap.xml

If you still feel any difficulty, you can ask in comments.

If you are a blogger user. Then this article belongs to you. Free Blogger Templates Providers with Premium Support and Features.

Learn more about robots txt at:

  1. MOZ
  2. robotstxt.org
  3. Google

More from jekesh kumar oad:

  1. WordPress Tutorials
  2. Blogger Tips and Tricks
  3. Search Engine Optimization
Robots txt (robots.txt) – How to add in WordPress and Blogger
4.2 (83.33%) 6 votes

Popular on Jekesh Kumar Oad Right Now!

Leave a Reply

Your email address will not be published. Required fields are marked *