How to Create, Write and Install Robots.txt with cPanel or FTP

July 27th, 2016. Posted by Solvid.

How to create and install robots.txt file

Robots.txt is a text file on the web server of the website that allows to control search engine bots and the way they interact with the site. Robots.txt is the first file that search bots access when attempting to crawl the website.

1. Creating Robots.txt

Step 1: Open a simple text editor, such as TextEditor or Notepad. Name the file robots.txt and select the location where you want the file to be saved.

Creating Robots.txt file Step 1

Step 2: Start the file with the first line. User-agent line specifies directions to all or a specific search bot.

  • When “*” is added after the User-agent line, the rule will apply to all search engine bots.
  • If you want the rule to only apply to a specific search bot, replace “*” with the name of the bot, for example Googlebot.

Creating Robots.txt file Step 2

Step 3: Decide what folders and files you don’t want to get indexed.

  • If you don’t want to block any of the folder or files, add the following line: Disallow:
  • If you want to block the whole folder from being indexed, add this line: Disallow: /myfolder/
  • If you want to block a specific file, add this line: Disallow: /myfile
  • In the example above, /wp-admin/ folder has been blocked from indexing by all search bots.

Creating Robots.txt file Step 3

2. Installing Robots.txt with CPanel

Step 1: Access your cPanel File Manager and choose the main site directory. Then, simply click on “Upload” button and upload your robots.txt file.

  • Alternatively, create a new robots.txt file within the cPanel File Manager by clicking on the “New File” button.

step-1-cpanel-uploading

Step 2: Go back to the cPanel File Manager. Go back to the main site directory and you will notice that your robots.txt file has been uploaded.

Installing Robots.txt with CPanel Step 2

3. Alternative Installation with FTP 

Step 1: Connect to the FTP. Connect to your website server using an FTP (File Transfer Protocol), such as FileZilla.

Installing Robots.txt with FTP Step 1

Step 2: Locate the main directory and upload. Navigate to the main website directory and simply drag and drop your robots.txt file.

Installing Robots.txt with FTP Step 2

For more information and rules about robots.txt, please read our Beginners Guide to Robots.txt 

Check backlinks, website traffic, organic keywords, keyword traffic & difficulty, CPC and more.

Dmytro Spilka on FacebookDmytro Spilka on InstagramDmytro Spilka on LinkedinDmytro Spilka on Twitter
Dmytro Spilka
Head Wizard at Solvid
Head Wizard at Solvid & Founder of Solvid Online Tools. Contributor for The Huffington Post, The Next Web, SEMRush, Lifehack, MyCustomer and more.
2016-12-08T21:46:03+00:00 September 9th, 2016|
SUBSCRIBE TO OUR NEWSLETTER!

SUBSCRIBE TO OUR NEWSLETTER!

Join our mailing list to receive the latest news and updates from our blog.

You have Successfully Subscribed!

Pin It on Pinterest