How to Create, Write and Install Robots.txt with cPanel or FTP

By Dmytro Spilka

Sep 9, 2016

How to create and install robots.txt file

Dmytro Spilka

Robots.txt is a text file on the web server of the website that allows to control search engine bots and the way they interact with the site. Robots.txt is the first file that search bots access when attempting to crawl the website.

1. Creating Robots.txt

Step 1: Open a simple text editor, such as TextEditor or Notepad. Name the file robots.txt and select the location where you want the file to be saved.

Creating Robots.txt file Step 1

Step 2: Start the file with the first line. User-agent line specifies directions to all or a specific search bot.

When “*” is added after the User-agent line, the rule will apply to all search engine bots.
If you want the rule to only apply to a specific search bot, replace “*” with the name of the bot, for example Googlebot.

Creating Robots.txt file Step 2

Step 3: Decide what folders and files you don’t want to get indexed.

If you don’t want to block any of the folder or files, add the following line: Disallow:
If you want to block the whole folder from being indexed, add this line: Disallow: /myfolder/
If you want to block a specific file, add this line: Disallow: /myfile
In the example above, /wp-admin/ folder has been blocked from indexing by all search bots.

Creating Robots.txt file Step 3

2. Installing Robots.txt with CPanel

Step 1: Access your cPanel File Manager and choose the main site directory. Then, simply click on “Upload” button and upload your robots.txt file.

Alternatively, create a new robots.txt file within the cPanel File Manager by clicking on the “New File” button.

step-1-cpanel-uploading

Step 2: Go back to the cPanel File Manager. Go back to the main site directory and you will notice that your robots.txt file has been uploaded.

Installing Robots.txt with CPanel Step 2

3. Alternative Installation with FTP

Step 1: Connect to the FTP. Connect to your website server using an FTP (File Transfer Protocol), such as FileZilla.

Installing Robots.txt with FTP Step 1

Step 2: Locate the main directory and upload. Navigate to the main website directory and simply drag and drop your robots.txt file.

Installing Robots.txt with FTP Step 2

For more information and rules about robots.txt, please read our Beginners Guide to Robots.txt.

Dmytro Spilka

Head Wizard

Get free online marketing and blogging insights<br>Unsubscribe at any time<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css"><style type="text/css"> #mc_embed_signup{background:transparent; clear:left; font:12px Helvetica,Arial,sans-serif; width:100%;} /* Add your own MailChimp form style overrides in your site stylesheet or in this style block. We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */</style><div id="mc_embed_signup"><form action="https://solvid.us13.list-manage.com/subscribe/post?u=5164852564d3ba7103743bc16&id=fa1473dcff" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate> <div id="mc_embed_signup_scroll">  <input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="Sign up for regular updates" required><div class="clear"><input type="submit" value="" name="subscribe" id="mc-embedded-subscribe" class="button"></div>  <div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_5164852564d3ba7103743bc16_fa1473dcff" tabindex="-1" value=""></div> </div></form></div>

Join 1000s of

using Solvid.

234 customers signed up in the last 30 days

SEO Accreditation Large

Solvid is a creative SEO, Content and Digital PR agency. Solvid is a registered trademark of Solvi & Heirs LTD, registered in England and Wales. Registered Address: 6 St. Davids Square, London, England, E14 3WA

VAT: GB 326425708

Reg: 09697233

020 7072 8788