Learn How To Create A Robots.txt File

All search engines, or at least all the important ones, now look for a robots.txt file as soon their spiders your web site. Robots.txt is a text file located in the root directory of your web site written to instruct search engine robots and spiders where they are allowed to crawl. You can create the robots.txt file manually, using any text editor or notepad. It should be an ASCII-encoded text file, not an HTML file and the filename should be lowercase. The robots.txt file is made up of two parts, the User-agent and the Disallow. The User-agent specifies which robots to allow or disallow and the Disallow specifies which directories robots can or cannot crawl.

Robots.txt Examples

# This example allows all robots to visit all files
User-agent: *
Disallow:

# Exclude all robots from part of the server
User-agent: *
Disallow: /scripts/
Disallow: /images/
Disallow: /admin/

# Example that tells all crawlers not to enter one specific file
User-agent: *
Disallow: /dir/file.html
Disallow: /dir/file2.html

# Block all images on your site from Google image search
User-agent: Googlebot-Image
Disallow: /

# To remove a specific image from Google Images
User-agent: Googlebot-Image
Disallow: /images/image.jpg

# Disallow all files with ? in url
User-agent: *
Disallow: Disallow: /*?*

 

WHAT IS PAGERANK?

  • Page Rank (PR) is a quantity defined by Google that provides a rough estimate of the overall importance of a web page. If a web page has a lot of links from large websites that also rank well, then the original web page is given a high ranking.

    A hyperlink to a page counts as a vote of support. The PageRank of a page is defined recursively and depends on the number and PageRank metric of all pages that link to it ("incoming links"). A page that is linked to by many pages with high PageRank receives a high rank itself.

WHAT ARE BACKLINKS?

  • Backlinks, which are sometimes called inbound links, are incoming links to a web page or the entire website. These inbound links are important to search engine rankings because many different search engines have algorithms that measure the number of backlinks a website or web page has, and ranks those web pages with more backlinks in a higher position.

PAGES