Webmaster Forum Rules | Posting Guide | Contact Us | Testimonials | Contributing Geek Program | Advertise on Geek/Talk
Welcome to the GeekTalk Webmaster Discussion Forums from GeekVillage.com

Click Here To Register. It's Free!

Go Back   geek/talk: Signature-free discourse for serious web publishers > YOUR CREATION: Building & Maintaining A Web Site > Web Design and Webmaster Issues
User Name
Password
Register FAQ Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Display Modes
Old 03-09-2016, 07:10 AM   #1
denielmark
Registered User
 
denielmark's Avatar
 
Join Date: Feb 2012
Location: USA
Posts: 26
Default What is robots.txt used for?

I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

Please guys clear it.
denielmark is offline   Reply With Quote
Old 07-29-2016, 12:23 AM   #2
daviddakarai
Registered User
 
Join Date: Dec 2014
Posts: 297
Default

The purpose of a robots.txt file is guide to search engines that not to crawl the contents of individual pages within a website. When a website submit to search engines, it look for the robots.txt file. Expect those given remaining url's are crawl by search engines. If search engines aren't find robots file they assume to crawl the entire website.
daviddakarai is offline   Reply With Quote
Old 08-02-2016, 01:37 AM   #3
Karen Jain
Registered User
 
Join Date: Jul 2015
Location: Bangalore
Posts: 103
Default

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
Karen Jain is offline   Reply With Quote
Old 08-08-2016, 08:03 AM   #4
varunindia88
Registered User
 
Join Date: Jul 2015
Location: Noida
Posts: 11
Default

Robots.txt help website for crawling and indexing certain part.
varunindia88 is offline   Reply With Quote
Old 08-16-2016, 08:00 AM   #5
logoguts
Registered User
 
logoguts's Avatar
 
Join Date: Nov 2014
Posts: 9
Default

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
logoguts is offline   Reply With Quote
Old 08-17-2016, 07:40 AM   #6
RH-Calvin
Registered User
 
Join Date: Jun 2013
Location: Forum
Posts: 108
Default

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
RH-Calvin is offline   Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is Robots.txt file and where it can be in a website? tuhineternalspring Search Engine Marketing & Optimization 11 09-02-2015 08:55 AM
What is Robots.txt file and where it can be in a website? Preston Google Search Engine and AdWords 25 03-11-2015 04:43 AM
Importance of Robots.txt File omkarsoft Search Engine Marketing & Optimization 10 12-18-2014 03:21 AM
Robots.txt Help yuv_luve Search Engine Marketing & Optimization 3 12-10-2014 12:38 AM
robots.txt? Correct or need more? modf Marketing Your Website Online 3 09-18-2000 01:52 PM

Please support our advertisers. They ensure our survival.

All times are GMT -5. The time now is 06:39 PM.


GeekVillage.com is copyright © 1998-2015 Curiosity Cave - Science gifts for clever kids. All rights reserved.
Powered by vBulletin® Version 3.7.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.