Webmaster Forum Rules | Posting Guide | Contact Us | Testimonials | Contributing Geek Program | Advertise on Geek/Talk
Welcome to the GeekTalk Webmaster Discussion Forums from GeekVillage.com

Click Here To Register. It's Free!

Go Back   geek/talk: Web publishing business discussion > YOUR CREATION: Building & Maintaining A Web Site > Web Design and Webmaster Issues
User Name
Register FAQ Calendar Search Today's Posts Mark Forums Read

Thread Tools Search this Thread Display Modes
Old 03-09-2016, 07:10 AM   #1
Registered User
denielmark's Avatar
Join Date: Feb 2012
Location: USA
Posts: 26
Default What is robots.txt used for?

I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

Please guys clear it.
denielmark is offline   Reply With Quote
Old 07-29-2016, 12:23 AM   #2
Registered User
Join Date: Dec 2014
Posts: 299

The purpose of a robots.txt file is guide to search engines that not to crawl the contents of individual pages within a website. When a website submit to search engines, it look for the robots.txt file. Expect those given remaining url's are crawl by search engines. If search engines aren't find robots file they assume to crawl the entire website.
daviddakarai is offline   Reply With Quote
Old 08-02-2016, 01:37 AM   #3
Karen Jain
Registered User
Join Date: Jul 2015
Location: Bangalore
Posts: 102

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
Karen Jain is offline   Reply With Quote
Old 08-08-2016, 08:03 AM   #4
Registered User
Join Date: Jul 2015
Location: Noida
Posts: 11

Robots.txt help website for crawling and indexing certain part.
varunindia88 is offline   Reply With Quote
Old 08-16-2016, 08:00 AM   #5
Registered User
logoguts's Avatar
Join Date: Nov 2014
Posts: 9

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
logoguts is offline   Reply With Quote
Old 08-17-2016, 07:40 AM   #6
Registered User
Join Date: Jun 2013
Location: Forum
Posts: 150

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
RH-Calvin is offline   Reply With Quote


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is Robots.txt file and where it can be in a website? tuhineternalspring Search Engine Marketing & Optimization 10 09-02-2015 08:55 AM
What is Robots.txt file and where it can be in a website? Preston Google Search Engine and AdWords 25 03-11-2015 04:43 AM
Importance of Robots.txt File omkarsoft Search Engine Marketing & Optimization 10 12-18-2014 03:21 AM
Robots.txt Help yuv_luve Search Engine Marketing & Optimization 3 12-10-2014 12:38 AM
robots.txt? Correct or need more? modf Marketing Your Website Online 3 09-18-2000 01:52 PM

Please support our advertisers. They ensure our survival.

All times are GMT -5. The time now is 08:58 PM.

GeekVillage.com is copyright © 1998-2015 Curiosity Cave - Science gifts for clever kids. All rights reserved.
Powered by vBulletin® Version 3.7.4
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.