What is robots.txt used for? - geek/talk: Web publishing business discussion

denielmark · 03-09-2016, 07:10 AM

I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

Please guys clear it.

daviddakarai · 07-29-2016, 12:23 AM

The purpose of a robots.txt file is guide to search engines that not to crawl the contents of individual pages within a website. When a website submit to search engines, it look for the robots.txt file. Expect those given remaining url's are crawl by search engines. If search engines aren't find robots file they assume to crawl the entire website.

Karen Jain · 08-02-2016, 01:37 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

varunindia88 · 08-08-2016, 08:03 AM

Robots.txt help website for crawling and indexing certain part.

logoguts · 08-16-2016, 08:00 AM

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

RH-Calvin · 08-17-2016, 07:40 AM

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

03-09-2016, 07:10 AM	#1
denielmark Registered User Join Date: Feb 2012 Location: USA Posts: 26	What is robots.txt used for? I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for? Please guys clear it.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What is Robots.txt file and where it can be in a website?	tuhineternalspring	Search Engine Marketing & Optimization	10	09-02-2015 08:55 AM
What is Robots.txt file and where it can be in a website?	Preston	Google Search Engine and AdWords	25	03-11-2015 04:43 AM
Importance of Robots.txt File	omkarsoft	Search Engine Marketing & Optimization	10	12-18-2014 03:21 AM
Robots.txt Help	yuv_luve	Search Engine Marketing & Optimization	3	12-10-2014 12:38 AM
robots.txt? Correct or need more?	modf	Marketing Your Website Online	3	09-18-2000 01:52 PM

07-29-2016, 12:23 AM	#2
daviddakarai Registered User Join Date: Dec 2014 Posts: 299	The purpose of a robots.txt file is guide to search engines that not to crawl the contents of individual pages within a website. When a website submit to search engines, it look for the robots.txt file. Expect those given remaining url's are crawl by search engines. If search engines aren't find robots file they assume to crawl the entire website.

08-02-2016, 01:37 AM	#3
Karen Jain Registered User Join Date: Jul 2015 Location: Bangalore Posts: 102	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

08-08-2016, 08:03 AM	#4
varunindia88 Registered User Join Date: Jul 2015 Location: Noida Posts: 11	Robots.txt help website for crawling and indexing certain part.

08-16-2016, 08:00 AM	#5
logoguts Registered User Join Date: Nov 2014 Posts: 9	Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

08-17-2016, 07:40 AM	#6
RH-Calvin Registered User Join Date: Jun 2013 Location: Forum Posts: 150	Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.