search engine spidering - geek/talk: Signature-free discourse for serious web publishers

brian 2468 · 02-20-2002, 07:17 PM

When the search engines have your site listed, will they continually crawl over and update newer pages? I have noticed a few of my pages were picked up besides my main site homepeage. I am going to make changes so the other pages that dont get crawled on have a better chance of being spidered........ will I need to submit again......

Animal Trial

sdarken · 02-20-2002, 11:47 PM

Most search engines will crawl your site and find new pages. Make sure that your new pages are easily reached from pages which are already indexed (and preferably from your homepage). Some search engines will not crawl very deeply so may not find all pages.

A good approach is to link to all your main pages from a sitemap and link to the sitemap from your home page.

brian 2468 · 02-21-2002, 12:01 AM

sdarken...thanks I made up some other pages which are linked back to the homepage, does the link have to be a visible link on the homepage because some are not.

sdarken · 02-21-2002, 11:42 AM

If it is important that spiders find your new pages, try and make sure that they are at least linked to a page which is linked to the home page. Obviously not every page will be linked to the home page but the flatter your site structure, the better.

Scuba · 03-04-2002, 06:53 AM

I have created a file for a site map ( http://www.highvid.com/site_map.shtml )
This page is more to flat the site down for SE's then to be usefull for users.

My question (if any body is into this type of stuff), will a page like this make it more simple for SE's to spider my site, or will I get some kind of penelty for doing it.

Curently I have not actualy linked this page from the home page

02-20-2002, 07:17 PM	#1
brian 2468 Registered User Join Date: Mar 2001 Location: Alberta, Canada Posts: 211	search engine spidering When the search engines have your site listed, will they continually crawl over and update newer pages? I have noticed a few of my pages were picked up besides my main site homepeage. I am going to make changes so the other pages that dont get crawled on have a better chance of being spidered........ will I need to submit again...... Animal Trial

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Post subject: Increasing Your Search Engine Ranking	montyauto	Search Engine Marketing & Optimization	2	07-16-2008 11:00 AM
www.Googer.net -- PPC Search Engine Website for sale	coosee	Website For Sale - Purchase - Barter - Trade	4	07-11-2008 01:30 AM
Monitoring Search Engine Positions	montyauto	Search Engine Marketing & Optimization	4	05-29-2008 02:12 AM
Page Rank search engine	Larwee	Search Engine Marketing & Optimization	0	12-31-2005 09:43 PM
Question on new search engine	krink1	Marketing Your Website Online	2	10-25-2001 04:23 PM

02-20-2002, 11:47 PM	#2
sdarken Registered User Join Date: Sep 1999 Location: San Francisco Posts: 1,396	Most search engines will crawl your site and find new pages. Make sure that your new pages are easily reached from pages which are already indexed (and preferably from your homepage). Some search engines will not crawl very deeply so may not find all pages. A good approach is to link to all your main pages from a sitemap and link to the sitemap from your home page.

02-21-2002, 12:01 AM	#3
brian 2468 Registered User Join Date: Mar 2001 Location: Alberta, Canada Posts: 211	sdarken...thanks I made up some other pages which are linked back to the homepage, does the link have to be a visible link on the homepage because some are not.

02-21-2002, 11:42 AM	#4
sdarken Registered User Join Date: Sep 1999 Location: San Francisco Posts: 1,396	If it is important that spiders find your new pages, try and make sure that they are at least linked to a page which is linked to the home page. Obviously not every page will be linked to the home page but the flatter your site structure, the better.

03-04-2002, 06:53 AM	#5
Scuba Registered User Join Date: Apr 2001 Location: The Holly Land Posts: 995	I have created a file for a site map ( http://www.highvid.com/site_map.shtml ) This page is more to flat the site down for SE's then to be usefull for users. My question (if any body is into this type of stuff), will a page like this make it more simple for SE's to spider my site, or will I get some kind of penelty for doing it. Curently I have not actualy linked this page from the home page