Webmaster Forum Rules | Posting Guide | Contact Us | Testimonials | Contributing Geek Program | Advertise on Geek/Talk
Welcome to the GeekTalk Webmaster Discussion Forums from GeekVillage.com

Click Here To Register. It's Free!

Go Back   geek/talk: Web publishing business discussion > YOUR CREATION: Building & Maintaining A Web Site > Web Design and Webmaster Issues
User Name
Password
Register FAQ Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Search this Thread Display Modes
Old 02-13-2004, 09:02 PM   #1
jessebhunt
Registered User
 
jessebhunt's Avatar
 
Join Date: Aug 2003
Location: Insomnia
Posts: 333
Default Program to capture a whole website

Hello everyone:

I need a program that can copy an entire website quickly and easily. I have a customer that wants me to capture his website and move it to my server and I don't have FTP access to his existing server.

I know that I could just save each individual page, but that would take a lot of time.

Any ideas?

Oh yeah... not to sound cheap, but FREE would be a good thing too!
jessebhunt is offline   Reply With Quote
Old 02-13-2004, 09:47 PM   #2
DougM
Registered User
 
Join Date: Sep 2003
Location: Texas
Posts: 663
Default

Never used it but I believe that is what this does

http://www.gnu.org/software/wget/wget.html
DougM is offline   Reply With Quote
Old 02-13-2004, 10:24 PM   #3
jessebhunt
Registered User
 
jessebhunt's Avatar
 
Join Date: Aug 2003
Location: Insomnia
Posts: 333
Default

Doug:

Thanks for your quick reply.

This looks pretty comlicated. Before I jump off into trying to figure it out, does anyone else know of something easier? Also, has anyone used this WGET before... if so, do you have any pointers?

Thanks.
jessebhunt is offline   Reply With Quote
Old 02-13-2004, 10:37 PM   #4
jessebhunt
Registered User
 
jessebhunt's Avatar
 
Join Date: Aug 2003
Location: Insomnia
Posts: 333
Default

Never mind.... I did some searching on Google and found this:

http://www.newsrobot.com/sbwcc/sbwcc.html

It worked wonderfully!

It took about 3 minutes for me to install the program, spider the site, and save all the pages & images!
jessebhunt is offline   Reply With Quote
Old 02-13-2004, 10:40 PM   #5
emmzee
I am a Contributing Geek. Are You?
 
emmzee's Avatar
 
Join Date: Jul 1999
Location: Mississauga, Ontario
Posts: 1,372
Default

On a related topic ... is there any way to block programs like this from spidering my sites?
__________________
Darren Hewer
NEW: Simpsons Trivia Quiz
My sites: DOS Games - Free Web Games - Video Game Stores - Windows Games - Why Faith
emmzee is offline   Reply With Quote
Old 02-13-2004, 10:47 PM   #6
b1ing
Registered User
 
Join Date: Feb 2004
Posts: 46
Default

I am sure you can block them by using .htaccess
b1ing is offline   Reply With Quote
Old 02-14-2004, 12:41 AM   #7
jessebhunt
Registered User
 
jessebhunt's Avatar
 
Join Date: Aug 2003
Location: Insomnia
Posts: 333
Default

emmzee:

I agree with you... I can see how software like this could be used for all of the wrong reasons!

I would never try to steal another webmasters hard work, but there are many people out there who would do it without a second thought.

I too would like to know what is needed in an .htaccess file in order to stop such programs from spidering my sites.
jessebhunt is offline   Reply With Quote
Old 02-14-2004, 04:52 AM   #8
Eriky
I am a Contributing Geek. Are You?
 
Join Date: Mar 2001
Posts: 309
Default

You can try to ban the user agent, but this can be faked easilly by most programs.

I created a script which I include on the content pages of my websites. This script first checks for ip's like the google crawlers. If the ip is not trusted that ip gets logged. If an ip has viewed more than X pages (depends on the size of the site), the visitor cannot use the site any more for that day. This is specifically created to stop these programs, but the side effects are not that nice. Some large ISP's their cache servers get banned from time to time too.
Eriky is offline   Reply With Quote
Reply

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
www.Googer.net -- PPC Search Engine Website for sale coosee Website For Sale - Purchase - Barter - Trade 4 07-11-2008 01:30 AM
Logo-Monster.com (Logos download store) Very big revenue ranooo Website For Sale - Purchase - Barter - Trade 0 07-19-2006 04:06 PM
Amazon Instant Website Program Larwee Making Money with CPA Programs 1 06-07-2006 01:09 AM
ADMIN: VIP geek/chat transcripts for 6/3/00. READ ME! Steve_S Web Design and Webmaster Issues 0 06-09-2000 04:43 PM
new 800 page book $16 with 2-tier affiliate program wileyp Making Money with CPC and/or CPM Programs 4 06-06-1999 01:40 PM

Please support our advertisers. They ensure our survival.

All times are GMT -5. The time now is 05:23 PM.


GeekVillage.com is copyright © 1998-2015 Curiosity Cave - Science gifts for clever kids. All rights reserved.
Powered by vBulletin® Version 3.7.4
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.