Webmaster Forum Rules | Posting Guide | Contact Us | Testimonials | Contributing Geek Program | Advertise on Geek/Talk
Welcome to the GeekTalk Webmaster Discussion Forums from GeekVillage.com

Click Here To Register. It's Free!

Go Back   geek/talk: Web publishing business discussion > YOUR PERFORMANCE: Website Promotion, Marketing and Growth > Google Search Engine and AdWords
User Name
Password
Register FAQ Calendar Search Today's Posts Mark Forums Read

Reply
 
Thread Tools Search this Thread Display Modes
Old 11-10-2004, 10:11 AM   #1
Eriky
I am a Contributing Geek. Are You?
 
Join Date: Mar 2001
Posts: 309
Default Googlebot log file script

I created a command line instruction to view what pages google visited on my site, together with some other info. It does not include the hits for Google Adsense. If you want those hits, remove the "grep -v Mediapartners-Google | " part.

The command is:
Code:
cat access_log | grep Googlebot | grep -v Mediapartners-Google | awk '{print $4,$9,$7**' | less
It assumes you are in the directory where your site's access_log file is.

The output format is:
date+time http return code file name

Code:
sort by return code and filename:
cat access_log | grep Googlebot | grep -v Mediapartners-Google | awk '{print $9,$7**' | sort | less
This example gives you a sorted list. Because it sorts by return code too, you can easilly find the 404 hits google encountered

You can fiddle with the print statement to get more or other info.

Just wanted to share this. If anyone here has something better or similar I'm very interested.

Last edited by Eriky; 11-10-2004 at 10:18 AM.
Eriky is offline   Reply With Quote
Old 11-10-2004, 10:23 AM   #2
Nintendo
I am a Contributing Geek. Are You?
 
Nintendo's Avatar
 
Join Date: Feb 2001
Location: Planet Zeekois
Posts: 1,289
Default

I've always used cgi and SSI to generate a log.

Code:
$database = "/home/virtual/site2/fst/var/www/html/google/google.txt";

$shortdate = `date +"%D %T %Z"`; 
chop ($shortdate);

if ($ENV{'HTTP_USER_AGENT'** =~ /google/) {
    open (DATABASE,">>$database");
    print DATABASE "$ENV{'REMOTE_ADDR'** - $ENV{'HTTP_USER_AGENT'** - $ENV{'REQUEST_URI'** - $shortdate\n";
    close(DATABASE);
**
Example.
__________________
MidSummer
Nintendo is offline   Reply With Quote
Old 11-10-2004, 10:28 AM   #3
Eriky
I am a Contributing Geek. Are You?
 
Join Date: Mar 2001
Posts: 309
Default

Your method is especially useful because you can view it from your browser and it's live too.

But I would prefer my method when you have telnet or ssh access to your log files and you're logged in to your server almost all day
Eriky is offline   Reply With Quote
Old 11-10-2004, 11:12 AM   #4
UncleCaveman
Registered User
 
UncleCaveman's Avatar
 
Join Date: Jan 2004
Location: Meridian, MS
Posts: 203
Default

That is extremely useful! Thanks guys!
UncleCaveman is offline   Reply With Quote
Reply

Bookmarks

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
ACE Image Hosting Script Czar Barter/Trade, Buy, & Sell 0 11-02-2006 09:54 AM
(php) need to log clicks to a .txt file Sn0rtin Web Development - Programming / Coding 2 04-09-2002 09:03 PM
Make Every File Request Go to a Script for Apache ccole Web Design and Webmaster Issues 0 11-15-2001 02:35 PM
Free web server log file analysis program Xeloc Web Design and Webmaster Issues 9 06-26-2000 07:47 AM
Looking for a Log file Analyzer tool Lavee Web Design and Webmaster Issues 4 09-03-1999 02:14 AM

Please support our advertisers. They ensure our survival.

All times are GMT -5. The time now is 09:02 PM.


GeekVillage.com is copyright © 1998-2015 Curiosity Cave - Science gifts for clever kids. All rights reserved.
Powered by vBulletin® Version 3.7.4
Copyright ©2000 - 2022, Jelsoft Enterprises Ltd.