PDA

View Full Version : Bandwidth usage?



maida
10-07-2003, 07:02 AM
I need to find out how much bandwidth we have been using over the past few months to see if we just need to buy a few Gig here and there as we need it or if we really need to upgrade to the next package.

For the first time in 5 years, we went over our bandwidth last month...and had our site shut down 3 times as a result of it, even though we upgraded to the next package.

Just trying to figure out if

1) we really actually went over the bandwidth
2) WestHost wasn't really keeping track until after 2.0 was released
3) the bandwidth was coming from somewhere else, as has happened to other customers

I have an open ticket #100006 since September 25 which is still marked as New in the RT system.

Thanks,

Maida

wildjokerdesign
10-07-2003, 08:48 AM
maida,

One way you can check to see if someone else is using your bandwidth is to give your access_log a "read" located in /var/log/httpd.

It can be a bit confusing to look at but if you scan down the page looking for single listed IP on the left the examine the entry. One such entry from my log looks lik this.

64.141.49.2 - - [06/Oct/2003:08:31:09 -0600]
"GET /images/gibbon.gif HTTP/1.0" 200 8394
"http://images.google.ca/imgres?imgurl=wildjokerdesign.com/images/gibbon.gif
&imgrefurl=http://wildjokerdesign.com/gibbon.html
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

Yep alot to look at there. (I did clean out some of the extra junk from the refering url and added returns to make it easier to see here) First you have th IP then date then the method and what was requested. Notice on the above that it was a requested by Goggle that is the url after that. Now the above entry I do not really worry about because I get alot of them and as in the case with it I see the same IP after that viewing my page. They just used google to find the images they wanted to look at.

Now if you found alot of single IPs requesting a "GET /imagedirectory/yourimage.jpg" and then after that it list "http://thepirateurl.com/stealingimages.htm" (bot of those are fictional) then you would now someone was trying to use your images and bandwidth.

I wish that Webalizer or Wusage could show this but as yet I have not been able to understand them very well and have found this is easier in a way. I keep thinking I will right a script to read the access_log that would find these but just haven't been able to wrap my mind around the logic of weeding them out. :)

Hope that gives you someplace to start.

Shawn

wildjokerdesign
10-07-2003, 02:24 PM
I just found this nifty little program that was suggested by anouther user and thought I would pass it on. http://www.weblogexpert.com/
it is free you use it from your own computer and not on your site so you have to download the /var/log/httpd/access_log to your own computer and then view the file. I was not only able to use the current access log but also the gzip ones to get a combined report.

SJP
10-07-2003, 04:52 PM
There are other steps you can take now. You can prevent domains outside your own from accessing files by configuring your web-site's htaccess. There's probably an on-line tutorial.

SJP

SJP
10-07-2003, 05:04 PM
I thought so. http://www.manual.westhost.com/part4.html#ipfiltering

SJP

Mafdet
10-07-2003, 05:19 PM
I can't use weblogexpert with a Mac but I did try searching the log as you suggested. Immediately found someone linking to an image for an eBay description of an online book. I sent a report to eBay, but that log text is so small I didn't look for more. Got to get new glasses. :)

maida
10-07-2003, 09:09 PM
There are other steps you can take now. You can prevent domains outside your own from accessing files by configuring your web-site's htaccess. There's probably an on-line tutorial.

I thought so. http://www.manual.westhost.com/part4.html#ipfiltering

SJP

Well, how would you set it up? Because if I set our IP as the only address for HTTP access, won't that prevent everyone else from viewing the site?

Maida

wildjokerdesign
10-07-2003, 09:36 PM
Maida,

If you know of an IP that is stealing your images you could block that IP by entering it and then unchecking the boxes. As a rule everyone is has access to your site. With IP Filtering you are actually disallowing the person access.

maida
10-07-2003, 11:15 PM
Please bear with me and I work this out.

According to my site manager, my bandwidth usage is:

09/01/03 12:05:46 AM 10/01/03 12:00:04 AM 15720.596531 MB

08/14/03 01:08:01 AM 09/01/03 12:05:46 AM 0.037971 MB

Doesn't that seem odd?

Of course, part of the reason is that my site was unavailable for nearly a week during that first period due to the upgrade debacle. But still, we've got a staunch visitor base that checked daily to see if the site was back up.

As of today, bandwidth usage is 3673.37 MB

But I need statistics from previous months because I find it hard to believe we went over quota just this month after 5 years of the same type of account.

-----------------------------------

So, I tried to look at Webalizer using my site manager, but nothing appears. I checked and it has been installed. Another 2.0 bug?

-----------------------------------

So I downloaded WebLog Expert Lite. Thanks for the link!

According to the program, the access_log file (over 11 MB!) Time range: 10/6/2003 23:25:06 - 10/7/2003 21:51:34

So that is nearly a typical day, right? And it says I've got about 556 MB of bandwidth use for this period. In one day. If we say it gets up to 600 MB in a 24 hour time period (nearly 2 more hours), and times that by 31 days, then that gives me 18600 MB of bandwidth use this month alone.

It doesn't give me a breakdown of all the URLs, and I can't get any program on my regular PC to look at this file without running out of memory (tried Homesite and Notepad).

Any ideas while I wait in vain for WestHost support?

Maida

maida
10-08-2003, 12:30 AM
Ah ha! I found lots of people using our images for avatars on various bulletin boards.

One gif was accessed over 300 times in 24 hours! another one over 200 times!

Unfortunately, the IP addresses are very varied.

Anyway to block domain names?

Maida

wildjokerdesign
10-08-2003, 10:11 AM
Did some web surfing and searching and came up with these answers.

From url http://www.webjourneymen.net/enhancements/index.html?
The full thread details more but this is what it came down to insert this into an .htaccess file in your image directory.



SetEnvIf Referer "^http://www.yourdomain.com/*" referer_ok
Order Deny,Allow
Deny from all
Allow from env=referer_ok

Not sure if you have to add the www or not this was thier example that worked on thier site. I imagin there is an or comand that could be used but not familiar with this area... but I am learning.


There where lots of choices even alot of software you could buy but the above seemed the easiest solution to me.

P.S. Looking at the above example it just dawned on me that if you made the check
SetEnvIf Referer "^http://(www.)?yourdomain.com/*" referer_ok I think it would work. My logic may be off. I am sure someone will have a comment on it.

maida
10-08-2003, 01:24 PM
Thanks, I found a easy to follow page here:

http://www.thesitewizard.com/archive/bandwidththeft.shtml

Very similar to what you posted. I'll try it on a couple of directories and see how it goes.

Thanks again!

Maida

FZ
10-08-2003, 02:00 PM
Shawn,

I think you need to escape the . like this:


SetEnvIf Referer "^http://(www\.)?yourdomain.com/.*" referer_ok

And here is another good tutorial on preventing hotlinking of images: http://www.wsabstract.com/howto/htaccess10.shtml

wildjokerdesign
10-08-2003, 03:48 PM
Shawn,

I think you need to escape the . like this:


SetEnvIf Referer "^http://(www\.)?yourdomain.com/.*" referer_ok

Yep looking at the url you supplied I think you are right. It is interesting that you have to escape the . in the group but not in the rest of the match.

I knew there would be many ways to do the same thing. Makes you wonder if one is better or if like in any "code" there are just more then one way to get to what you want?

Mafdet
10-08-2003, 05:49 PM
If I ever have some spare time I'm going to try setting up one of these methods. Previously when I found hot-linked images I switched the image to one that says "Image stolen from www.mydomain.com." A lot of hassle to do it that way, but it was fun going to the site and seeing my little sign. :evil:

maida
10-10-2003, 05:24 PM
QUICK UPDATE 10/10/03:
After 30 minutes on initial hold and an hour on the phone for various tickets:
They are unable to access bandwidth usage prior to the 2.0 "upgrade" because "they are in the old system."

Which, what? They threw away? Didn't have back ups of? No longer know the passwords to?

:evil:


I need to find out how much bandwidth we have been using over the past few months to see if we just need to buy a few Gig here and there as we need it or if we really need to upgrade to the next package.

For the first time in 5 years, we went over our bandwidth last month...and had our site shut down 3 times as a result of it, even though we upgraded to the next package.

Just trying to figure out if

1) we really actually went over the bandwidth
2) WestHost wasn't really keeping track until after 2.0 was released
3) the bandwidth was coming from somewhere else, as has happened to other customers

I have an open ticket #100006 since September 25 which is still marked as New in the RT system.

Thanks,

Maida