PDA

View Full Version : Bandwidth



Fat Kid Mike
02-28-2004, 12:13 AM
Okay, so i searched and searched and these are most of the reasons the bandwith gets used: downloading, uploading, emails. Now for the question: What else sucks up bandwidth? Does hosting a signature picture used in forums eat it up also (hosting the file)?

jalal
02-28-2004, 02:42 AM
Spiders.

FZ
02-28-2004, 04:54 AM
Does hosting a signature picture used in forums eat it up also (hosting the file)?

Of course it does - that file is on your account, and is transferred to people when they view a forum page with that "signature picture" on it, so it does use bandwidth (not to mention web space).

torrin
02-29-2004, 04:56 PM
Now for the question: What else sucks up bandwidth?

Viruses. (or actually worms)

Who here remembers code red and nimba requests for default.ida and cmd.exe?

FZ
02-29-2004, 05:35 PM
Oh yeah I remember those! I remember coming across them in my access log and web stats and wondering what the heck it was...

wildjokerdesign
02-29-2004, 06:51 PM
Yep me too. Did those reqest really eat much bandwidth or since they would return a page not found would they have pulled very little bandwidth?

torrin
02-29-2004, 11:45 PM
Well, from my access-log, I see that each request for cmd.exe sends back 1946 bytes and there is 19 requests from Feb 22, 2003 until March 1, 2003. That's 8 days.

1946 bytes x 19 = 36974 bytes
8 / 365 = 36974 / x
x = 1686938.75 bytes per year

That's pretty insignficant when you compare it to the amount of bandwidth we get.

But, during the height of the worm attack, I was getting many more that 19 requests over 8 days. So I think it would equal quite a bit. Does anybody have any statistics from that time?

torrin
02-29-2004, 11:50 PM
I just looked and found out that the 1946 bytes is from my missing.html. missing.html is sent out whenever a file that doesn't exist is requested, so I guess you can minimize what is by having a smaller missing.html file.

wildjokerdesign
03-01-2004, 05:30 AM
So I wonder if you could minimize it even more by not using a missing.html on your site and simply not serving them any page. This does make me wonder about the sites where I use a script to return a customized missing page. I wonder if that is eating a bit more bandwidth since it is processing thier request and sending them a page that trys to direct them to the correct one. I have that on a couple sites where I have renamed or moved pages.

I guess in a way for me it really does not matter since I have never used all my bandwidth on an account and had to buy extra. I don't think any of my clients have ever had to either.

jalal
03-01-2004, 06:23 AM
What you need to do is simply block any requests for cmd.exe (and for root.exe, anything for _vti_bin and so on, look in your logs).

The best way to do that is to use one of Westhosts greatest gifts to us webmasters, the ability to use mod_rewrite. You can put mod_rewrite rules in httpd.conf, for example:
RewriteRule /cmd\.exe$ - [F]
RewriteRule /root\.exe$ - [F]
RewriteRule \_vti\_ - [F]

(the [F] means 'forbidden').

In theory you can put the rewrite rules in an .htaccess file, but I haven't tried that.

For more on mod_rewrite, check out:
http://httpd.apache.org/docs/mod/mod_rewrite.html
and
http://www.engelschall.com/pw/apache/rewriteguide/

HTH

wildjokerdesign
03-01-2004, 06:22 PM
In theory you can put the rewrite rules in an .htaccess file, but I haven't tried that.


Yes I have used it in image directories to stop hotlinking and it works.

torrin
03-02-2004, 09:29 AM
Yes I have used it in image directories to stop hotlinking and it works.

Interesting.

Could you post an example of that?

wildjokerdesign
03-02-2004, 06:11 PM
I have found tons of variations to do this but this one seems to work on WestHost. I simply put this in an .htaccess file and place it in the directory I wish to protect
Rewriteengine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule \.(gif|jpg)$ http://yourdomain.com/hotlink.gif [R,NC]

The two lines with yourdomain are replaced with the obvious. You can also allow other domains to use the images by adding them using the formate in those two lines. The last line is an image that is returned to them instead of what they where trying to link to that states the image linked to was not allowed with my dot com.

EDIT there are only 5 lines to the code on my screen I am getting more returns. Each new line starts with Rewrite

gnossos
03-11-2004, 08:00 AM
to keep things in one thread, does anyone see anything wrong with this .htaccess file?:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://gnossos.com.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.gnossos.com.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://gnossos.com:80.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.gnossos.com:80.*$ [NC]
RewriteRule \.(exe)$ - [F]

doesn't do squat, e.g., I can still go directly to the file and d/l it w/o having to through the links from my pw protected page which I had wanted to do.
muchas gracias for any input.

FZ
03-11-2004, 10:36 AM
As far as I know, since going to a file directly does not generate a referrer, you will always be able to access it... The only way you can really restrict access is by password protecting the directory it is located in (but for images this is not feasible). To test if your code is really working, you should try displaying an image on your server in a file from a host/domain other than gnossos.com. I did try that, but it did display the image... However, it could easily be because my ISP had cached it when I went to view your page (to look for an image to try and "steal").

dansroka
03-13-2004, 01:10 PM
Wildjoker -- this is an AMAZING tip. Thank you for it! I gave it a try, and HEY it works! This is one of the tips that solves such an annoying problem for me. I am a photographer, and I want to post my photos online, but want some protection (a little) from people linking to them

I want to understand how this works a little better. I put this in a subdirectory on my site (eg., site.com/photos/.htaccess) that is filled with JPEGs, and it worked fine. I have it return a warning image that lives on my top level (site.com/warning.gif).

But couldn't I just put this .htaccess file on my top level, so that no images on my site can be linked to? (site.com/.htacess) When I try this, it works -- sort of. The images are blocked, but the warning.gif image does not load. My browser gives me the error: "too many HTTP redirects".

The .htaccess file I am using is:


Rewriteengine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yoursite.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www. yoursite.com/.*$ [NC]
RewriteRule \.(gif|jpg)$ http://yoursite.com/warning.gif [R,NC]

Thanks!

wildjokerdesign
03-13-2004, 01:34 PM
:)
Because you have put it in the root directory you have also blocked access to your warning.gif that is in that root directory.

Not sure if there would be a way around this or not. I'll run some tesl and let you know. I have an idea but not sure about it.

wildjokerdesign
03-13-2004, 01:55 PM
Ok give this a try. Put the image that you want to be displayed i.e. the warning.gif in your icon directory. It is located here /var/www/icons . You well see other images in this directory that are used by something else. Then use this code in your .htaccess file in that you place in /var/www/html


Rewriteengine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule \.(gif|jpg)$ http://yourdomain.com/icons/warning.gif [R,NC]

Ofcourse you need to replace yourdomain with the appropriate domain.

This worked on one of my accounts as far as I can tell.

dansroka
03-13-2004, 02:52 PM
Doh! Of course, I was blocking *all* images. Scales fall from my eyes, etc. etc. While the yourdomain.com/icons is an alias located outside of the normal html directory. Got it!

I think I'll stick with the let strict version -- I do have one folder of images I want to be visible to anyone (containing avatars, etc.).

Unless I could put the global block on my whole website, and create a .htaccess file for one public folder that *allowed* linking. Hmm... maybe I'll try that some day when it is less hot.

Thanks!

jalal
03-21-2004, 03:21 PM
You can put mod_rewrite rules in httpd.conf, for example:
RewriteRule /cmd\.exe$ - [F]
RewriteRule /root\.exe$ - [F]
RewriteRule \_vti\_ - [F]

(the [F] means 'forbidden').

In theory you can put the rewrite rules in an .htaccess file, but I haven't tried that.

Just to add a couple of things to tie up this thread...

If you want to add the above rewrite rules into an .htaccess file, then you need to leave off the beginning slash, as one of the differences between using a rewriterule in .htaccess and in httpd.conf is that in the .htaccess file the first slash is dropped. So it becomes:
RewriteRule cmd\.exe$ - [F]

Secondly, if you put the mod_rewrite rules in an .htaccess file, especially if it is in the root directory, then the file gets used for every file access. Whereas if you put it into the httpd.conf file it is read in at startup. So, there is a big efficiency gain there if its important for you.

HTH
8)

Cyle
04-27-2004, 01:56 PM
Hey wildjokerdesign, thanks for the code!

I just did a search in the forums and I just used your .htaccess file for my own site on westhost: www.angelicdreams.com

It works like a charm!

Question: Looking at this, can the .(gif|jpg) be changed to prevent hotlinking to .mpg files?

wildjokerdesign
04-28-2004, 07:46 AM
Yes I think you are right in thinking that. The RewriteRule is what it is saying to look for and change so in the image example above the htaccess file is saying to look for any file that ends in a gif or jpg exstension.

gnossos
04-28-2004, 09:04 AM
hmnn...maybe you gentlemen can give me a hand in something similar. I have some exe's on my website but I don't want people to be able to download them by just going to the file directly, rather I only want them to be accessible from the links within the protected area of my website. I put together the followingn for an .htaccess file that is in the directory of the exe's but now when ever I try to access one of the files, by direct linking or via the links on my site I get an internal server error
.htaccess file:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://gnossos.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.gnossos.com/.*$ [NC]
RewriteRule \.(exe)$ - [F]
[R,NC]

wildjokerdesign
04-28-2004, 09:38 AM
Not sure about an answer for the sever error but couldn't you simply put the same .htaccess file that you are using in the directory with the links to the exe files that protects it in the one with the .exe directory? I think that way once they have logged in to the directory with the links they would then be allowed access to the exe directory.

gnossos
04-28-2004, 09:46 AM
in theory...yes, however in practice I'm using php sessions to protect the page and not http protection. Is it possible to have the link page pass in the url the information from the session to the http proected files directory (e.g., having it be seamless to the user). Although I suppose that this method would require storing the password in the session which is not necessarily the most secure method of doing things.

jalal
04-28-2004, 09:50 AM
...on my site I get an internal server error
.htaccess file:

RewriteEngine on
RewriteCond %{HTTP_REFERE} !^$
RewriteCond %{HTTP_REFERE} !^http://gnossos.com/.*$ [NC]
RewriteCond %{HTTP_REFERE} !^http://www.gnossos.com/.*$ [NC]
RewriteRule \.(exe)$ - [F]
[R,NC]

Well, it should be HTTP_REFERER not HTTP_REFERE

And if you look in /var/log/httpd/error_log you should see details of what the error is.

wildjokerdesign
04-28-2004, 10:10 AM
Thanks Jalal I did not even notice that. That may solve all your problems gnossos and be the easiest solution.

gnossos
04-28-2004, 11:31 AM
...on my site I get an internal server error
.htaccess file:

RewriteEngine on
RewriteCond %{HTTP_REFERE} !^$
RewriteCond %{HTTP_REFERE} !^http://gnossos.com/.*$ [NC]
RewriteCond %{HTTP_REFERE} !^http://www.gnossos.com/.*$ [NC]
RewriteRule \.(exe)$ - [F]
[R,NC]

Well, it should be HTTP_REFERER not HTTP_REFERE

And if you look in /var/log/httpd/error_log you should see details of what the error is.


hmnn...yes that would cause problems. Unfortunately that was just a typing error on my part, the actual .htaccess file has that field correctly entered.

gnossos
04-28-2004, 11:33 AM
hmnn...interesting, error log is showing:
invalid command '[R,NC]'

wildjokerdesign
04-28-2004, 12:08 PM
I did some looking around on the net and came up with a few links that might help.


http://www.webmasterworld.com/forum88/3400.htm
Has this link in it http://www.phpbuilder.com/columns/index.php3?cat=6&subcat=36

http://us4.php.net/features.http-auth

http://www.weberdev.com/get_example.php3/130


Using this search in google may help get some answers also

.htaccess site:php.net

It returns tons of results that I think might be applicable to your situation. This would be ways to use your php script to possible log work with the .htpasswrd file. Although some of the threads I read seemed to indicate it was not possible others seemed to say it was.

I think you would want to group the commands together [F,R,NC] but not sure of that. Do you even need the [R,NC] for what you are trying to accomplish. Can't remember the commands off the top of my head.

jalal
04-28-2004, 01:14 PM
hmnn...interesting, error log is showing:
invalid command '[R,NC]'

I think the first line needs to be something like RewriteRule or so, just putting bare options like '[NC]' on a line is not going to work.

gnossos
04-28-2004, 01:44 PM
joker,
thanks for the links, I tried altering it to remove that last line [R,NC] and I don't get an internal server error, however, the .htaccess file doesn't do a **** thing. I can access the file by just typing the url to it into the browser and my understanding was that the .htaccess file would disallow that.

jalal
04-28-2004, 02:08 PM
I think you need to make a link from another site and click on that. Just typing the link into your browser won't provide a REFERRER line for the rewriteengine to match on.

gnossos
04-28-2004, 02:11 PM
hmnn...jalal, thanks that makes sense, not sure of the syntax, but is there a way to say if referrer blank or if referrer not my domain?

jalal
04-28-2004, 02:28 PM
You are already doing that... in fact you are checking if its a blank referrer, so just typing it in the location bar of the browser will work.

You're checking for:
referrer != ''
referrer != 'gnossos.com'
referrer != 'www.gnossos.com'

if all those conditions are true, then you don't allow the browser and return a forbidden code.

HTH

gnossos
04-28-2004, 02:38 PM
ok, well my understanding of modrewrite stuff is pretty rudimentry. Wouldn't typing the full address in the address bar directly have no referrer? If that is the case then shouldn't the check be true, e.g., if no referrer and the user should then be forwarded on to the login page. I think perhaps I don't understand how the .htaccess file interprets the conditions, e.g., if 1st condition then skip to end, or all conditions must be true or...?
thanks jalala, appreciate the help.

wildjokerdesign
04-28-2004, 02:40 PM
So it stands to reason that if you removed referre !=" then when folks typed it into thier browser address window then they would get a forbidden right? So then the only way they could get to those files are via a link from your site. I think I followed all that correctly.

gnossos
04-28-2004, 02:42 PM
well that did it, muchas gracias gentlemen. I'd still like to know how the conditions are interpreted, e.g., and all or first true skips to the restriction?

wildjokerdesign
04-28-2004, 04:26 PM
This may the best place to find that out http://httpd.apache.org/docs/mod/mod_rewrite.html I am still looking at it at the moment but I think you find it interesting.