PDA

View Full Version : Keeping Resources Below 1%



benjo4u
04-07-2009, 09:47 AM
I was surprised in January to find out that I need to keep cpu and memory usage below 1%. I had been less than 1% cpu and about 5% memory. Since then, I've been given a deadline in April to fix it (though one email allowed me up to 128mb RAM). I have low traffic and though I was using Java and a high php memory limit then, I have hacked and hacked away at my mysql and apache settings. The best I've gotten down to us about 126mb or 3% of ram and cpu seems good around 1%. (I'm on the "special" $10 hosting. If I don't fix it soon, I'll have to go to semi-dedicated, which is like 54).

My getstats test
#!/usr/bin/env bash
usage=0; for mem in `ps aux | grep / | grep -v grep | awk '{print $6}'`;
do usage=`expr $mem + $usage`; done; echo $usage;
ps aux|awk 'NR > 0 { s +=$3 }; END {print "cpu %",s}';
ps aux|awk 'NR > 0 { s +=$4 }; END {print "mem %",s}'


My settings
mysql
/etc/my.cnf
skip-locking
#skip-innodb
#new
max_connections = 15
# was 75
#new
max_user_connections = 15
#was 75
join_buffer_size = 3M
#join_buffer_size = 8M
#new
#key_buffer = 16M
key_buffer = 8M
key_buffer_size = 8M
#key_buffer_size = 64M
query_cache_limit = 128K
#query_cache_limit = 1M
query_cache_limit = 128K
#query_cache_limit = 1M
query_cache_size = 2M
#query_cache_size = 16M
query_cache_type = 1
max_allowed_packet = 1M
max_heap_table_size = 10M
#was 8
#max_heap_table_size = 12M
read_buffer_size = 2M
record_buffer = 3M
sort_buffer_size = 1M
#sort_buffer_size = 6M
#new myisam
myisam_sort_buffer_size = 16M
# was 32
table_cache = 1024
#table_cache = 8M
#thread_cache_size = 20
#table_cache = 8M
#thread_cache_size = 20
thread_cache_size = 286
thread_stack = 128K
tmp_table_size = 8M
#was 8
#tmp_table_size = 16M
innodb_log_file_size = 64M
innodb_log_buffer_size = 1M
innodb_flush_log_at_trx_commit = 2
innodb_flush_method = O_DIRECT
innodb_thread_concurrency = 4
innodb_buffer_pool_size = 32M
#was 64
#new
interactive_timeout = 25
#new
wait_timeout = 1000
#skip-networking

PHP
/etc/php.ini
extension_dir = ""
post_max_size = 200M
upload_max_filesize = 200M
;extension=ffmpeg.so
;extension=phpshield.5.2.lin
date.timezone= America/Chicago
memory_limit = 8M

Apache
/etc/httpd/conf/httpd.conf
<IfModule prefork.c>
StartServers 1
MinSpareServers 1
MaxSpareServers 1
ServerLimit 3
# was 25
MaxClients 2
#was 20
MaxRequestsPerChild 500
#was 4000
</IfModule>
<IfModule worker.c>
StartServers 1
MaxClients 2
#was 50
MinSpareThreads 1
MaxSpareThreads 3
#was 5
ThreadsPerChild 8
#was 10
MaxRequestsPerChild 500
#was 0
</IfModule>
KeepAliveTimeout 2
#was 15
#LoadModule ldap_module modules/mod_ldap.so
#LoadModule auth_ldap_module modules/mod_auth_ldap.so
#LoadModule dav_svn_module modules/mod_dav_svn.so
#LoadModule authz_svn_module modules/mod_authz_svn.so
#LoadModule jk_module modules/mod_jk.so
#LoadModule python_module modules/mod_python.so

Testing:
I used this code to estimate httpd MaxClients
#!/usr/bin/env bash
APACHE="httpd"

RSS=`ps -aylC $APACHE |grep "$APACHE" |awk '{print $8'} |sort -n |tail -n 1`
RSS=`expr $RSS / 1024`
echo $RSS
echo "Stopping $APACHE to calculate free memory"
/etc/init.d/$APACHE stop &> /dev/null
MEM=`free -m |head -n 2 |tail -n 1 |awk '{free=($4); print free}'`
#usage=0; for mem in `ps aux | grep / | grep -v grep | awk '{print $6}'`;
#do usage=`expr $mem + $usage`; done;
#MEM=`expr $usage / 1024`
#echo $usage
echo $MEM
echo "Starting $APACHE again"
/etc/init.d/$APACHE start &> /dev/null
echo "MaxClients should be around" `expr $MEM / $RSS`


I used this code to tune mysql:
tuning-primer.sh
(For both I had to install elements of procps-3.2.3 such as free, vmstat, and uptime)

Here's a snapshot of my server:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 16805 0.0 0.0 4484 1056 ? Ss Apr06 0:00 /root/usr/local/bin/svnserve -d --listen-port 3690 -r /root/usr/local
root 23012 0.0 0.0 2876 1020 ? S Apr06 0:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --datadir=/var/lib/mysql --p
root 23047 0.2 0.4 166104 18280 ? Sl Apr06 2:23 /usr/local/mysql/libexec/mysqld --basedir=/usr/local/mysql --datadir=
root 23256 0.0 0.2 72740 10568 ? Ss Apr06 0:01 /usr/sbin/httpd
root 23257 0.0 0.1 29652 5448 ? S Apr06 0:00 /usr/sbin/fcgi-
root 28777 0.4 1.1 80508 44724 ? S 08:42 0:36 /usr/sbin/httpd
root 32682 0.4 1.1 79292 43868 ? S 09:00 0:26 /usr/sbin/httpd
root 22492 0.0 0.0 2652 1440 ? Ss 10:34 0:00 -sh
root 24589 0.0 0.0 3572 944 ? R+ 10:44 0:00 ps auwx

Any advice is appreciated
(and yes, I've seen http://members.westhost.com/reducing-resources.html )

benjo4u
04-07-2009, 09:59 AM
I left out, if it matters, that I have installed ioncube, eaccelerator, fastcgi, perl, python, mysql, ruby, and ruby on rails

# -- Added by your Site Manager for ionCube 3.1.32 -- #
zend_extension=/usr/local/php/lib/php/extensions/no-debug-non-zts-20060613/ionc$

extension = /root/usr/local/phpshield/phpshield.5.2.lin
# -- Added by your Site Manager for eAccelerator 0.9.5.3 -- #
extension="eaccelerator.so"
eaccelerator.shm_size="32"
eaccelerator.cache_dir="/var/cache/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="1"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.filter=""
eaccelerator.shm_max="0"
eaccelerator.filter=""
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="0"
eaccelerator.shm_prune_period="0"
eaccelerator.shm_only="0"
eaccelerator.compress="1"
eaccelerator.compress_level="9"
# -- Added by your Site Manager for eAccelerator 0.9.5.3 -- #

I just tried disabling phpshield and ioncube but it didn't seem to make much of a difference

benjo4u
04-08-2009, 08:27 AM
I disabled mod_python and fastcgi in /etc/httpd/conf.d and that seems to have helped a bit, though now I'm not sure how to run python based programs like for example Trac

benjo4u
04-08-2009, 08:44 AM
I also tried modifying eaccelerator as follows:

eaccelerator.shm_size="16"
#was 32
eaccelerator.shm_prune_period="259200"
eaccelerator.shm_only="0"
eaccelerator.compress="1"
eaccelerator.compress_level="9"
eaccelerator.keys = "disk_only"
eaccelerator.sessions = "disk_only"
eaccelerator.content = "disk_only"

but that didn't seem to make a difference so I changed it back

benjo4u
04-10-2009, 11:16 AM
<IfModule prefork.c>
StartServers 1
#was 1
MinSpareServers 1
#was 1
MaxSpareServers 1
#was 1
ServerLimit 30
# was 25 4
MaxClients 30
#was 20 4
MaxRequestsPerChild 2000
#was 4000 1000
</IfModule>
<IfModule worker.c>
StartServers 1
MaxClients 50
#was 50 3
MinSpareThreads 1
MaxSpareThreads 5
#was 5 3
ThreadsPerChild 10
#was 5 3
ThreadsPerChild 10
#was 10 8
MaxRequestsPerChild 0
#was 0 1000
</IfModule>

Also
Timeout 40
#was 300
MaxKeepAliveRequests 200
#was 50

and I disabled mod_perl.conf

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 29905 0.0 0.0 2500 1020 ? S Apr07 0:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --
root 29949 0.2 0.6 174848 26124 ? Sl Apr07 12:11 /usr/local/mysql/libexec/mysqld --basedir=/
root 29988 0.0 0.0 5236 872 ? Ss Apr07 0:00 /root/usr/local/bin/svnserve -d --listen-po
root 7448 0.0 0.0 2436 1464 ? Ss 08:50 0:00 -sh
root 14133 0.0 0.4 67400 17204 ? Ss 12:02 0:00 /usr/sbin/httpd
root 14137 0.2 0.8 71720 33660 ? S 12:02 0:01 /usr/sbin/httpd
root 14159 0.2 1.1 71100 43416 ? S 12:02 0:02 /usr/sbin/httpd
root 17035 0.0 0.0 2652 944 ? R+ 12:16 0:00 ps aux

johnson5
04-16-2009, 08:10 PM
I too discovered this limit recently with the VPS hosting (roughly $10/month).

In regards to CPU, 1% means 1% of the total CPU capacity of your machine (so if you have a 2 CPU box you're supposed to use 2 seconds or less of CPU time per 100 seconds of elapsed real-world time).

What's interesting is a friend signed up recently and got a quad CPU account with 2x the RAM that my box has (for same price), just by chance due to which machine his account was installed on. This means he has 2x the computational power and 2x the memory limit for the same price :( Oh well...

kgriffiths
04-18-2009, 12:27 AM
Dear Johnson5,

We are continually upgrading our servers to accomidate our clients needs and to keep up with the ever changing world of technology. Some of our serves have been upgraded recently and some have yet to be upgraded. But we will eventually get them all to the same level. And then we will again "Dance that same old tune" all over again.