PDA

View Full Version : Slow e-mail receiving.



erich
04-08-2004, 08:00 AM
We are experiencing a problem with receiving e-mail on our domain. It affects multiple e-mail addresses, so is not isolated to a single account.

When a message with an attachment is sent to us, sometimes it will not arrive for several hours. This is true for messages sent from within and outside of our domain.

It's not a problem 100% of the time, but has been consistent enough that I'm being pressured to get it resolved since our accounts are used for business and we frequently e-mail spreadsheets, etc.

Any thoughts? Is there a setting I need to change?

Thanks,

Erich

denis
04-08-2004, 08:33 AM
Erich,

We've got the same problem. Westhost's been looking into it for almost 2 weeks. So far they don't seem to know what's causing this.

I think I'll send a gentle reminder for an update and let you know what they say.

Denis

denis
04-08-2004, 08:41 AM
Erich,

This gets kind of funny. Here's a reply I found on the web to my ticket (which was marked 'closed'):


We are still working on a long term solution to this problem but we have made some other configuration changes on your account that should help. Keep an eye on it for the next few days and let us know if there are messages that end up taking longer than 20 minutes to be delivered. If you have any other questions just let us know.


Oddly enough this was sent by Westhost last night and it still hasn't been delivered to my mailbox (about 13 hours it appears).

I hope this gets resolved soon.

Denis

erich
04-08-2004, 08:49 AM
I hope so too. My company certainly won't wait long for a fix. I personally really like WestHost, and use them for several other sites on the side. But, the company I work for just expects things to work. If they don't, my job depends on finding something that does.

Erich

FZ
04-08-2004, 09:07 AM
I have the same problem, but I have discovered 2 (manual) ways to speed up the delivery of mail (has worked 100% of the time for me):

1. Log in via an SSH session, and type sendmail -q -v and press enter. This will clear the mail queue and deliver mail messages that are waiting to be delivered (why they end up in a queue and not the actual mailbox is a mystery to me). This is the first thing I do when I know someone has sent me mail and I do not see it. Like you say, it appears to only affect larger e-mails.

2. Disable SpamAssassin, or do an installation of your own, of the latest version (2.63). The reason I say this is because I remember when I was using the Site Manager installation of SA, the SA process would be running for several minutes (sometimes more than half an hour) on mail (taking longer the larger the e-mail). Something is definitely up with the default WestHost config of SA.

Try these out and see if it helps. Obviously, this isn't a long term solution, but we could figure something out (such as running a cron job on Sendmail) if needed.

Let me know if this works for you too.

denis
04-08-2004, 09:49 AM
Hi Fayez,

Thanks for the advice. I removed SpamAssassin (through Manager) and tried 'sendmail -q -v' but no mail appeared.

Maybe I'll try removing the part of my procmailrc file that sends .pif's, .scr's and such to /dev/null. The code is as follows:



# VIRUS ATTACHMENT BLOCK
:0
* ^Content-Type: MULTIPART/MIXED
{
:0B
* ^Content-Disposition: (attachment|inline);
* filename=\/".*\.(bat|bif|exe|pif|com|vbs|cpl|scr)
/dev/null
}
# END VIRUS ATTACHMENT BLOCK


Any other advice? Thanks.

Denis

FZ
04-08-2004, 11:29 AM
Hi Denis,

You're welcome. If you are expecting mail to arrive, and it does not get delivered after processing the mail queue, that means that the mail simply isn't on your account yet. What that could mean is that the sender's mail server is having problems and cannot connect to yours, which has caused the delay. Now, whether this is a WestHost connection problem or not, I cannot tell, but using the two methods above should get mail through to you as soon as it is received on your account.

You do not need to remove Procmail code. I've used Procmail recipes for a very long time, some of them very complex, and it has never slowed receiving of mail for me. Also, like I said the SpamAssassin thing seems to only delay by a few minutes, so if you really need it, you can turn SpamAssassin back on.

Finally, seeing as you are using code to delete viral mail, I want to point out that this could be a reason why your mail is so "lagged". Until a few days ago, I was also running an e-mail account that was totally flooded with them (in excess of 2000 per day/100+MB bandwidth usage). While I had that, I found delays in receiving mail to be very common. However, as I said, processing the mail queue always used to deliver them to the mailbox(es). Anyway, here's another command you might want to save/remember:

mailq

What that does is print the first (or last?) 20 messages in your mail queue, i.e. From: and To:. It is quicker than doing a sendmail -q -v since you'll be able to check if your mail is actually in the queue before you go about processing it.

One last question for you: are you having this delay problem with e-mails from one particular ISP? Or is it any relatively large e-mail that seems to take forever to get to you?

erich
04-08-2004, 11:36 AM
FZ, you asked "...are you having this delay problem with e-mails from one particular ISP? Or is it any relatively large e-mail that seems to take forever to get to you?"

It mostly seems to occur when an e-mail has an attachment. Sometimes they are e-mails within our domain (just our mail server), but we have also experienced the problem with e-mails from other domains with nothing in common.

Regards,

Erich

FZ
04-08-2004, 11:48 AM
Hi Erich,

Have you tried processing the mail queue? Tried turning off SpamAssassin? Assigned home directories to the internal mail accounts you use?

What WestHost server is your account on? Perhaps you are on the same server as Denis and it is that server itself that has problems?

denis
04-08-2004, 11:52 AM
Hi Fayez,

Thanks again. The problem usually occurs with larger files (we're an engineering company...lots of AutoCAD files). However, it has even happened with Westhost support email, which I find to be kind of odd. It is not limited to one particular ISP.

I tried your 'mailq' command. It came back with '/var/spool/mqueue is empty'. In actual fact there is a file called '/var/spool/mqueue/}?' which appears to contain one SPAM message. I don't know if this is significant or not.

After the deletions I mentioned, I tried sending some messages with and without attachments to myself through our ISP and they came very quickly. I don't know if that's because of what I did or because I've been on WH's case to resolve this (and they did). I guess I'll have to wait and see if there are more delays.

Thanks again for all the help.

Denis

FZ
04-08-2004, 12:20 PM
Hi Denis,

Hmm, usually implies it could be "discrepancies" caused by congestion/downtime/connection problems (possibly on the WestHost side). The fact that WestHost e-mail is also delayed seems to point at connection problems on WestHost itself, too! Anyway, if internal e-mails are really important to you, then you should look into a better solution than e-mailing files back and forth. You could easily whip something up in PHP which lets you upload files (and delete them) with access controlled via usernames and passwords (Apache's .htaccess). It's literally a couple of lines of code. In fact, I've actually got that code in use on my site, and I'd be happy to give it to you so you can modify it (or help you modify it if you need that). If you are interested in that , let me know.

As for /var/spool/mqueue, it's a folder, not a file. It contains files though, several corresponding to each "section" of a single e-mail. It is not unusual to have the odd "orphan" mail in there which is not reported. In this case, since it is Spam, just delete the file.

If you do get a response from WestHost, please do let us know what they said, and get on their case to let you know exactly what they did ;)

erich
04-09-2004, 09:15 AM
Hi Erich,

Have you tried processing the mail queue? Tried turning off SpamAssassin? Assigned home directories to the internal mail accounts you use?

What WestHost server is your account on? Perhaps you are on the same server as Denis and it is that server itself that has problems?

I did process the mail queue and there were no messages sitting. I will do this the next time we know we have an undelivered message waiting.

I have not turned off SpamAssassin...we absolutely need it. If it needs upgrading, then WestHost should upgrade it.

How do I determine what server I am on?

Erich

FZ
04-09-2004, 11:18 AM
I have not turned off SpamAssassin...we absolutely need it. If it needs upgrading, then WestHost should upgrade it.

True, but WestHost will not upgrade it, because it is working fine in the VPS environment. Anyway, this isn't necessary because as I said, it only delays mail by a few minutes.

As for determining which server you are on, I tried to look for the post in which I detailed that, but it seems to have been deleted/archived. However, it's simple: Just go to yourdomain.com/mananger/ - this will redirect you to 69.36.161.1xx/php/login/login_screen.php?vds_ip... The server you are on is the number xx.

erich
04-09-2004, 12:06 PM
As for determining which server you are on, I tried to look for the post in which I detailed that, but it seems to have been deleted/archived. However, it's simple: Just go to yourdomain.com/mananger/ - this will redirect you to 69.36.161.1xx/php/login/login_screen.php?vds_ip... The server you are on is the number xx.

That would make our server 03.

Erich

Shoat
04-09-2004, 01:11 PM
I too have been having the same mail sending problem, except we are sending messages to many recipients instead of big files (mailing lists). FZ's "sendmail -q -v" tip worked like a charm...thanks!

Edit: Sorry about that FZ :)

FZ
04-09-2004, 01:55 PM
You mean FZ ;)

Glad it worked for you :)

erich
04-09-2004, 03:38 PM
Ok, we just ran into the problem again. A normal e-mail with a small Excel file attached was sent from one of our accounts to the other. After an hour, it had not arrived.

So, I logged in, did mailq and saw it sitting there. So, then I did sendmail -q -v and the mail was delivered.

Why does our mail just sit there without getting delivered to an account?

Erich

FZ
04-09-2004, 03:46 PM
My guess is it happens because your mail server (or Sendmail program) is busy handling other mails when that one comes through, and it is saved to be processed at a later time. I say that because until the other day I was receiving literally thousands of mails every day, and frequently had this problem (mail being queued). I've now got rid of that account, and I've received lots of e-mails with large attachments that might otherwise have been sitting in the queue for a few hours before getting to me.

Actually, the mail queue is supposed to be processed automatically, but obviously only at specified intervals. I'm pretty sure the interval as specified on each WestHost VPS is 2 hours. I will see if I can find the setting in the Sendmail configuration that lets you change how frequently the mail queue is processed (assuming it exists).

WestHost - MMellor
04-09-2004, 04:12 PM
Hello Everyone,

We are aware that some of our clients are experiencing mail delays. This is something that we are looking into. We will keep you updated on anything that we find. Thank you for your patience.

FZ
04-09-2004, 04:57 PM
Alright, I did some (light) research. I'm no Sendmail expert, but here is what I understood from the stuff that I read (and looked up in WestHost configuration):

- All messages are initially queued, and then processed.

- If there is a large load on the system (in terms of number of processes running), messages are not processed, but are left in the queue.

- Every xx minutes/hours (cannot find where WestHost has set the interval, maybe outside the VPS?), the queue is processed. HOWEVER, WestHost seems to have set the default to process only 20 messages every time the queue is processed. So, 20 messages will be delivered, and then it will wait another xx minutes/hours. As I said above, I think this is set to 2 hours by WestHost.

- Messages in the queue are sorted (and processed) by priority. This can be changed.

- Eventually all mail is delivered.

So, my recommendation to you would be:

1. Either send your mail with high/urgent priority, or change the way the queue is sorted. Some of the other ways it can be sorted are time, modification, and hostname.

or

2. Change the number of messages processed every time the queue is run (set it a little higher).

or

3. Set a cron job to clear the queue (i.e. execute sendmail -q) every x minutes, for example every 20 minutes (depending on how much mail you get).

For options where I've said you can change the configuration, let me know and I'll help you out. Just keep in mind that WestHost chose defaults where they could and set values in other places, and they must have done it for a reason. Changing your config might help with your problem, but if anything else goes wrong, it's at your own risk! HOWEVER, I do not see any harm in using option 3 to process your mail queue faster (no config changes required).

Here's a nice guide I found on this topic:

http://nscp.upenn.edu/aix4.3html/aixbman/commadmn/ml_queue.htm

Note that some of the things on that page do not apply to the WestHost VPS environment, such as starting and stopping the Sendmail daemon (since WestHost does not run it as a daemon), the /etc/rc.tcpip stuff, etc.

denis
04-14-2004, 07:43 AM
Erich,

Are you losing any email that you know of...i.e. it never arrives? I think I may have that problem. An email that WH tech support sent last Wed. has not arrived, but the mail logs appear to indicate that it was delivered 6 minutes after it was sent.

I hope this gets sorted out soon.

Are you on a Starter plan? We're on Server wdclnx20.

Regards,

Denis

erich
04-14-2004, 11:01 AM
Erich,

Are you losing any email that you know of...i.e. it never arrives? I think I may have that problem. An email that WH tech support sent last Wed. has not arrived, but the mail logs appear to indicate that it was delivered 6 minutes after it was sent.

I hope this gets sorted out soon.

Are you on a Starter plan? We're on Server wdclnx20.

Regards,

Denis

Hi Denis,

Actually, we are on the Advanced 4 package with additional bandwidth purchased. Probably a few gigs shy of dedicated hosting being cost effective. So, you can imagine the frustration we're having.

But, to answer your question about missing mails, we haven't noticed that yet. Of course, we wouldn't know about what we don't receive unless someone calls us. It appears all of our messages get delivered.

Regards,

Erich

erich
04-20-2004, 12:50 PM
1. Log in via an SSH session, and type sendmail -q -v and press enter. This will clear the mail queue and deliver mail messages that are waiting to be delivered (why they end up in a queue and not the actual mailbox is a mystery to me). This is the first thing I do when I know someone has sent me mail and I do not see it. Like you say, it appears to only affect larger e-mails.

This is interesting...

As we have had mail stuck several times, I have confirmed with mailq and then cleared the queue with sendmail -q -v. But, today I received an error message:

Skipping queue run -- load average too high

What is that about?

WestHost...what can you do?

Erich

erich
04-20-2004, 12:56 PM
Erich,

Are you losing any email that you know of...i.e. it never arrives? I think I may have that problem. An email that WH tech support sent last Wed. has not arrived, but the mail logs appear to indicate that it was delivered 6 minutes after it was sent.

I hope this gets sorted out soon.

Are you on a Starter plan? We're on Server wdclnx20.

Regards,

Denis

Denis,

I think it may have occured last week. I sent an attached file from home (an Earthlink account) to work. It was a small Excel file, nothing massive at all. It never arrived and did/does not appear in the queue. I double checked at home to make sure the message was sent...and that it was sent to the right address...it was.

So, now we may be losing some messages.

Erich

FZ
04-20-2004, 12:59 PM
I think that ties in with what I explained in my previous post. Either you will have to wait and try again later, or make some modifications to your Sendmail settings (I can let you know what to change, but I don't know if that will fix the problem).

Log in via SSH and type psx and press enter - see how many processes are running?

erich
04-20-2004, 01:01 PM
I think that ties in with what I explained in my previous post. Either you will have to wait and try again later, or make some modifications to your Sendmail settings (I can let you know what to change, but I don't know if that will fix the problem).

Log in via SSH and type psx and press enter - see how many processes are running?

13 processes running. Low? High?

Erich

FZ
04-20-2004, 01:33 PM
Sorry, I should have been more specific... What processes are they? Sendmail processes? Or just Apache and mySQL stuff?

j103c
04-28-2004, 01:36 PM
Any update on this? Sent an e-mail from pine to users on the same domain and took about 3 hrs.. just curious if any updates. Doesn't happy to all e-mail thankfully, seems pretty sporadic.

FZ
04-28-2004, 04:23 PM
Have you set up all your e-mail accounts with home directories? Disabled SpamAssassin (for debugging purposes, to see if it speeds up delivery)?

j103c
04-28-2004, 04:55 PM
Hey FZ,

Yes they have home directories.

No I haven't tried disabling SpamAssassin. It's pretty sporadic when they get 'stuck' in the queue. 2 e-mails were sent at the same time and one made it through while this particular one didn't, and they were just small test e-mails... so not sure that it would get hung up in SpamAssassin?

I am kind of referencing status from WH based on the post from MMellor several above this one.

FZ
04-29-2004, 07:51 PM
I'm stumped :? Did you check your mailq too?

j103c
04-29-2004, 07:59 PM
I'm not sure what's going on at this point, because some e-mail problems for one user turned out how they were setting their personal SpamAssassin rules. They were blocking some e-mails they expected by accident. ;)

However there was that one message that did take a couple hours, but at this point I'm thinking it's more user-side than server-side. :D

Unless I see it happen again, I'm not going to worry about it anymore..

wiltech
05-05-2004, 10:22 PM
I have a client that is paranoid about slow/ potentially missing email. Recent server loads or whatever is plagueing westhost have nearly forced me to move his account off Westhost. I hope they find whatever the problem is or get connections/hardware upgraded if that is the problem.

Jonathon at techsupport helped explain some of what might be happening - agreeing pretty much with what ZL and others have said. We set up a cron job that runs every fifteen minutes and executes sendmail -q to force the queue to be emptied. So far that has fixed the problem and satisfied my client.

I noticed on the sendmail.org site that there have been several releases of sendmail past what WH runs (8.11.6) that address CRITICAL security issues. I didn't read what the issues are but they may be related to DNS attacks or something that could be causing high loads. Perhaps an upgrade is in order.

Kalin