PDA

View Full Version : Deleting spam with Procmail



Armadillo
09-20-2003, 04:13 PM
I am experimenting with using procmail to filter my spam. Fayez has been a big help, but I must still be doing something wrong.
Below is the procmailrc file I am using....


MAILDIR=/
LOGFILE=/.procmaillog
LOGABSTRACT=YES
SHELL=/bin/sh

:0:
* ^X-Spam-Flag: YES
/dev/null

:0:
${DEFAULT}

Now, the mail I receive does not have any spamassassin data in the header (it did before) and in the log file only 3 of 43 spam mails was marked as spam and sent to /dev/null. Normally most of my mail is marked as spam (spam level at 4).

Could the procmail file be interfering with spamassassin?

FZ
09-20-2003, 04:41 PM
Does mail that has the X-Spam-Flag: YES come through to your mail program (i.e. does it sometimes "miss" mail)?

I think this is related to a SpamAssassin/Procmail configuration issue I came across about a week ago. It's quite a serious problem in that it delays mail receiving by causing Sendmail to "hang" and then eventually timeout and give up. The only solution I could find to this problem was to remove SA using the Site Manager and then install (an configure) a copy of it manually. However, I think there may be a way for you to salvage your existing config. Before you do that though, send yourself a test e-mail (to your WestHost 2.0 domain) from a non-WestHost server (e.g. Hotmail or Yahoo). Time how long it takes for you to receive that mail. If it takes, say, 10 minutes or more, then you can pretty much assume this bug affects you too. If you receive it immediately without problems, then you should stop reading because this does not apply to you.

Open up /etc/procmailrc in a text editor and look for this line:


:0fw: spamassassin.lock
| /spamassassin -D


I'm just guessing that is what the path is as it is used - it may be different. But that does not matter. What you need to do is remove the -D completely (so that there is nothing after ...spamassassin on that line). Save/upload the file (you may want to backup the old version first). Then send yourself a test mail and see if you get it (i.e. how quickly you receive it). When you get it, check the headers to make sure it has passed through SpamAssassin. If this works, watch your mail closely over the next few days and see if it picks up more spam or not.

Let me know how it goes.

Armadillo
09-20-2003, 06:26 PM
Thanks for the help Fayez.

All of my mail tagged as spam had the "X-Spam-Flag: YES", non-spam does not. There should have been many more spams than what procmail deleted.

I seem to have the bug you described too. I made the changes to "/etc/procmailrc" you suggested and sent several mails via hotmail and my ISP. None of the mail arrived.

I changed the file back, uploaded it, renamed my other procmailrc (the one I added), and still could not get any mail.
I believe I may have uploaded it incorrectly (WSFTP Pro says it is a binary file).
I uploaded it again in ascii and sent another mail from hotmail. It did arrive and it had the spamassassin data in the header.
Should that file be uploaded as ascii or binary?

Anyway, I'm going to watch my mail to make sure I didnt screw up anything before I try again.
I guess this is going to be more complicated than I thought.

FZ
09-20-2003, 06:44 PM
I'm sorry you had problems after making the changes I suggested. It's really strange that you did though, as you shouldn't really have had any problems (all we did was remove a parameter/argument being passed to SA that shouldn't be there anyway).

I don't think you should try it again. I don't want you to lose mail while trying something I suggested.

About the X-Spam-Flag header, what I meant to say was do you receive mail in your e-mail program that is marked as spam instead of having it deleted on the server? This is what I meant by it "missing mail" - that it deletes some of the X-Spam-Flag: YES marked e-mails and others it does not. If it deletes all of the X-... marked ones then it is not a Procmail problem.

You say on the old server it used to pick up more spam - did you have any custom scores assigned in your user_prefs? If so, make sure to copy them across to your new user_prefs file.

Finally, have you made the changes to /etc/mail/spamassassin/local.cf that are required to get it working properly? Open it up with a text editor and change each occurrence of "true" to 1 and each "false" to 0 - this seems to be a misconfiguration issue when SA is installed through the Site Manager. Once you have done that, make sure you do the same with your user_prefs (found in /.spamassassin/). Also make sure your score of 4 is assigned correctly - to check this, have a look at the headers of incoming e-mail and confirm that they indicate SA's threshold is set to this value.

When you sent test messages to yourself, did you notice a delay in receiving mail? If you did, you can confirm that you have the same problem as I had by renaming your .procmailrc to something else (i.e. temporarily disabling it) and then sending yourself some new test e-mails and checking to see if they take the same amount of time to be received. If they take a considerably shorter time to be received, then the bug affects you too. You could also check this with SSH, but for sake of keeping this post even remotely user-friendly/readable I'm going to leave that explanation for another time (let me know if you want it).

Armadillo
09-20-2003, 07:31 PM
I dont think your information was at fault, it may have been me doing somthing wrong.

I'll recap what I did. Everything takes place with WH2.0.
Before I tryed anything, I studied my e-mail and tweaked spamassassin so that most spam I get is marked as spam. Outlook is set up to move all spam to another folder when downloaded. Most mail I get is spam and is marked as such. This works great, but I would rather not have to download it.

So, as a test, I made a procmailrc and uploaded it to "/".
After about 12 hours, I checked my mail. None of the mail I recieved was marked as spam and all were lacking spamassassin data in the header.
So, I checked the log to see how many spams had been sent to dev/null. Only 3 mails had been deleted. It should have been much more, perhaps 75% of the total. It seems that spamassassin missed a lot.

After applying the changes you suggested I could not recieve any mail. I may have uploaded it incorrectly. I managed to fix that per my other post.
I am now recieving mail and marked spam as I was before. Everything is back to normal.

Could my uploading it in binary mode have caused that problem?

Also, can anyone suggest a good book about unix/linux servers?

FZ
09-20-2003, 07:41 PM
Thanks for the detailed reply. I have a much better picture of what you are trying to do now, and what the problem is.

Let's try some log-based debugging. Add this line to your .procmailrc (at the top, after LOGABSTRACT=YES):


VERBOSE=YES

What that will do is add a lot more detailed info to your .procmaillog. Send a mail or two and then have a look at the log file in a text editor and see what's happening. In fact, if you could post some of it here that might help as well.

About the file upload mode, I have never downloaded and re-uploaded config files - I always use SSH and Pico (a text editor) - but if I had to guess I would say ASCII is the correct mode. But hey, whichever works, right? :)

Armadillo
09-21-2003, 02:00 AM
Hummmm....

I just downloaded my main procmailrc (/etc/procmailrc) and added the X-Spam-Flag filter after the spamassassin code already there. I was putting the X-Spam-Flag filter in a seperate procmailrc before.
I uploaded it.
I sent two test mails.
Both mails arrived in a timely maner and both still had the spamassassin data in the header.
:o
Maybe having two procmailrc files confused something?
This is all new to me.

I'm going to let it go for 12 hours or so, and see what it does.

Ohhhh, and I found a site that will send a test spam to see if the filter is working. It is https://messaging.its.monash.edu.au/mail/spam/test.cgi

FZ
09-21-2003, 06:37 AM
Alright, I think that pretty much (finally) confirms that the problem you were having is the same as mine.

Technically, it shouldn't get it "confused". The file you edited is global - the separate one you had before was a "local" one - both should be read properly. I wanted to have functionality with local .procmailrc's, so I took a different route to solving the problem - installing and configuring SpamAssassin myself instead of through the Site Manager.

But if this way works for you, that's great.

Let me know how it goes after the 12 hour test period ;)

Armadillo
09-21-2003, 10:18 PM
Cool.
I waited almost 24 hours and downloaded my e-mail. Usually, I get 80-100 mails a day, almost all spam.

Now, I only had 15 e-mails!
Most of them were also spam, but had low scores. They all had spamassassin data in the headers.
8)

Now, I need to figure out how to read the log. It is set to "/proc/self/fd/2" but I can not download it.

I may also put a warning on my contact page, to ask that any important messages be sent in ways that will not be mistaken for spam.

torrin
09-22-2003, 09:12 AM
Now, I need to figure out how to read the log. It is set to "/proc/self/fd/2" but I can not download it.

Yea, if it's pointing to /proc/self, the only way to access it is through the program itself.

FZ
09-22-2003, 11:08 AM
Armadillo,

I'm glad everything is working for you now. Why don't you just change the name of the log file to something else (e.g. "/procmail.log")? It shouldn't "break" anything (but you might as well back up the old version before you play with it).

Armadillo
09-26-2003, 12:43 AM
Woh!
Checked my mail today and got over 90 spams!
My filter stopped functioning.
:shock:
So, I checked my procmailrc file and it seems Westhost changed it. Here is the relevent part....


:0:
* ^X-Spam-Flag: YES
/dev/null# SPAMASSASSIN BLOCK

# The condition line ensures that only messages smaller than 250 kB
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam
# isn't bigger than a few k and working with big messages can bring
# SpamAssassin to its knees.
:0
* ! ^FROM_DAEMON
* < 256000
{
:0 fw: /var/lock/spamassassin.lock
| /spamassassin
}
# END SPAMASSASSIN BLOCK

So, that was why my filter stopped functioning. I put the filter after the spamassassin block again and it should function again (I hope).
:roll:
Yes, it did.

FZ
09-26-2003, 06:39 AM
Smart move by WestHost - I believe they had the same kind of condition in their global procmailrc pre-2.0. It would be nice though if they could somehow add their own conditions to everyone's files instead of replacing them. But then I suppose that is what local procmailrcs are for!

bnicolas
10-07-2003, 03:45 PM
here is my .procmailrc, I'm having no luck getting flagged spam deleted... any ideas? I added the stuff about limiting to 250KB and smaller cuz of Armadillo's suggestions above. Sending the real spam to /dev/null was working fine befoe 2.0 migration, and Westhost refuses to acknowledge any of my support requests!... I don't know what happened after the last couple of e-mails from C. Russell I thought they'd sorted out there problems but I guess they're still swamped?...


:0:
* ^X-Spam-Status: Yes
/dev/null

# The condition line ensures that only messages smaller than 250 kB
# (250 * 1024 = 256000 bytes) are processed by SpamAssassin. Most spam
# isn't bigger than a few k and working with big messages can bring
# SpamAssassin to its knees.
:0
* ! ^FROM_DAEMON
* < 256000
{
:0 fw: /var/lock/spamassassin.lock
| /spamassassin
}
# END SPAMASSASSIN BLOCK

FZ
10-07-2003, 04:58 PM
bnicolas,

You're probably going to kick yourself when you hear the solution: SpamAssassin is only being executed AFTER your rule that is checking for the X-Spam-Status header (which obviously does not yet exist). Copy and paste your first (header-check) block to the end of your procmailrc (or below the Spamassasin block) and it should work. Test it by moving mail instead of deleting it first!

Let me know if that helps.

avila
10-07-2003, 08:00 PM
Fayez,
I've been trying procmail scripts with SpamAssassin, and am receiving hundreds (literally, hundreds) of "Returned Mail" and "Delivery Failure" notices within 4-6 hours.

Is there a way to combine your original procmail script:

TO_^user@rhdigital.biz| TO_^anotheruser@rhdigital.biz|

which by the way, works beautifully, with something that will *also* recognize and dev/null email marked by SpamAssassin? Maybe this is asking for too much, but I think if the script could incoporate invalid addresses as well as recognized spam flags, this would solve the nondeliverable notices.

Thanks again for all your help.

FZ
10-07-2003, 09:05 PM
Hi Avila,

Yes, of course that is possible ;)

Look for this line in your existing code:


^TO_user@rhdigital.biz|^TO_anotheruser@rhdigital.b iz

Now just replace it with


^TO_user@rhdigital.biz|^TO_anotheruser@rhdigital.b iz|^X-Spam-Flag: YES

What that gate (|) character symbolizes is an OR condition - so in this case if the e-mail is addressed to user@ or it is addressed to anotheruser@ or it has been marked as spam (and you are using the code from http://forums.westhost.com/phpBB2/viewtopic.php?p=4763#4763), it will be bounced (and deleted).

Let me know if that works for you or not.