Alert of unusually large queue

[Kaleb Hosie]

We host a mail server which runs Postfix and there has been a few times where one of our clients computers becomes infected with malware and the password is compromised.



How this has come to my attention is because every once in a while, I will login to the mail server and see an unusually large mail queue which is all being sent to one domain.



Is it possible to monitor the queue automatically and have it send me an alert if the postfix queue reaches over a certain threshold?



Thanks everyone

[Wietse Venema]

Post by Kaleb Hosie
We host a mail server which runs Postfix and there has been a few
times where one of our clients computers becomes infected with
malware and the password is compromised.
How this has come to my attention is because every once in a while,
I will login to the mail server and see an unusually large mail
queue which is all being sent to one domain.
Is it possible to monitor the queue automatically and have it send
me an alert if the postfix queue reaches over a certain threshold?

To fight symptoms, run a cron job every 10 minutes or so:

#!/bin/sh

postqueue -p | awk '
BEGIN { limit = 10240 }
/^-- .+ Kbytes in .+ Request/ { queue_len = $5}
END { if (queue_len > limit)
print "Queue size", queue_len | "mail -s 'Queue size problem' root"
}
'

To throttle clients that send too much mail, see postfwd, policyd
and the like.

Wietse

[James Day]

Sent: 16 October 2012 13:50
Subject: Alert of unusually large queue
We host a mail server which runs Postfix and there has been a few times where one of our clients computers becomes infected with malware and the password is compromised.
 
How this has come to my attention is because every once in a while, I will login to the mail server and see an unusually large mail queue which is all being sent to one domain.
 
Is it possible to monitor the queue automatically and have it send me an alert if the postfix queue reaches over a certain threshold?
 
Thanks everyone

I use the following to do just that. I'm sure there is a better way but I fudged this together myself

Script 1:

#!/bin/bash

/usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}' > /etc/postfix/mailq_count

Script 2:

#!/bin/bash

mailq_count="/bin/cat /etc/postfix/mailq_count"

if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is" `$mailq_count`|/usr/sbin/sendmail -f ***@example.com ***@example.com ; fi


These run as cron jobs every few minutes.

Hope that helps.

Kind regards,

James Day
(IT Engineer)

[/dev/rob0]

Post by Wietse Venema

Post by Kaleb Hosie
We host a mail server which runs Postfix and there has been a
few times where one of our clients computers becomes infected
with malware and the password is compromised.
How this has come to my attention is because every once in a
while, I will login to the mail server and see an unusually
large mail queue which is all being sent to one domain.
Is it possible to monitor the queue automatically and have it
send me an alert if the postfix queue reaches over a certain
threshold?

snip

Post by Wietse Venema
To throttle clients that send too much mail, see postfwd,
policyd and the like.

+1, you need to be proactive against this kind of thing. In addition
to client rate limiting, you should use content filtering of your
submission stream. The vast majority of such ratware will have
URIBL-listed content in the spew, so SpamAssassin URIBL lookups are
likely to be very effective.

This is the growing threat against email, given the overall success
of DNSBLs against the previous generation of ratware.

[Reindl Harald]

Post by James Day
I use the following to do just that. I'm sure there is a better way but I fudged this together myself
#!/bin/bash
/usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}' > /etc/postfix/mailq_count
#!/bin/bash
mailq_count="/bin/cat /etc/postfix/mailq_count"
These run as cron jobs every few minutes.

thank you for that

i optimized this to one script without temp-file

mailq_count=`expr $mailq_count + 0`
this makes sure that we have a number if queue is empty
otherwise: /usr/local/bin/watch-queue.sh: line 4: [: -gt: unary operator expected
_________________

#!/bin/bash
mailq_count=`/usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}'`
mailq_count=`expr $mailq_count + 0`
if [ $mailq_count -gt 50 ]; then
echo "Mail count on Server is $mailq_count"
fi

[Jan P. Kessler]

Hey guys,
I'm not sure, if sending an e-mail about a "full mailqueue"-condition is
the best way to go ;-)

cheers, Jan

[Reindl Harald]

Post by Jan P. Kessler
Hey guys,
I'm not sure, if sending an e-mail about a "full mailqueue"-condition is
the best way to go ;-)

depends

if you have no bulk-mail on your server it will tak enot too long
to find a good value to adjust the "50" and as example if i have
500 queued messages i like to look if there is soemthing going
wrong

the only optimizing for me would be to send another notify
if the count goes down and prevent sending multiple notfies
after reaching the configured limit

[CSS]

Post by Jan P. Kessler
Hey guys,
I'm not sure, if sending an e-mail about a "full mailqueue"-condition is
the best way to go ;-)

Any of these checks could be handled by Nagios or anything else that can easily execute a remote command. Or tie it into your snmp daemon…

But yeah, a problem with a giant queue that piles up between cron'd intervals could certainly lead to some missed alerts. :)

Charles

Post by Jan P. Kessler
cheers, Jan

[Robert Schetterer]

Post by Jan P. Kessler
Hey guys,
I'm not sure, if sending an e-mail about a "full mailqueue"-condition is
the best way to go ;-)
cheers, Jan

perhaps use nagios, xymon etc monitor prog

so i.e the monitor client actions some mailq watch script
which results go to the monitor server, which alerts you via mail and/or
sms
--
Best Regards
MfG Robert Schetterer

[*] sys4 AG

http://sys4.de, [phone]
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

[Jan P. Kessler]

Post by Reindl Harald

Post by Jan P. Kessler
I'm not sure, if sending an e-mail about a "full mailqueue"-condition is
the best way to go ;-)

depends
if you have no bulk-mail on your server it will tak enot too long
to find a good value to adjust the "50" and as example if i have
500 queued messages i like to look if there is soemthing going
wrong

What I meant was, that there is a good chance, that you will not receive
this notification, because whatever condition causes your mails to stuck
in the queue could stop that notification, too ;-)

As mentioned by other posters you should set up a real monitoring
system, that periodically checks your queue or generates an alert (e.g.
snmp trap) on the server which does not rely on the mechanism that you
are trying to monitor (here smtp).

cheers, jpk

[Wietse Venema]

Post by Jan P. Kessler
As mentioned by other posters you should set up a real monitoring
system, that periodically checks your queue or generates an alert (e.g.
snmp trap) on the server which does not rely on the mechanism that you
are trying to monitor (here smtp).

To monitor an SMTP server, try to send a test message into it, and
raise an alarm if that test message is not delivered to mailbox or
smtp within some deadline.

Wietse

[James Day]

-----Original Message-----
Sent: 22 October 2012 15:44
Subject: Re: Alert of unusually large queue

Post by Reindl Harald

Post by Jan P. Kessler
I'm not sure, if sending an e-mail about a "full mailqueue"-condition
is the best way to go ;-)

depends
if you have no bulk-mail on your server it will tak enot too long to
find a good value to adjust the "50" and as example if i have
500 queued messages i like to look if there is soemthing going wrong

What I meant was, that there is a good chance, that you will not receive
this notification, because whatever condition causes your mails to stuck
in the queue could stop that notification, too ;-)
As mentioned by other posters you should set up a real monitoring
system, that periodically checks your queue or generates an alert (e.g.
snmp trap) on the server which does not rely on the mechanism that you
are trying to monitor (here smtp).
cheers, jpk

That's a good point, it might be worthwhile looking into something like a php script that interfaces with an SMS API. I've seen that done in the past.

Kind regards,

James Day
(IT Engineer)

[l***@airstreamcomm.net]

Post by Wietse Venema

Post by Kaleb Hosie
We host a mail server which runs Postfix and there has been a few
times where one of our clients computers becomes infected with
malware and the password is compromised.
How this has come to my attention is because every once in a while,
I will login to the mail server and see an unusually large mail
queue which is all being sent to one domain.
Is it possible to monitor the queue automatically and have it send
me an alert if the postfix queue reaches over a certain threshold?

#!/bin/sh
postqueue -p | awk '
BEGIN { limit = 10240 }
/^-- .+ Kbytes in .+ Request/ { queue_len = $5}
END { if (queue_len > limit)
print "Queue size", queue_len | "mail -s 'Queue size problem' root"
}
'
To throttle clients that send too much mail, see postfwd, policyd
and the like.
Wietse

Another method would be to use SNMP monitoring, which we have setup and
works quite nicely as our monitoring system will send email alerts, sms
messages, and phone calls. On the mail server (centos 5/6) there is a
script that is grabbing the size of the mailq:

mon_queue.sh

#!/bin/bash

# assume queue length of zero
count=0

# Place the output of postqueue into the array variable
# Output should look something like:
# -- 285 Kbytes in 20 Requests.
# (without the leading "# ")
queuelength=( $(/usr/sbin/postqueue -p | tail -n 1) )

# Make sure the array has six elements, we are interested in the
# fifth element (index 4 because arrays in BASH are 0-based)
if (( ${#queuelength[*]} == 6 ))
then
if (( queuelength[4] > 0 ))
then
count=${queuelength[4]}
else
count=0
fi
elif (( ${#queuelength[*]} == 4 )) && [[ ${queuelength[*]} = "Mail
queue is empty" ]]
then
count=0
else
# unknown output from postqueue
count="0"
fi

echo ${count}

exit

Then inside snmpd.conf add the line:

exec postqueue /usr/bin/sudo /path/to/script/mon_queue.sh


Now you should have an snmp oid of something like
.1.3.6.1.4.1.2021.8.1.101.1 which can be used to query. Then setup your
SNMP monitoring server with thresholds for queue size, and enjoy having
a historical perspective on queues.