blocking compromised sasl users ?

[Voytek]

it looks like I have a couple of compromised user accounts on one of the
domains on this server, I've changed the user password then even deleted
the user (through postfixadmin) but that didn't help..? I can see in the
log this:

Oct 8 00:27:57 emu postfix/smtpd[7655]: 87E6B5E791:
client=unknown[104.200.78.121], sasl_method=LOGIN,
sasl_username=***@dom.org.au
Oct 8 00:27:58 emu postfix/smtpd[7678]: 645845FCCE:
client=unknown[104.200.78.121], sasl_method=LOGIN,
sasl_username=***@dom.org.au
Oct 8 00:28:02 emu postfix/smtpd[7678]: 3F6925FB48:
client=unknown[104.200.78.121], sasl_method=LOGIN,
sasl_username=***@dom.org.au
Oct 8 00:28:02 emu postfix/smtpd[7655]: 56C165FD24:
client=unknown[104.200.78.121], sasl_method=LOGIN,
sasl_username=***@dom.org.au

I've also tried adding to main.cf this "check_sasl_access
hash:/etc/postfix/sasl_access"

smtpd_recipient_restrictions =.
reject_unknown_sender_domain,
reject_unknown_recipient_domain,.
reject_non_fqdn_sender,.
reject_non_fqdn_recipient,.
reject_unlisted_recipient,.
check_policy_service inet:127.0.0.1:7777,.
permit_mynetworks,
check_sasl_access hash:/etc/postfix/sasl_access
permit_sasl_authenticated,
reject_unauth_destination,
check_recipient_access hash:/etc/postfix/recipient_no_checks,
check_recipient_access pcre:/etc/postfix/recipient_checks.pcre,
check_helo_access hash:/etc/postfix/helo_checks,
check_sender_access hash:/etc/postfix/sender_checks,
check_client_access hash:/etc/postfix/client_checks,
check_client_access pcre:/etc/postfix/client_checks.pcre,
reject_rbl_client zen.spamhaus.org,
reject_rhsbl_client dbl.spamhaus.org,
reject_rhsbl_sender dbl.spamhaus.org,
reject_rbl_client psbl.surriel.com,
reject_rhsbl_sender dsn.rfc-ignorant.org,
check_policy_service inet:127.0.0.1:10031

# cat /etc/postfix/sasl_access
cas HOLD
bank HOLD
***@dom.org.au HOLD
***@dom.org.au HOLD

but I see new log entries all the time,

what do I need to do ?

thanks

[Voytek]

Post by Voytek
it looks like I have a couple of compromised user accounts on one of
the domains on this server, I've changed the user password then even
deleted the user (through postfixadmin) but that didn't help..? I can
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
I've also tried adding to main.cf this "check_sasl_access
hash:/etc/postfix/sasl_access"
# cat /etc/postfix/sasl_access

that exact authentication name.

Viktor,

sorry, attempted to anonymize email addresses, BUT, overlooked the last
two, only annoymized domains in the last two

in the /etc/postfix/sasl_access names are correct,

I've used both with and without domain

V

[n***@devels.es]

Is 104.200.78.121 listed in your $permit_mynetworks parameter, or a CIDR
that contains it?
Did you postmap /etc/postfix/sasl_access?
Did you try ***@dom.org.au as entry?
Did you try cas@ as entry?

Regards,

Nicolás

Post by Voytek

Post by Voytek
it looks like I have a couple of compromised user accounts on one of
the domains on this server, I've changed the user password then even
deleted the user (through postfixadmin) but that didn't help..? I can
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
I've also tried adding to main.cf this "check_sasl_access
hash:/etc/postfix/sasl_access"
# cat /etc/postfix/sasl_access

that exact authentication name.

Viktor,
sorry, attempted to anonymize email addresses, BUT, overlooked the last
two, only annoymized domains in the last two
in the /etc/postfix/sasl_access names are correct,
I've used both with and without domain
V

[Voytek]

On Thu, October 8, 2015 12:54 am, ***@devels.es wrote:

Nicolás, thanks

Post by n***@devels.es
Is 104.200.78.121 listed in your $permit_mynetworks parameter, or a CIDR
that contains it?

no
# grep 104.200 main.cf
#

Post by n***@devels.es
Did you postmap /etc/postfix/sasl_access?

yes
how to do that ?

Post by n***@devels.es

Post by Voytek

Post by Voytek
it looks like I have a couple of compromised user accounts on one
of the domains on this server, I've changed the user password then
even deleted the user (through postfixadmin) but that didn't help..?
I can
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
I've also tried adding to main.cf this "check_sasl_access
hash:/etc/postfix/sasl_access"
# cat /etc/postfix/sasl_access

that exact authentication name.

Viktor,
sorry, attempted to anonymize email addresses, BUT, overlooked the last
two, only annoymized domains in the last two
in the /etc/postfix/sasl_access names are correct,
I've used both with and without domain
V

[l***@go2france.com]

Post by Voytek
it looks like I have a couple of compromised user accounts on one of
the
domains on this server, I've changed the user password then even
deleted
the user (through postfixadmin) but that didn't help..? I can see in
the
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
I've also tried adding to main.cf this "check_sasl_access
hash:/etc/postfix/sasl_access"
smtpd_recipient_restrictions =.
reject_unknown_sender_domain,
reject_unknown_recipient_domain,.
reject_non_fqdn_sender,.
reject_non_fqdn_recipient,.
reject_unlisted_recipient,.
check_policy_service inet:127.0.0.1:7777,.
permit_mynetworks,
check_sasl_access hash:/etc/postfix/sasl_access
permit_sasl_authenticated,
reject_unauth_destination,
check_recipient_access hash:/etc/postfix/recipient_no_checks,
check_recipient_access pcre:/etc/postfix/recipient_checks.pcre,
check_helo_access hash:/etc/postfix/helo_checks,
check_sender_access hash:/etc/postfix/sender_checks,
check_client_access hash:/etc/postfix/client_checks,
check_client_access pcre:/etc/postfix/client_checks.pcre,
reject_rbl_client zen.spamhaus.org,
reject_rhsbl_client dbl.spamhaus.org,
reject_rhsbl_sender dbl.spamhaus.org,
reject_rbl_client psbl.surriel.com,
reject_rhsbl_sender dsn.rfc-ignorant.org,
check_policy_service inet:127.0.0.1:10031
# cat /etc/postfix/sasl_access
cas HOLD
bank HOLD
but I see new log entries all the time,
what do I need to do ?

I solved our serious compromised user problems with postfwd
rate-limiting, both per user and per IP. when thresholds were met,
HOLD all mail and send a alert. I refined the per-user rate-limiting
with multiple thresholds, by harvesting which users were legitimate
senders of high volumes.

I also wrote little script that HOLDs any user that sends from more
than <threshold> IPs in <x> minutes. First, I harvested 90 days of
logs to see which user legitimately sent from several IPs and put
those users in a whitelist for the too-many-IPs control, but not from
the postfwd rate-limiting controls.

Len

[Voytek]

I think I've stopped compromised user sending by stopping and restarting
Postfix, prior to that, I've reloaded Postfix after adding/postmaping
sasl_access list - that didn't help, only stopping Postfix stopped it

I'm worried that 'there is more' ?

I've found one more compromised user by searching offending IP in prior
maillog

[Viktor Dukhovni]

Post by Voytek
I think I've stopped compromised user sending by stopping and restarting
Postfix, prior to that, I've reloaded Postfix after adding/postmaping
sasl_access list - that didn't help, only stopping Postfix stopped it

With Berkeley-DB tables, updated tables are only picked up by smtpd
when a client disconnects and a new client connects.

So if a client was hanging on to a single connection and sending
lots of messages back to back without disconnecting, it might be
able to continue despite table changes.

If your smtpd is not chrooted, you might have better luck with CDB,
than Berkeley DB, though I am not sure whether tinycdb (like DJB's
original implementation) detects table file changes and automatically
reopens the table on the fly.

Otherwise, you may be better off with SQL or LDAP tables, which can
change in real time.

Post by Voytek
I'm worried that 'there is more' ?

There's nothing more.

[Viktor Dukhovni]

Post by Voytek
it looks like I have a couple of compromised user accounts on one of the
domains on this server, I've changed the user password then even deleted
the user (through postfixadmin) but that didn't help..? I can see in the
client=unknown[104.200.78.121], sasl_method=LOGIN,
client=unknown[104.200.78.121], sasl_method=LOGIN,
I've also tried adding to main.cf this "check_sasl_access
hash:/etc/postfix/sasl_access"
# cat /etc/postfix/sasl_access
cas HOLD
bank HOLD

Notice that the logs say "***@dom.org.com", but you're not blocking
that exact authentication name.

[Voytek]

Post by Viktor Dukhovni

Post by Voytek
I think I've stopped compromised user sending by stopping and
restarting Postfix, prior to that, I've reloaded Postfix after
adding/postmaping sasl_access list - that didn't help, only stopping
Postfix stopped it

With Berkeley-DB tables, updated tables are only picked up by smtpd
when a client disconnects and a new client connects.
So if a client was hanging on to a single connection and sending
lots of messages back to back without disconnecting, it might be able to
continue despite table changes.

so, seeing as I didn't see anymore connection from that IP after Postfix
restart, that would tend to confirm above, I think ?

Post by Viktor Dukhovni
If your smtpd is not chrooted, you might have better luck with CDB,
than Berkeley DB, though I am not sure whether tinycdb (like DJB's original
implementation) detects table file changes and automatically reopens the
table on the fly.
Otherwise, you may be better off with SQL or LDAP tables, which can
change in real time.

my users/domain are in MySQL - but, again, if I understand it correctly.
on a single connection sceanrio, that wouldn't help ?

Post by Viktor Dukhovni

Post by Voytek
I'm worried that 'there is more' ?

There's nothing more.

Viktor, thanks!

I should've stopped/restarted immediately...

thanks again for all replies!

[Viktor Dukhovni]

Post by Voytek

Post by Viktor Dukhovni
With Berkeley-DB tables, updated tables are only picked up by smtpd
when a client disconnects and a new client connects.
So if a client was hanging on to a single connection and sending
lots of messages back to back without disconnecting, it might be able to
continue despite table changes.

so, seeing as I didn't see anymore connection from that IP after Postfix
restart, that would tend to confirm above, I think ?

No. Confirmation would be looking at the logs of the ongoing mails
*before* the restart and seeing whether all the mail came in over
a single connection (same pid, no per-connection "connect from" or
"disconnect from" log entries for that pid between "client="
per-queue-file log entries).

Post by Voytek

Post by Viktor Dukhovni
Otherwise, you may be better off with SQL or LDAP tables, which can
change in real time.

my users/domain are in MySQL - but, again, if I understand it correctly.
on a single connection sceanrio, that wouldn't help ?

What would help is putting the "check_sasl_access" table in SQL.

Post by Voytek
I should've stopped/restarted immediately...

No, instead put your access table in SQL (possibly CDB would work
too, but I'm not sure), that way you don't need reload or restart.

[Voytek]

Post by Viktor Dukhovni
There's nothing more.

Viktor,
thanks again for your help and explanation, just found this, I think I can
call it a day now:

Oct 8 02:08:41 emu postfix/smtpd[29357]: connect from
unknown[104.200.78.121]
Oct 8 02:08:44 emu postfix/smtpd[29357]: warning:
unknown[104.200.78.121]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
Oct 8 02:08:49 emu postfix/smtpd[32593]: connect from
unknown[104.200.78.121]
Oct 8 02:08:56 emu postfix/smtpd[32593]: warning:
unknown[104.200.78.121]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
Oct 8 02:09:09 emu postfix/smtpd[29357]: warning:
unknown[104.200.78.121]: SASL LOGIN authentication failed: UGFzc3dvcmQ6
Oct 8 02:09:19 emu postfix/smtpd[32593]: warning:
unknown[104.200.78.121]: SASL LOGIN authentication failed: Connection lost
to authentication server
Oct 8 02:14:09 emu postfix/smtpd[29357]: timeout after AUTH from
unknown[104.200.78.121]
Oct 8 02:14:09 emu postfix/smtpd[29357]: disconnect from
unknown[104.200.78.121]
Oct 8 02:14:19 emu postfix/smtpd[32593]: timeout after AUTH from
unknown[104.200.78.121]
Oct 8 02:14:19 emu postfix/smtpd[32593]: disconnect from
unknown[104.200.78.121]
Oct 8 02:17:30 emu postfix/anvil[22737]: statistics: max connection count
2 for (smtp:104.200.78.121) at Oct 8 02:08:49

[Voytek]

Post by Viktor Dukhovni
No. Confirmation would be looking at the logs of the ongoing mails
*before* the restart and seeing whether all the mail came in over
a single connection (same pid, no per-connection "connect from" or
"disconnect from" log entries for that pid between "client="
per-queue-file log entries).

# grep 104.200.78.121 /var/log/maillog | grep 7678 |wc
4094 37433 624635
# grep 104.200.78.121 /var/log/maillog | grep 4450 |wc
20 234 3389
# grep 104.200.78.121 /var/log/maillog | grep 7655 |wc
3989 36096 593584
# grep 104.200.78.121 /var/log/maillog | wc
8178 74815 1235011


# grep 104.200.78.121 /var/log/maillog | grep 7655 | grep 'connect'
Oct 7 21:54:22 emu postfix/smtpd[7655]: connect from unknown[104.200.78.121]
# grep 104.200.78.121 /var/log/maillog | grep 7678 | grep connect
Oct 7 21:44:09 emu postfix/smtpd[7678]: connect from unknown[104.200.78.121]

Post by Viktor Dukhovni

Post by Voytek

Post by Viktor Dukhovni
Otherwise, you may be better off with SQL or LDAP tables, which can
change in real time.

my users/domain are in MySQL - but, again, if I understand it
correctly. on a single connection sceanrio, that wouldn't help ?

What would help is putting the "check_sasl_access" table in SQL.

Post by Voytek
I should've stopped/restarted immediately...

No, instead put your access table in SQL (possibly CDB would work
too, but I'm not sure), that way you don't need reload or restart.

I'll look at that in the morning, thanks again

[Quanah Gibson-Mount]

--On Wednesday, October 07, 2015 5:06 PM +0000 Viktor Dukhovni

Post by Viktor Dukhovni
What would help is putting the "check_sasl_access" table in SQL.

Post by Voytek
I should've stopped/restarted immediately...

No, instead put your access table in SQL (possibly CDB would work
too, but I'm not sure), that way you don't need reload or restart.

So if they are in the SASL table, does it force close their connection?
Just want to be sure that if I implement this via an LDAP table, that the
spammer doesn't go on spamming once the user password is changed and the
account is unlocked.

--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Viktor Dukhovni]

Post by Viktor Dukhovni
What would help is putting the "check_sasl_access" table in SQL.

Post by Voytek
I should've stopped/restarted immediately...

No, instead put your access table in SQL (possibly CDB would work
too, but I'm not sure), that way you don't need reload or restart.

So if they are in the SASL table, does it force close their connection? Just
want to be sure that if I implement this via an LDAP table, that the spammer
doesn't go on spamming once the user password is changed and the account is
unlocked.

The "check_sasl_access" restriction consults an access(5) table
and can return any supported access(5) result.

In this case, it might make sense to go with:

***@example.com 521 5.7.1 Account disabled

which drops the connection as documented.

[Quanah Gibson-Mount]

--On Wednesday, October 07, 2015 11:07 PM +0000 Viktor Dukhovni

Post by Viktor Dukhovni

Post by Quanah Gibson-Mount

Post by Viktor Dukhovni
What would help is putting the "check_sasl_access" table in SQL.

Post by Voytek
I should've stopped/restarted immediately...

No, instead put your access table in SQL (possibly CDB would work
too, but I'm not sure), that way you don't need reload or restart.

So if they are in the SASL table, does it force close their connection?
Just want to be sure that if I implement this via an LDAP table, that
the spammer doesn't go on spamming once the user password is changed and
the account is unlocked.

The "check_sasl_access" restriction consults an access(5) table
and can return any supported access(5) result.
which drops the connection as documented.

Awesome, ty!

--Quanah


--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Viktor Dukhovni]

Post by Viktor Dukhovni

Post by Viktor Dukhovni
What would help is putting the "check_sasl_access" table in SQL.

Post by Voytek
I should've stopped/restarted immediately...

No, instead put your access table in SQL (possibly CDB would work
too, but I'm not sure), that way you don't need reload or restart.

So if they are in the SASL table, does it force close their connection? Just
want to be sure that if I implement this via an LDAP table, that the spammer
doesn't go on spamming once the user password is changed and the account is
unlocked.

The "check_sasl_access" restriction consults an access(5) table
and can return any supported access(5) result.
which drops the connection as documented.

Mind you, if they log in the mean time, and don't send any mail,
the connection is timed out. If they do try to send mail, the
transaction is refused. When the error limit is exceeded the
connection is closed.

So the exposure is not so bad even without dropping the connection,
but dropping may better, if the MUA of the unfortunate user handles
this in an acceptable way (not much worse than what you get by
refusing messages and not closing).

[Quanah Gibson-Mount]

--On Wednesday, October 07, 2015 11:13 PM +0000 Viktor Dukhovni

Post by Viktor Dukhovni
Mind you, if they log in the mean time, and don't send any mail,
the connection is timed out. If they do try to send mail, the
transaction is refused. When the error limit is exceeded the
connection is closed.
So the exposure is not so bad even without dropping the connection,
but dropping may better, if the MUA of the unfortunate user handles
this in an acceptable way (not much worse than what you get by
refusing messages and not closing).

Ok. What I want to avoid is this:

User account is compromised
Spammer creates a persistent connection to send out spam
Admin adds compromised user to the SASL map
Admin contacts user, has them change their password
Admin removes user from the SASL map
Compromised connection is still open to postfix, and spam continues until
postfix is restarted, and the spammer can no longer auth because the
password was changed.

I see this fairly frequently with our customers, where they don't
understand why simply having the user change their password doesn't stop
the spammer from being able to send out email, because postfix "logs" an
auth for every one of the emails sent out over the persistent connection,
even thought they actually only have auth'd when initially opening the
connection.

--Quanah



--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration

[Viktor Dukhovni]

Post by Quanah Gibson-Mount
User account is compromised
Spammer creates a persistent connection to send out spam
Admin adds compromised user to the SASL map
Admin contacts user, has them change their password
Admin removes user from the SASL map
Compromised connection is still open to postfix, and spam continues until
postfix is restarted, and the spammer can no longer auth because the
password was changed.
I see this fairly frequently with our customers, where they don't understand
why simply having the user change their password doesn't stop the spammer
from being able to send out email, because postfix "logs" an auth for every
one of the emails sent out over the persistent connection, even thought they
actually only have auth'd when initially opening the connection.

Yes, just changing the password is not enough in the face of
persistent connections. In SMTP the SASL authentication status is
a *connection* property not a per-message property.

To lock out existing connections, an access rule has to be added
to deny access until after the password is changed. The access
rule can be via "check_sasl_access" or via a "policy" lookup
performed with every transaction.

Just changing the password does nothing to invalidate existing
connections.

[Rob Sterenborg (Lists)]

Post by Viktor Dukhovni
If your smtpd is not chrooted, you might have better luck with CDB,
than Berkeley DB, though I am not sure whether tinycdb (like DJB's
original implementation) detects table file changes and automatically
reopens the table on the fly.

It seems it does:

postfix/smtpd[20438]: table cdb:/path/to/recipients(0,lock|fold_fix|
utf8_request) has changed -- restarting


--
Rob

[Viktor Dukhovni]

Post by Rob Sterenborg (Lists)

Post by Viktor Dukhovni
If your smtpd is not chrooted, you might have better luck with CDB,
than Berkeley DB, though I am not sure whether tinycdb (like DJB's
original implementation) detects table file changes and automatically
reopens the table on the fly.

postfix/smtpd[20438]: table cdb:/path/to/recipients(0,lock|fold_fix|
utf8_request) has changed -- restarting

That's not sufficient. It would have to see the new data without
restarting. With DJB's CDB, I think the underlying mmaped file is
reopened transparently if it changes. With tinycdb, it might not
be. At the very least, that can't happen if Postfix is chrooted,
or the table can only be opened by root.

--
Rob

[Wietse Venema]

Post by Rob Sterenborg (Lists)

Post by Viktor Dukhovni
If your smtpd is not chrooted, you might have better luck with CDB,
than Berkeley DB, though I am not sure whether tinycdb (like DJB's
original implementation) detects table file changes and automatically
reopens the table on the fly.

postfix/smtpd[20438]: table cdb:/path/to/recipients(0,lock|fold_fix|
utf8_request) has changed -- restarting

Yes, but the check happens at the beginning of an SMTP session, not
in the middle. A Postfix process does not reopen files mid-flight
because it may not have sufficient privileges to do so (iin addition,
cdb files are renamed, not overwritten, so the lookup result does
not change during the lifetime of the SMTP daemon process; if you
need instant visibility, use LMDB, LDAP or *SQL).

Wietse

[Rob Sterenborg (Lists)]

Post by Viktor Dukhovni

Post by Rob Sterenborg (Lists)

Post by Viktor Dukhovni
If your smtpd is not chrooted, you might have better luck with CDB,
than Berkeley DB, though I am not sure whether tinycdb (like DJB's
original implementation) detects table file changes and automatically
reopens the table on the fly.

postfix/smtpd[20438]: table cdb:/path/to/recipients(0,lock|fold_fix|
utf8_request) has changed -- restarting

That's not sufficient. It would have to see the new data without
restarting. With DJB's CDB, I think the underlying mmaped file is
reopened transparently if it changes. With tinycdb, it might not
be. At the very least, that can't happen if Postfix is chrooted,
or the table can only be opened by root.

As the filename implies it contains recipients. When the file is
updated, I see the above line and after that Postfix knows about any
change in the file.

There's one thing though that I overlooked when posting and what you're
referring to: we're not running Postfix chrooted so we wouldn't be
running into that.

Post by Viktor Dukhovni
Yes, but the check happens at the beginning of an SMTP session, not
in the middle. A Postfix process does not reopen files mid-flight
because it may not have sufficient privileges to do so (iin addition,
cdb files are renamed, not overwritten, so the lookup result does
not change during the lifetime of the SMTP daemon process; if you
need instant visibility, use LMDB, LDAP or *SQL).

I see. Except for recipients we already have most lookup tables in SQL.
So far having recipients in a regularly updated cdb hasn't been a
problem for us, but it seems it could be in the future so I think I'll
change that to SQL too.


--
Rob

[Viktor Dukhovni]

Post by Wietse Venema
Yes, but the check happens at the beginning of an SMTP session, not
in the middle. A Postfix process does not reopen files mid-flight
because it may not have sufficient privileges to do so (iin addition,
cdb files are renamed, not overwritten, so the lookup result does
not change during the lifetime of the SMTP daemon process; if you
need instant visibility, use LMDB, LDAP or *SQL).

I see. Except for recipients we already have most lookup tables in SQL. So
far having recipients in a regularly updated cdb hasn't been a problem for
us, but it seems it could be in the future so I think I'll change that to
SQL too.

Using CDB or Berkeley DB is unlikely to be a problem for recipient
validation tables. It is only a problem for "check_sasl_access"
access(5) tables, because you really want changes to these to apply
to existing connections "in flight". With most other data, a bit
of delay in visibility for clients that stay connected is not a
problem.