Too many recipients: recipient_limit + hard_error_limit below 50

[Mark.Martinec+ (Mark Martinec)]

I'll describe a problem where one Postfix is unable to deliver
a many-recipient mail to another Postfix which has unreasonably
low smtpd_recipient_limit - and try to propose some solution.
The guilty one here in my opinion is the smtpd_hard_error_limit.

The rfc2821 does describe a procedure (4.5.3.1 Size limits and
minimums / recipients buffer) to be followed when the sending
mailer has more recipients than the recipient is willing to handle.
It involves sending '452 Too many recipients' for each excessive
recipient, then proceeding to DATA and acomplish that the first
chunk of recipients does receive the mail, and the rest can be
retried on the next attempt. The rfc2821 also requires that the
limit must be at least 100.

I know that setting smtpd_recipient_limit below 100 is a violation
of standard, yet some sites choose to do so, and I have no influence
over them, myself being on the sending side. I can only try
to learn their limit, and set up a special Postfix service with
reduced *_destination_recipient_limit to feed such sites.

So far I lived under the impression that the rfc2821/4.5.3.1
procedure would cope even in cases with a limit below 100, say 25.
This time I took the time to verify that this is not the case,
and unnecessarily so.

The showstopper is the smtpd_hard_error_limit with a default value
of 20, which is being triggered by '452 Too many recipients' replies
to RCPT TO command.

What happens it:
- sending Postfix tries to deliver a 50-recipient mail;
- recipient has smtpd_recipient_limit=25 (too low), and
the smtpd_hard_error_limit=20 (a default);
- it accepts first 25 'RCPT TO' with a 250 status;
- it then accepts the next 20 'RCPT TO's giving a 452 for each;
- at this point the receiving side drops the connection due to
the smtpd_hard_error_limit being reached, and the sending side
never gets a chance to go to DATA transfer and get the job done
at least for the first valid 25 recipients.

I propose that Postfix:
- does not count 452 replies to 'RCPT TO' commands against the
smtpd_hard_error_limit, either not counting them at all,
or establishing another limit for them with higher default value;
and/or
- that a startup warning is issued when the sum of
smtpd_recipient_limit + smtpd_hard_error_limit is below 100.

Mark

[Wietse Venema]

Post by Mark.Martinec+ (Mark Martinec)
- does not count 452 replies to 'RCPT TO' commands against the
smtpd_hard_error_limit, either not counting them at all,
or establishing another limit for them with higher default value;

Sorry, no exceptions.

Post by Mark.Martinec+ (Mark Martinec)
and/or
- that a startup warning is issued when the sum of
smtpd_recipient_limit + smtpd_hard_error_limit is below 100.

That does not solve your problem: there is no requirement that the
CLIENT sends AT MOST 100 recipients.

There is, however, a requirement in the RFC that the SERVER accept
AT LEAST 100 recipients. This can be enforced on the Postfix server
side.

I'll put that on the TODO list for Postfix 2.2 snapshots. This is
too late for making an incompatible change with 2.1.

Wietse

[Wietse Venema]

Post by Wietse Venema
That does not solve your problem: there is no requirement that the
CLIENT sends AT MOST 100 recipients.
There is, however, a requirement in the RFC that the SERVER accept
AT LEAST 100 recipients. This can be enforced on the Postfix server
side.

Though the RFC is no so bold, I've always interpreted the requirement that
the server accept at least 100 to mean that no client may send more than
100 except by prior arrangement with the specific target server.

But that's not what the RFC says. If it wanted a upper bound of
100 for sending, and a lower bound of 100 for receiving, then it
would have specified an internet-wide limit of 100.

Likely the RFC expects the server to patiently return an unbounded number
of 452 responses without dropping the connection. This is not in fact
reasonable.

That is how I read the RFC. Send 4xx for the recipients that you
won't accept now so they can be sent as part of a later MAIL
transaction. Aside from smtpd_hard_error_limit, this is exactly
what Postfix implements.

The failure of the RFC to spell this out is IMHO a bug in the RFC.

RFC 821 recommended sending "552 Too many recipients" but that
was fixed later in RFC 1123 or something. Postfix always treats
5XX as a don't try again class error.

Wietse

[Wietse Venema]

Post by Wietse Venema
That does not solve your problem: there is no requirement
that the CLIENT sends AT MOST 100 recipients.

True, but at least for interoperability between Postfixes,
which by default limit the number of recipients sent to 50,
smtpd_hard_error_limit >= 50 - smtpd_recipient_limit
smtpd_recipient_limit >= 50 - smtpd_hard_error_limit
would deal for most practical purposes with ignorant sites,
after they eventually upgrade to the version with enforcement.

As long as Postfix is not the largest email market share holder,
optimizing for Postfix-to-Postfix does not make sense. Amdahl's
law says optimize the common case not the exception.

I'm not inclined to put in all restrictions on parameters that
don't work anyway if you're talking to non-Postfix systems.

Wietse

[Wietse Venema]

1. I presume ESMTP PIPELINING makes the whole 452 detection a bit
difficult as a single Ethernet packet will carry around 50 of these
messages - that way, the client will have already sent more than the
server's smtpd_hard_error_limit before even seeing the first 452
response.

There is a reason why Postfix sends significantly less recipients
than the required 100 minimum server limit.

2. Would there be a chance that the Postfix smtp client (this is 2.2
stuff, of course) caches - for a certain MX or recipient - after what
recipient count the site started sending out 452 replies? Then the
first retry might stand a chance of getting the mail through in small
bits.

Yes. This cache already exists and it is called the deferred queue.
If an MTA does not accept $smtpd_destination_recipient_limit
recipients (default: 50), then:

1 - Either the remote site is mis-configured. Tough, it's their
mail that they aren't receiving and that will teach them.

2 - The local MTA sends more than the recommended number of
recipients. Tough. It's their queue and that will teach them.

I don't see why Postfix needs to be extended up with additional
protocol handling features just because some untermensch changes
parameters without knowing what they are doing.

The fix is not to added more mechanisms to Postfix, but to educate.

- Release N: Log the warning and ignore invalid the parameter
setting.

- Release N+M: Abort when an invalid setting is found.

I am considering putting in the "release N" change by 2.1.

Wietse

[Wietse Venema]

Don't shoot yourself in the foot. Do not set the recipient_limit below
100. Perhaps the right "fix" is to simply disallow recipient limits under
100 in smtpd...

This doesn't work, because the "fix" will a. not help delivery to
misconfigured non-Postfix MTAs and b. it will take years before the
"fix" has propagated to Debian Linux, for instance. Given past
experience with release cycles, chances are such a patch would take
until 2008 to become effective in Debian/stable anyways.

Can anyone explain if this <100 limit is a Debian feature or
just an idiot adminstrator thing?

Wietse

[Wietse Venema]

Post by Wietse Venema

This doesn't work, because the "fix" will a. not help delivery to
misconfigured non-Postfix MTAs and b. it will take years before the
"fix" has propagated to Debian Linux, for instance. Given past
experience with release cycles, chances are such a patch would take
until 2008 to become effective in Debian/stable anyways.

Can anyone explain if this <100 limit is a Debian feature or
just an idiot adminstrator thing?

Which MTA is the original Poster talking about?

Postfix, with (smtpd_recipient_limit + smtpd_hard_error_limit) << 100.

In either case, no point optimizing the main release because some
minority of MTAs on the Internet is mis-configured.

[Wietse Venema]

1. I presume ESMTP PIPELINING makes the whole 452 detection a bit
difficult as a single Ethernet packet will carry around 50 of these
messages - that way, the client will have already sent
more than the
server's smtpd_hard_error_limit before even seeing the first 452
response.

I'm glad Matthias brought this up. This is the exact problem which
happens with Exchange 2000/2003 when Postfix is used as a relay_host
with the default smtpd_hard_error_limit (20) and smtpd_recipient_limit
(1000). Exchange pipelines as many recipients as it possibly can until
it gets a 452, but Postfix hangs up on it because of the
smtpd_hard_error_limit. So you have deadlock and out of the box
Exchange isn't compatible with Postfix.

You mean, they will send over a thousand recipients pipelined, and
then expect to get a burst of tens of 452 replies back when they
hit the server's recipient limit?

Just trying to figure out what is going on in the brains there. The
workaround for these would be to zero the soft and hard error limits.

Wietse

---
RFC 2821 section 4.5.3.1 mandates the minimum number of recipients any
SMTP implementation supports to be 100. This is for inbound.

[Wietse Venema]

Post by Wietse Venema

1. I presume ESMTP PIPELINING makes the whole 452 detection a bit
difficult as a single Ethernet packet will carry around 50 of these
messages - that way, the client will have already sent
more than the
server's smtpd_hard_error_limit before even seeing the first 452
response.

I'm glad Matthias brought this up. This is the exact problem which
happens with Exchange 2000/2003 when Postfix is used as a relay_host
with the default smtpd_hard_error_limit (20) and smtpd_recipient_limit
(1000). Exchange pipelines as many recipients as it possibly can until
it gets a 452, but Postfix hangs up on it because of the
smtpd_hard_error_limit. So you have deadlock and out of the box
Exchange isn't compatible with Postfix.

You mean, they will send over a thousand recipients pipelined, and
then expect to get a burst of tens of 452 replies back when they
hit the server's recipient limit?

The way I understand Chris's post, they will send the entire envelope
(arbitrarily large) if they can. Pipelining will allow them to send
blocks of recipient addresses (as many as fit in a TCP window), the first
452 returned will terminate the processing of the envelope.
Problem is, Postfix has to be willing to return 452 errors for a whole
window full of recipients without dropping the connection.
Indeed having the recipient limit violations count towards the hard error
limit is problematic, but not bounding 452 error counts is also not
acceptable.
I think that the solution (some future release) is to keep a separate
counter for recipient limit violations, and set a more generous bound for
the counter (at least 1000). In addition, recipient limit violation would
not cause any tarpit delays.

Like this?


*** ./smtpd.c- Wed Apr 21 09:02:58 2004
--- ./smtpd.c Wed Apr 21 10:43:44 2004
***************
*** 1414,1419 ****
--- 1414,1420 ----
}
if (var_smtpd_rcpt_limit && state->rcpt_count >= var_smtpd_rcpt_limit) {
state->error_mask |= MAIL_ERROR_POLICY;
+ state->error_count--;
smtpd_chat_reply(state, "452 Error: too many recipients");
return (-1);
}

[Wietse Venema]

Post by Wietse Venema

The way I understand Chris's post, they will send the entire envelope
(arbitrarily large) if they can. Pipelining will allow them to send
blocks of recipient addresses (as many as fit in a TCP window), the first
452 returned will terminate the processing of the envelope.
Problem is, Postfix has to be willing to return 452 errors for a whole
window full of recipients without dropping the connection.
Indeed having the recipient limit violations count towards the hard error
limit is problematic, but not bounding 452 error counts is also not
acceptable.
I think that the solution (some future release) is to keep a separate
counter for recipient limit violations, and set a more generous bound for
the counter (at least 1000). In addition, recipient limit violation would
not cause any tarpit delays.

Like this?
*** ./smtpd.c- Wed Apr 21 09:02:58 2004
--- ./smtpd.c Wed Apr 21 10:43:44 2004

Actually, this is better: don't raise the error condition.

However, I agree that this is not safe unless there is a hard limit
on the amount of replies that Postfix will give in this manner.

Wietse

*** ./smtpd.c- Wed Apr 21 09:02:58 2004
--- ./smtpd.c Wed Apr 21 10:49:03 2004
***************
*** 1413,1421 ****
}
}
if (var_smtpd_rcpt_limit && state->rcpt_count >= var_smtpd_rcpt_limit) {
- state->error_mask |= MAIL_ERROR_POLICY;
smtpd_chat_reply(state, "452 Error: too many recipients");
! return (-1);
}
if (SMTPD_STAND_ALONE(state) == 0) {
if ((err = smtpd_check_rcpt(state, argv[2].strval)) != 0) {
--- 1413,1420 ----
}
}
if (var_smtpd_rcpt_limit && state->rcpt_count >= var_smtpd_rcpt_limit) {
smtpd_chat_reply(state, "452 Error: too many recipients");
! return (0);
}
if (SMTPD_STAND_ALONE(state) == 0) {
if ((err = smtpd_check_rcpt(state, argv[2].strval)) != 0) {

[Wietse Venema]

Post by Wietse Venema
Actually, this is better: don't raise the error condition.
However, I agree that this is not safe unless there is a hard limit
on the amount of replies that Postfix will give in this manner.

Thanks Wietse.
Which version will this patch be incorporated into?

With Postfix 2.1, the 452 replies are pipelined without error sleep
delays, unless the client already bumped the error counter past
the soft error limit. Changing that requires redesign and testing,
and will push the 2.1 release until July 2004.

With Postfix 2.0, the 452 replies are not pipelined. Postfix will
pause $smtpd_error_delay for every 452 reply (more delay if the
client already bumped the error counter past the soft error limit).
Changing that is beyond the support contract for a stable release.

Postfix 1.1 is no longer supported.

Wietse

[Wietse Venema]

I'm glad Matthias brought this up. This is the exact problem which
happens with Exchange 2000/2003 when Postfix is used as a relay_host
with the default smtpd_hard_error_limit (20) and smtpd_recipient_limit
(1000). Exchange pipelines as many recipients as it possibly can until
it gets a 452, but Postfix hangs up on it because of the
smtpd_hard_error_limit. So you have deadlock and out of the box

How many recipients will Exchange 2000/2003 have sent "ahead of
the server replies" when the server starts replying "452 too many
recipients"?

Simply accepting an unlimited number of RCPT TO commands is not an
option. Postfix maintains a record of SMTP commands for trouble
shooting. This record size must be limited in size to avoid memory
exhaustion vulnerabilities.

Wietse

[Wietse Venema]

Post by Wietse Venema

I'm glad Matthias brought this up. This is the exact problem which
happens with Exchange 2000/2003 when Postfix is used as a relay_host
with the default smtpd_hard_error_limit (20) and smtpd_recipient_limit
(1000). Exchange pipelines as many recipients as it possibly can until
it gets a 452, but Postfix hangs up on it because of the
smtpd_hard_error_limit. So you have deadlock and out of the box

How many recipients will Exchange 2000/2003 have sent "ahead of
the server replies" when the server starts replying "452 too many
recipients"?
Simply accepting an unlimited number of RCPT TO commands is not an
option. Postfix maintains a record of SMTP commands for trouble
shooting. This record size must be limited in size to avoid memory
exhaustion vulnerabilities.

I came at the same number from the other end: 1000*2048 = 2MB for
maximal length RCPT TO commands.

I do hope that actual implementations need much less than this.

By the way, do we know how MS pipelining works? Postfix buffers up
to 4kbyte of command or reply text in process, so that multiple
commands or replies are sent at once, and avoids long periods of
silence by flushing the buffer early.

Wietse

[Wietse Venema]

Post by Mark.Martinec+ (Mark Martinec)

I'm glad Matthias brought this up. This is the exact problem which
happens with Exchange 2000/2003 when Postfix is used as a relay_host
with the default smtpd_hard_error_limit (20) and

smtpd_recipient_limit

(1000). Exchange pipelines as many recipients as it

possibly can until

it gets a 452, but Postfix hangs up on it because of the
smtpd_hard_error_limit. So you have deadlock and out of the box

How many recipients will Exchange 2000/2003 have sent "ahead of
the server replies" when the server starts replying "452 too many
recipients"?

I can get it to top out between 300 and 400 (using RCPT TO: addresses
approximately 20 characters long).

Ok, that's about 10 kbytes. the proposed 1000 recipient overshoot limit
can deal with this.

Wietse

[Wietse Venema]

Only if your recipient limit is < 100 or you receive mail from pipelined
clients that push their luck by sending more than 1000 (actually your
recipient limit) recipients.

It appears to be 10,000

..

smtpd_recipient_limit = 1000

Not a problem unless you receive mail with more than 1020 recipients.

Wietse

[Wietse Venema]

Post by Wietse Venema

I'm glad Matthias brought this up. This is the exact problem which
happens with Exchange 2000/2003 when Postfix is used as a relay_host
with the default smtpd_hard_error_limit (20) and smtpd_recipient_limit
(1000). Exchange pipelines as many recipients as it possibly can until
it gets a 452, but Postfix hangs up on it because of the
smtpd_hard_error_limit. So you have deadlock and out of the box

How many recipients will Exchange 2000/2003 have sent "ahead of
the server replies" when the server starts replying "452 too many
recipients"?
Simply accepting an unlimited number of RCPT TO commands is not an
option. Postfix maintains a record of SMTP commands for trouble
shooting. This record size must be limited in size to avoid memory
exhaustion vulnerabilities.

is 13 bytes. So 1000 RCPT commands are at least 12K which should be
plenty. Here's a more complete patch (less postconf.5 and derived
documentation changes)...

I have only one change: the recipient overshoot count is reset only
after successful delivery. I moved this to the rcpt_reset() routine
so it also works after RSET.

The nasty is that the overshoot count reset must not happen without
command history reset. Currently, rcpt_reset() is always called as
the last in a fixed reset sequence of (chat_reset, mail_reset,
rcpt_reset) but enforcing this sequence would require something
like a DAMN_THE TORPEDOES_AND_RESET_EVERYTHING() macro.

Wietse

@@ -1713,6 +1723,7 @@
if (state->err == CLEANUP_STAT_OK) {
state->error_count = 0;
state->error_mask = 0;
+ state->rcpt_limit_count = 0;
state->junk_cmds = 0;
if (state->queue_id)
smtpd_chat_reply(state, "250 Ok: queued as %s", state->queue_id);

[Wietse Venema]

Post by Wietse Venema

is 13 bytes. So 1000 RCPT commands are at least 12K which should be
plenty. Here's a more complete patch (less postconf.5 and derived
documentation changes)...

I have only one change: the recipient overshoot count is reset only
after successful delivery. I moved this to the rcpt_reset() routine
so it also works after RSET.

A related observation is that returning 4XX codes instead of 5XX codes
(soft_bounce = yes, and the various reject codes) is not always safe in
combination with low hard error limits.
The hard error limit used to be 100, it is now 20, but a legitimate sender
with 20 (or is it 21) bad addresses that are refused with 4XX codes in an
envelope with 100 recipients will never be able to send to the good
recipients and the message will stay in the sender's queue until it times
out. So sometimes returning 5XX codes is safer than returning 4XX codes.

The old limit was 100, in the expectation that reasonable clients
would not send more than the RFC required server supported limit
of 100. This theory was disproved already today.

On the client side (Postfix 2.2?), this means that it may be reasonable
when some recipients are accepted and others deferred, but message
delivery is aborted with a 421 error, to immediately reconnect and retry
just the accepted recipients so that at least some progress is made.

By this reasoning the hard error limit would have to exceed the
recipient limit (1000) + the overshoot limit (1000) which practically
means that the hard/soft limit should be inifinite by default.

Wietse
limit

[Wietse Venema]

Post by Wietse Venema

On the client side (Postfix 2.2?), this means that it may be reasonable
when some recipients are accepted and others deferred, but message
delivery is aborted with a 421 error, to immediately reconnect and retry
just the accepted recipients so that at least some progress is made.

By this reasoning the hard error limit would have to exceed the
recipient limit (1000) + the overshoot limit (1000) which practically
means that the hard/soft limit should be inifinite by default.

I don't understand... I was suggesting a client change... Indeed viewed
more broadly in the context of accommodating clients that can't/don't
implement such workarounds, indeed in theory enough 4XX recipients can
prevent delivery to good recipients, but in practice this is rare.

To understand: I'm trying to solve the problem at the source (it's
the server that initiated the disconnect, after all) instead of
piling up workarounds on the client (reconnect immediately after
421 and resume delivery in the middle of the recipient list).

If the server is willing to receive up to 1000 recipients, then it
should be prepared for the possibility that the first 999 recipients
will have to be 4XX deferred and that only the last recipient will
be accepted, however unlikely that may be. Overshoot by agressive
PIPELINING is a separate issue.

Possible server side solutions:

1 - 4XX Errors don't count towards the server hard error limit. This
means maintaining multiple error counters because I don't want to
allow 4XX errors to happen without some limit.

2 - 4XX Errors do count towards the hard error limit, and the default
server hard error limit equals the recipient limit, so that the
server will not disconnect prematurely while receiving legitimate
mail. It will just slow down a little after the soft error limit
is reached.

The change #2 is possible for Postfix 2.1.

In order to fix 2.0, both the server soft and hard error limits
would have to default to the (possibly reduced) server recipient
limit. Small sites will not notice any difference at all except
that Postfix wastes less time, so the number of SMTPD processes
goes down a little. Big sites will notice fewer "too many errors"
while receiving multi-recipient mail.

Sites that are using older versions will eventually be replaced.
I prefer to eliminate the problem at the source, instead of keeping
the problem in the server and piling up workarounds in the client.

My only recommendation might be to change the *default* error limits as
smtpd_soft_error_limit = 20
smtpd_hard_error_limit = 100
Those who find that more aggressive limits (10/20 or even 1/2!) work for
them can be so picky at their own peril, but the smtp(8) client can be
enhanced to work around the problem by retrying accepted recipients or if
all are deferred with 4XX and the session aborts with 421, by retrying the
remaining recipients, so that each delivery makes progress or discovers
that all the recipients are unquestionably not deliverable at that time.

The server hard limit of 100 does not solve the problem as long as
the server recipient limit is an order of magnitude larger. Although
only few sites receive 1000-recipient mail, that is no excuse to
ship a default configuration that cripples such sites.

Changing the SMTP client to reconnect after 421 and to resume in
the middle of a recipient list means investing a lot of energy
energy in a workaround. I would much prefer to eliminate the
problem at the source, and eliminate the premature server disconnect
from the default configuration.

This means adding some extra warnings to the new server hard limit
documentation that reducing the hard limit may result in persistent
mail delivery failures.

Wietse

[Wietse Venema]

For all,
smtpd_recipient_limit = 1
smtpd_soft_error_limit = 1
smtpd_hard_error_limit = 2
smtpd_error_sleep_time = 20s
http://www.plusone.com/gaptuning/postfix/exchange040421.log

Looks like up to some 400 recipients of overshoot.

Postfix 2.1.0 (and 2.0.20) will handle that it without sweat.

Thanks for the numbers.

Wietse

Looks like an ~12K window... If you set the recipient limit to 1 (test
environment) the soft error limits to 1, the hard error limit to 2 and
the sleep time 20s, how many recipients does Exchange manage to put on
the wire (tcpdump)...

Chris

[Wietse Venema]

Post by Wietse Venema

http://www.plusone.com/gaptuning/postfix/exchange040421.log

Looks like up to some 400 recipients of overshoot.
Postfix 2.1.0 (and 2.0.20) will handle that it without sweat.
Thanks for the numbers.

Thank you for implementing the change so quickly, and retroactively!
Just to be clear given the backscatter-effect of reading this entire
thread: You've adopted Viktor's patch (with your own modification) that
creates a smtpd_recipient_count_error_limit variable which is by default
set to 1000.

No. I wrote a patch, Victor extended it, and I fixed Victor's extension.

What should one do with their smtpd_(hard|soft)_error_limit variables?
I've mine set to 1000 (from defaults) to workaround this. What will the
new defaults be (same, different)?

I am doing nothing with them. That change would be too visible and
most certainly could not be made retroactively.

The pipelining overshoot fix should take care of the most common
problem: sending more recipients than the hard error limit allows.

You can see the result in the first Postfix 2.2 snapshot. 2.1 is
frozen and will be released in a day or so. Further changes must
be field tested in 2.2 snapshots.

Wietse