Balancing destination concurrency + rate delay

[Steve Jenkins]

I have custom transports set up for Gmail, Hotmail, Yahoo, and AOL. And as
anyone sending lots of mail will probably agree, Yahoo is the toughest to
manage. :)

Here are my current main.cf settings:

yahoo_initial_destination_concurrency = 1
yahoo_destination_concurrency_limit = 4
yahoo_destination_recipient_limit = 2
yahoo_destination_rate_delay = 1s

I'd like to solicit opinions as to what's more likely to keep a big MTA
like Yahoo "happy" - a lower destination concurrency limit or a higher rate
delay? Obviously, the "right" answer will be some balance of the two, and
what works for one mailer won't work for another, because sender reputation
is involved. But my mailers are still slowly delivering my backlogged
newsletter that's been sending for 24+ hours now, and I want to deliver as
efficiently as possible without upsetting the remote MTAs. I just want to
know which settings I should fiddle with, while I still have plenty of
outgoing mail with which to experiment.

Thx,

SteveJ

[Wietse Venema]

Post by Steve Jenkins
yahoo_destination_concurrency_limit = 4
yahoo_destination_recipient_limit = 2
yahoo_destination_rate_delay = 1s

As documented, rate_delay enforces a delay BETWEEN deliveries to
the same destination, and therefore, the concurrency to that
destination is always 1. I see no way to enforce delays BETWEEN
simultaneous deliveries to the same destination.

The documentation also has a warning concerning per-destination
recipient limit (the concept of destination depends on the recipient
limit). Different destinations are delivered in parallel.

Wietse

[Steve Jenkins]

Post by Wietse Venema

Post by Steve Jenkins
yahoo_destination_concurrency_limit = 4
yahoo_destination_recipient_limit = 2
yahoo_destination_rate_delay = 1s

As documented, rate_delay enforces a delay BETWEEN deliveries to
the same destination, and therefore, the concurrency to that
destination is always 1. I see no way to enforce delays BETWEEN
simultaneous deliveries to the same destination.

Thanks for clearing that up. So I can tinker with destination concurrency
OR rate delay, but not both.

So now I suppose my follow-up question is what's more likely to keep the
big MTAs happier? Delays between individual deliveries or limited
concurrency?

Post by Wietse Venema
The documentation also has a warning concerning per-destination
recipient limit (the concept of destination depends on the recipient
limit). Different destinations are delivered in parallel.

I read that warning, but wasn't sure I understood it properly. I know that
setting destination recipient limit to anything above one defines a
"destination" as a domain, so by setting it to 2 we're saying "wait 1s
between every delivery to a Yahoo domain," right? And if every message sent
is unique, would that mean there'd be no difference between a setting of 2
and the default of 50 in this case?

[Rafael Azevedo]

I read that warning, but wasn't sure I understood it properly. I know that setting destination recipient limit to anything above one > defines a "destination" as a domain, so by setting it to 2 we're saying "wait 1s between every delivery to a Yahoo domain,"
right? And if every message sent is unique, would that mean there'd be no difference between a setting of 2 and the default of 50
in this case?

That's right, but Postfix will send one message at time to the destination, even if the message is not unique.
I'd suggest you to map all Yahoo's domains. They have basically every extension for each country (ie: yahoo.fr, yahoo.de, etc). Putting them all in the same transport queue will help you to control parallel deliveries, or forcing them all to use the same destination (ie: yahoo:yahoo.com on transport table).

I'm not sure about how Postfix would behave if you use 2 or 50 in the default_destination_concurrency_limit , but I guess if you use 2 it will limit the concurrency delivery to the same destination to 2 at time. If you use 50, you'll have 50 postfix threads sending to the same destination (I wouldnt suggest that if you dont want to get blocked by Yahoo). I guess Wietse can help you better than me in this case.

Good luck.
BR
- Rafael

[Wietse Venema]

Post by Steve Jenkins
I read that warning, but wasn't sure I understood it properly. I
know that setting destination recipient limit to anything above
one defines a "destination" as a domain, so by setting it to 2
we're saying "wait 1s between every delivery to a Yahoo domain,"

As for what settings work better with high-volume receivers, I
suggest a search query for "aol postmaster", "yahoo postmaster" etc.

With per-destination recipient limit > 1, destination = domain,
and different destinations(=domains) are delivered in parallal
subject to concurrency limits. This enforces fairness between
different destinations(=domains).
With rate delay > 0, per-destination(=domain) concurrency is 1.

With per-destination recipient limit = 0, destination = recipient,
and different destinations(=recipients) are delivered in parallal
subject to concurrency limits. This enforces fairness between
different destinations(=recipients).
This setting is used by the local delivery agent.

Wietse

[Noel Jones]

Post by Wietse Venema

Post by Steve Jenkins
I read that warning, but wasn't sure I understood it properly. I
know that setting destination recipient limit to anything above
one defines a "destination" as a domain, so by setting it to 2
we're saying "wait 1s between every delivery to a Yahoo domain,"

As for what settings work better with high-volume receivers, I
suggest a search query for "aol postmaster", "yahoo postmaster" etc.
With per-destination recipient limit > 1, destination = domain,
and different destinations(=domains) are delivered in parallal
subject to concurrency limits. This enforces fairness between
different destinations(=domains).
With rate delay > 0, per-destination(=domain) concurrency is 1.
With per-destination recipient limit = 0, destination = recipient,

that was intended to read:
With per-destination recipient limit = 1, destination = recipient,

Post by Wietse Venema
and different destinations(=recipients) are delivered in parallal
subject to concurrency limits. This enforces fairness between
different destinations(=recipients).
This setting is used by the local delivery agent.
Wietse

[Steve Jenkins]

Post by Wietse Venema
As for what settings work better with high-volume receivers, I
suggest a search query for "aol postmaster", "yahoo postmaster" etc.

Agreed - but Yahoo is really the only one we're having issues with (even
after complying with all their guidelines here):

http://help.yahoo.com/kb/index?page=content&y=PROD_MAIL_ML&locale=en_US&id=SLN3435

Yes, we do confirmed opt-in, and DKIM, and we unsubscribe bounces within 24
hours, and have FBLs set up, and dedicated IPs, and have good sender
reputation scores, and have completed Yahoo's Bulk Sender Application Form,
etc, etc. The fact that 2 days after sending nearly 400K newsletters to our
subscribers, we still have 70K in our fallback relay's deferred queue, and
that 100% of them are @yahoo.com addresses, means we still need to work on
our Yahoo-specific Postfix settings.

Unfortunately, Yahoo's official word on outbound mailer settings is too
general to be of much use:

"*Use common-sense settings.* While we have not published guidelines for
numbers of connections you can concurrently use, we ask that you treat our
resources with respect. The more you take, the fewer there are for others,
which may force us to defer your connections."

So that's why I'm asking other Postfix users who run high-volume mailers if
they would be so kind as to share their experiences with Postfix
concurrency limits and/or rate delays when dealing with Yahoo in
particular. Is that a taboo subject here? Or perhaps just not the right
forum in which to discuss it?

Thx,

SteveJ

[Robert Schetterer]

Post by Steve Jenkins
Agreed - but Yahoo is really the only one we're having issues with (even
after complying with all their guidelines here)

last time i had to do this , yahoo needs 3 weeks for whitelisting
the new ip, used for a mail list server

at the end if you already followed recommands for bulk mails on this
list ( and archive ) with your mail server setup , there is less what
you can do more, you may try more cascading (virtual)mailservers on
other ips and/or more instances etc as well as other tricks
but there is no magic ( beyond paying money ) to press someone accept
your mail in timelimits you prefer, at last its not a question of
postfix, its yahoos ( in this case ) decision


Best Regards
MfG Robert Schetterer

[Viktor Dukhovni]

Post by Steve Jenkins
Agreed - but Yahoo is really the only one we're having issues with (even
http://help.yahoo.com/kb/index?page=content&y=PROD_MAIL_ML&locale=en_US&id=SLN3435

Yes, they are willing to cripple SMTP and expect everyone to cope,
because they are too big to ignore. :-)

Rumour has it that established proffesional bulk-senders typically
don't turn up new IP addresses (no existing good reputation) to
full capacity overnight. Rather, their sending software gradually
ramps-up load to a new address allowing its reputation to build-up
over time (the first few weeks may be at a low load, then add more,
...).

Once your IP reputation is good, you should be able to use reasonable
concurrency... You may want disable the demand connection cache
for Yahoo. This cache is primarily useful for destinations that
sporadically have some of their MX records unreachable at the TCP
layer.

Yahoo's MX SMTP pool essentially never has an IP address that is
completely down. Instead, you can set a low "smtp_connect_timeout",
just in case.

master.cf:
yahoo unix - - n - - smtp
-o smtp_connect_timeout=$yahoo_connect_timeout
-o smtp_connection_cache_on_demand=$yahoo_connection_cache_on_demand

main.cf:
# Good enough for light RTT to the moon
yahoo_connect_timeout=3s

# Don't annoy them with connection reuse.
yahoo_connection_cache_on_demand=no

At that point you may not even need rate delays, just set a modest
concurrency, and typical SMTP transaction latency of 0.2-0.5s (
with spam checks, RBL lookups, ...) will give you at most 2-5
messages per unit concurrency per second.

[Steve Jenkins]

On Fri, Jan 18, 2013 at 11:36 AM, Viktor Dukhovni <

Post by Viktor Dukhovni
Yes, they are willing to cripple SMTP and expect everyone to cope,
because they are too big to ignore. :-)

Sad, but true.

Post by Viktor Dukhovni
At that point you may not even need rate delays, just set a modest
concurrency, and typical SMTP transaction latency of 0.2-0.5s (
with spam checks, RBL lookups, ...) will give you at most 2-5
messages per unit concurrency per second.

Would this be considered "modest?"

yahoo_initial_destination_concurrency = 1
yahoo_destination_concurrency_limit = 4
yahoo_destination_recipient_limit = 2
#yahoo_destination_rate_delay = 2s
yahoo_connect_timeout=3s
yahoo_connection_cache_on_demand=no

Thx,

SteveJ

[Viktor Dukhovni]

Post by Steve Jenkins

Post by Viktor Dukhovni
At that point you may not even need rate delays, just set a modest
concurrency, and typical SMTP transaction latency of 0.2-0.5s (
with spam checks, RBL lookups, ...) will give you at most 2-5
messages per unit concurrency per second.

Would this be considered "modest?"
yahoo_initial_destination_concurrency = 1
yahoo_destination_concurrency_limit = 4
yahoo_destination_recipient_limit = 2
#yahoo_destination_rate_delay = 2s
yahoo_connect_timeout=3s
yahoo_connection_cache_on_demand=no

You'll have to figure that out for yourself mostly, my gut reaction
is that the recipient limit is too low, if you ever did have a
multi-recipient message, you'd split it into too many pieces.
Rather, set something in the 10-20 range (the RFC requires
servers to support at least 100, and Postfix by default sends
up to 50 and takes in up to 1000).

After burning in your IP at low volume, with mail typically not
delayed by Yahoo at all, I would try to grow the (destination aka
connection) concurrency limit to something in the 5-10 range.

The main thing that will matter is "burning in" a new IP address,
once they've learned you're not evil, I would hope their rate limits
won't get in your way at all.