Discussion:
pipe flags vs lmtp
(too old to reply)
Daniel L. Miller
2012-04-07 06:49:40 UTC
Permalink
With the pipe transport, there are flag settings that affect how the
message is processed - particularly the ability to add the Delivered-To
and X-Original-To headers. Is similar functionality available with lmtp?
--
Daniel
Wietse Venema
2012-04-07 11:30:12 UTC
Permalink
Post by Daniel L. Miller
With the pipe transport, there are flag settings that affect how the
message is processed - particularly the ability to add the Delivered-To
and X-Original-To headers. Is similar functionality available with lmtp?
All Postfix features are documented. If documentation is missing,
please report a bug.

Wietse
Daniel L. Miller
2012-04-07 17:21:20 UTC
Permalink
Post by Wietse Venema
Post by Daniel L. Miller
With the pipe transport, there are flag settings that affect how the
message is processed - particularly the ability to add the Delivered-To
and X-Original-To headers. Is similar functionality available with lmtp?
All Postfix features are documented. If documentation is missing,
please report a bug.
That's my question - I don't know if the feature I'm asking about exists!

I assume therefore that the ability to add the Delivered/Original
headers within Postfix only exists for the pipe transport - and is
unavailable for the other transports.
--
Daniel
Daniel L. Miller
2012-04-07 18:09:27 UTC
Permalink
Post by Daniel L. Miller
Post by Wietse Venema
Post by Daniel L. Miller
With the pipe transport, there are flag settings that affect how the
message is processed - particularly the ability to add the Delivered-To
and X-Original-To headers. Is similar functionality available with
lmtp?
All Postfix features are documented. If documentation is missing,
please report a bug.
That's my question - I don't know if the feature I'm asking about exists!
I assume therefore that the ability to add the Delivered/Original
headers within Postfix only exists for the pipe transport - and is
unavailable for the other transports.
I see the virtual agent explicitly adds these headers. But no mention
is made of that in the lmtp or smtp agents.

So I have to return to my original question - is there a way to have the
Delivered-To and X-Original-To headers added to mail processed by lmtp?
--
Daniel
/dev/rob0
2012-04-07 18:29:16 UTC
Permalink
Post by Daniel L. Miller
Post by Daniel L. Miller
Post by Wietse Venema
Post by Daniel L. Miller
With the pipe transport, there are flag settings that affect how
the message is processed - particularly the ability to add the
Delivered-To and X-Original-To headers. Is similar
functionality available with lmtp?
All Postfix features are documented. If documentation is missing,
please report a bug.
That's my question - I don't know if the feature I'm asking about
exists!
I assume therefore that the ability to add the Delivered/Original
headers within Postfix only exists for the pipe transport - and is
unavailable for the other transports.
I see the virtual agent explicitly adds these headers. But no
mention is made of that in the lmtp or smtp agents.
So I have to return to my original question - is there a way to
have the Delivered-To and X-Original-To headers added to mail
processed by lmtp?
It's not possible to list all features which are not implemented.
Documentation should only attempt to list and describe features which
ARE implemented.

So, it appears that you're talking about a feature request.
--
http://rob0.nodns4.us/ -- system administration and consulting
Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Wietse Venema
2012-04-07 20:20:20 UTC
Permalink
Post by Daniel L. Miller
Post by Wietse Venema
Post by Daniel L. Miller
With the pipe transport, there are flag settings that affect how the
message is processed - particularly the ability to add the Delivered-To
and X-Original-To headers. Is similar functionality available with lmtp?
All Postfix features are documented. If documentation is missing,
please report a bug.
That's my question - I don't know if the feature I'm asking about exists!
Does the documentation say that it exists? If it does not say that
the feature exists, then it does not exist.

This may be a revolutionary concept in today's world, but I can
assure you that accurate documentation used to be the norm, and
that if was unacceptable to provide UNIX commands without proper
manpage.

Wietse
Daniel L. Miller
2012-04-09 01:11:53 UTC
Permalink
Post by Wietse Venema
Post by Daniel L. Miller
Post by Wietse Venema
Post by Daniel L. Miller
With the pipe transport, there are
flag settings that affect how the message is processed - particularly
the ability to add the Delivered-To and X-Original-To headers. Is
similar functionality available with lmtp?
Post by Wietse Venema
Post by Daniel L. Miller
Post by Wietse Venema
All Postfix features are
documented. If documentation is missing, please report a bug.
Post by Wietse Venema
Post by Daniel L. Miller
That's
my question - I don't know if the feature I'm asking about exists!
Does the documentation say that it exists? If it does not say that
Post by Wietse Venema
the
feature exists, then it does not exist.
Post by Wietse Venema
This may be a revolutionary
concept in today's world, but I can
Post by Wietse Venema
assure you that accurate
documentation used to be the norm, and
Post by Wietse Venema
that if was unacceptable to
provide UNIX commands without proper
Post by Wietse Venema
manpage.
Whatever I did or said
that got people upset - I apologize. Please allow me to rephrase.

I
make use of the Delivered-To and X-Original-To headers that are set via
the pipe transport (and when I used it, the virtual transport). I'm
using the Dovecot lda via the pipe connection. Dovecot offers a lmtp
delivery agent that is (so I'm told) more efficient.

I don't see any
flag options for the ltmp transport that are similar to pipe - however
in the past people have suggested ways of using existing Postfix
features to accomplish specific tasks in ways that were not immediately
obvious to me - generally because while the features were documented, I
either didn't understand or simply didn't think of using them in that
particular fashion. The lmtp transport has a whole host of available
settings - many of which I don't immediately understand. So - with this
preface in mind, I humbly ask:

Is there a method by which I can have
these headers added while using lmtp? If not, then before I make a
feature request via whatever the appropriate and correct channels may be
- is there a particular reason why these headers are not already an
option via lmtp (aside from nobody asking for or seeing the need
previously). Is there an architectural or conceptual reason why these
headers should not be added via an lmtp connection?
--
Daniel
Noel Jones
2012-04-09 02:44:59 UTC
Permalink
I make use of the Delivered-To and X-Original-To headers that are set via the pipe transport (and when I used it, the virtual transport). I'm using the Dovecot lda via the pipe connection. Dovecot offers a lmtp delivery agent that is (so I'm told) more efficient.
Is there a method by which I can have these headers added while using lmtp?
No, the lmtp transport does not have those features. As a general
rule, those headers are intended to be added by the "final delivery"
agent, which lmtp is not.

There isn't any workaround other than using the pipe transport instead.
If not, then before I make a feature request via whatever the appropriate and correct channels may be - is there a particular reason why these headers are not already an option via lmtp (aside from nobody asking for or seeing the need previously). Is there an architectural or conceptual reason why these headers should not be added via an lmtp connection?
The request has been made and noted previously, and will be
addressed as time and resources permit. I don't know of any
structural reasons that lmtp can't have these options, but unifying
the options of all the postfix transports requires careful planning
to make sure the end result is sane, documented, maintainable, and
consistent across all of postfix. Adding this feature to lmtp
without a good idea of how that will fit into future modifications
of other transports is likely to be counter-productive in the long run.
The actual coding is probably no more than 25% of the job, and
probably the easiest part.



-- Noel Jones
/dev/rob0
2012-04-09 03:06:14 UTC
Permalink
Whatever I did or said that got people upset - I apologize.
Please allow me to rephrase.
I was not upset, hope you did not take it that way. Although now that
you mention it, I could be a bit annoyed at the line wrapping in this
post. ;)
I make use of the Delivered-To and X-Original-To headers that
are set via the pipe transport (and when I used it, the virtual
In general I don't care for using mail headers to control delivery.
transport). I'm using the Dovecot lda via the pipe connection.
Dovecot offers a lmtp delivery agent that is (so I'm told) more
efficient.
Probably so. It's a single long-running process, as opposed to
invocation of a separate LDA process per delivery.
I don't see any flag options for the ltmp transport that are
similar to pipe - however in the past people have suggested ways of
using existing Postfix features to accomplish specific tasks in
ways that were not immediately obvious to me - generally because
while the features were documented, I either didn't understand or
simply didn't think of using them in that particular fashion. The
lmtp transport has a whole host of available settings - many of
which I don't immediately understand. So - with this preface in
Is there a method by which I can have these headers added while
using lmtp?
I looked at "postconf | grep ^lmtp", and saw this gem:

http://www.postfix.org/postconf.5.html#lmtp_header_checks

which can probably do what you need. See also:

http://www.postfix.org/header_checks.5.html

Unfortunately this is a bit of a kludge, because there's no sure way
to prepend a header specific to each recipient (unless I missed
something, in which case I will probably be corrected. :) )
If not, then before I make a feature request via whatever the
appropriate and correct channels may be
(Right here would be the best place.)
- is there a particular reason why these headers are not already
an option via lmtp (aside from nobody asking for or seeing the
need previously). Is there an architectural or conceptual reason
why these headers should not be added via an lmtp connection?
I can think of a few. First, lmtp(8) is actually the smtp(8) binary
addressed by a different name. smtp does not do final delivery of
mail (final to US, but not going straight to a mailbox.) It would be
presumptuous to add Delivered-To: headers, and wrong when there are
multiple recipients.

Second, it really seems more appropriate to do this in the LMTP
daemon. It gets the envelope recipient for each mail and can do this
(in theory) much better and more accurately than Postfix can. We
don't know what the LMTP daemon is going to do with that mail ...
perhaps it will be rewritten and forwarded on somehow.

I think if there is to be a feature request here, it should be for a
macro for lmtp_header_checks to access the envelope recipient. But
that might not be possible, because although LMTP means you get
per-recipient codes after DATA, it might not mean that the DATA is
actually split.

That seems to be the bottom line of all this musing: it won't work
right unless the DATA is split per-recipient. So you need to take
this up on the Dovecot list (which I see you have done.)
--
http://rob0.nodns4.us/ -- system administration and consulting
Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Timo Sirainen
2012-04-09 04:09:28 UTC
Permalink
Post by /dev/rob0
Post by Daniel L. Miller
- is there a particular reason why these headers are not already
an option via lmtp (aside from nobody asking for or seeing the
need previously). Is there an architectural or conceptual reason
why these headers should not be added via an lmtp connection?
Second, it really seems more appropriate to do this in the LMTP
daemon. It gets the envelope recipient for each mail and can do this
(in theory) much better and more accurately than Postfix can. We
don't know what the LMTP daemon is going to do with that mail ...
perhaps it will be rewritten and forwarded on somehow.
There's a problem with aliases that LMTP server can't solve. Lets say I have two aliases:

***@domain -> ***@domain
***@domain -> ***@domain

The LMTP server sees RCPT TO:<***@domain> for mails that arrive to both of them. If the sender sent the mail to only one of them, the original address can at least in theory be read from the Received: lines. If the sender sent the mail to both of these aliases, the LMTP server won't see the originating address anywhere at all.
Wietse Venema
2012-04-09 13:25:06 UTC
Permalink
Post by Timo Sirainen
There's a problem with aliases that LMTP server can't solve. Lets
to both of them. If the sender sent the mail to only one of them,
the original address can at least in theory be read from the
Received: lines. If the sender sent the mail to both of these
aliases, the LMTP server won't see the originating address anywhere
at all.
I wonder if careful use of the DSN extension would help. With DSN,
the SMTP/LMTP client sends the original recipient with:

RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...

That's either the address that the Postfix SMTP server received
with RCPT TO + ORCPT, or Postfix's best guess (which could be a
bare username in the case of command-line submission, something
that should be fixed).

One problem is that DSN hands off the responsibility to notify the
sender about successful delivery. I don't know if that is desirable.

Wietse
Timo Sirainen
2012-04-10 16:03:55 UTC
Permalink
Post by Wietse Venema
Post by Timo Sirainen
There's a problem with aliases that LMTP server can't solve. Lets
to both of them. If the sender sent the mail to only one of them,
the original address can at least in theory be read from the
Received: lines. If the sender sent the mail to both of these
aliases, the LMTP server won't see the originating address anywhere
at all.
I wonder if careful use of the DSN extension would help. With DSN,
RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...
Does Postfix already send this if LMTP server advertises DSN?
Post by Wietse Venema
One problem is that DSN hands off the responsibility to notify the
sender about successful delivery. I don't know if that is desirable.
Not necessarily a problem, but I can't seem to find any discussion on how it should interact with Sieve. This is clear:

* NOTIFY=FAILURE + Sieve discard: Don't send DSN

But nothing else is. I'm guessing perhaps:

* NOTIFY=SUCCESS + Sieve discard: Send a success DSN
* NOTIFY=FAILURE + Sieve reject: Send a failure MDN, but no DSN
* NOTIFY=SUCCESS + Sieve reject: Send a failure MDN, but no DSN
* NOTIFY=FAILURE + Sieve ereject: Reject the mail on DATA reply, no DSN
* NOTIFY=SUCCESS + Sieve ereject: Reject the mail on DATA reply, no DSN

RET=FULL also seems like it could only be abused..
Wietse Venema
2012-04-10 16:28:28 UTC
Permalink
Post by Timo Sirainen
Post by Wietse Venema
I wonder if careful use of the DSN extension would help. With DSN,
RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...
Does Postfix already send this if LMTP server advertises DSN?
Yes :-) It's the same code for both SMTP and LMTP.
Post by Timo Sirainen
Post by Wietse Venema
One problem is that DSN hands off the responsibility to notify the
sender about successful delivery. I don't know if that is desirable.
Not necessarily a problem, but I can't seem to find any discussion
* NOTIFY=FAILURE + Sieve discard: Don't send DSN
* NOTIFY=SUCCESS + Sieve discard: Send a success DSN
* NOTIFY=FAILURE + Sieve reject: Send a failure MDN, but no DSN
* NOTIFY=SUCCESS + Sieve reject: Send a failure MDN, but no DSN
* NOTIFY=FAILURE + Sieve ereject: Reject the mail on DATA reply, no DSN
* NOTIFY=SUCCESS + Sieve ereject: Reject the mail on DATA reply, no DSN
I think that reject/tempfail are easy - just report 5xx or 4xx at
end-of-data time, and the MTA will take care of notification.

With SUCCESS, the LMTP server would be responsible for sending the
notification. If the LMTP server already has the ability to "forward"
mail, and if that code path can be reused, then sending the "success"
notification might not cost a lot of extra code.
Post by Timo Sirainen
RET=FULL also seems like it could only be abused..
Sure. Postfix switches to RET=HDRS when the bounce message size
exceeds some threshold.

Wietse
Timo Sirainen
2012-04-10 16:37:26 UTC
Permalink
Post by Wietse Venema
Post by Timo Sirainen
Post by Wietse Venema
I wonder if careful use of the DSN extension would help. With DSN,
RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...
Does Postfix already send this if LMTP server advertises DSN?
Yes :-) It's the same code for both SMTP and LMTP.
OK. I guess I'll look into adding DSN support then.
Post by Wietse Venema
Post by Timo Sirainen
Post by Wietse Venema
One problem is that DSN hands off the responsibility to notify the
sender about successful delivery. I don't know if that is desirable.
Not necessarily a problem, but I can't seem to find any discussion
* NOTIFY=FAILURE + Sieve discard: Don't send DSN
* NOTIFY=SUCCESS + Sieve discard: Send a success DSN
* NOTIFY=FAILURE + Sieve reject: Send a failure MDN, but no DSN
* NOTIFY=SUCCESS + Sieve reject: Send a failure MDN, but no DSN
* NOTIFY=FAILURE + Sieve ereject: Reject the mail on DATA reply, no DSN
* NOTIFY=SUCCESS + Sieve ereject: Reject the mail on DATA reply, no DSN
I think that reject/tempfail are easy - just report 5xx or 4xx at
end-of-data time, and the MTA will take care of notification.
"ereject" can do that, but "reject" is specified to always send MDN. How the MDN is supposed to interact with DSN is a bit unclear. I was thinking that discard and reject could maybe be treated as if the delivery was successful and that the discard/reject was more of a post-delivery "client" operation. Anyway, I guess I'll go and ask in Sieve mailing list what others have done.
Post by Wietse Venema
With SUCCESS, the LMTP server would be responsible for sending the
notification. If the LMTP server already has the ability to "forward"
mail, and if that code path can be reused, then sending the "success"
notification might not cost a lot of extra code.
Yes, there's already existing code for sending DSN, so should be pretty easy to add.
Wietse Venema
2012-04-10 17:25:24 UTC
Permalink
Post by Timo Sirainen
Post by Wietse Venema
Post by Timo Sirainen
Post by Wietse Venema
I wonder if careful use of the DSN extension would help. With DSN,
RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...
Does Postfix already send this if LMTP server advertises DSN?
Yes :-) It's the same code for both SMTP and LMTP.
OK. I guess I'll look into adding DSN support then.
RFC 6009 covers some aspects of Sieve/DSN (making DSN attributes
accessible in Sieve, not how to send DSN notification).

Wietse
Viktor Dukhovni
2012-04-11 01:42:49 UTC
Permalink
Post by Wietse Venema
Post by Timo Sirainen
Post by Wietse Venema
RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...
Does Postfix already send this if LMTP server advertises DSN?
Yes :-) It's the same code for both SMTP and LMTP.
Since in most cases the LMTP server is not a queueing MTA, I would
recommend a delivery agent option in Postfix that suppresses DSN
NOTIFY=... transmission to the LMTP server. Still send ORCPT, but
handle (any final) DSN in Postfix.

DSN I think only makes sense with LMTP when the LMTP server is a
content filter that wants to be able to selectively NAK recipients
after content. With LMTP used to deliver to a mail store, I don't
see the point of delegating DSN notices from the MTA to the LDA.
--
Viktor.
Wietse Venema
2012-04-11 01:48:38 UTC
Permalink
Post by Viktor Dukhovni
Post by Wietse Venema
Post by Timo Sirainen
Post by Wietse Venema
RCPT TO:<final-rcpt> ORCPT=rfc822;orig-rcpt ...
Does Postfix already send this if LMTP server advertises DSN?
Yes :-) It's the same code for both SMTP and LMTP.
Since in most cases the LMTP server is not a queueing MTA, I would
recommend a delivery agent option in Postfix that suppresses DSN
NOTIFY=... transmission to the LMTP server. Still send ORCPT, but
handle (any final) DSN in Postfix.
DSN I think only makes sense with LMTP when the LMTP server is a
content filter that wants to be able to selectively NAK recipients
after content. With LMTP used to deliver to a mail store, I don't
see the point of delegating DSN notices from the MTA to the LDA.
Sure. The LMTP defaults are historically targeted at content filters
("lmtp_assume_final = no"). I have no idea what portion of the
installed base would be affected if we were to make a compatibility
breaking change.

Wietse
Viktor Dukhovni
2012-04-11 04:04:41 UTC
Permalink
Post by Wietse Venema
Post by Viktor Dukhovni
Since in most cases the LMTP server is not a queueing MTA, I would
recommend a delivery agent option in Postfix that suppresses DSN
NOTIFY=... transmission to the LMTP server. Still send ORCPT, but
handle (any final) DSN in Postfix.
DSN I think only makes sense with LMTP when the LMTP server is a
content filter that wants to be able to selectively NAK recipients
after content. With LMTP used to deliver to a mail store, I don't
see the point of delegating DSN notices from the MTA to the LDA.
Sure. The LMTP defaults are historically targeted at content filters
("lmtp_assume_final = no"). I have no idea what portion of the
installed base would be affected if we were to make a compatibility
breaking change.
I was thinking of an off-by-default feature, which would I guess
require a new switch. It may be cleaner to incompatibly change
lmtp_assume_final, but I am not an expert on IMAP mail stores and
their LMTP front-ends. There are basically three products to
consider:

- Cyrus IMAP
- Dovecot IMAP
- Zimbra (if it implements LMTP in front of the store).

There is not much diversity in the high-end mailstore product
catalogue, and low-end products don't do LMTP.

Perhaps folks with expertise in Cyrus, Dovecot, Zimbra (and one
or two LMTP-capabale mail stores I might have missed) can comment
on what if any DSN support they offer, and specifically whether
they expect to send failure and/or success DSNs.

I always thought that a final LMTP server only responds 2XX to an
MTA when the message is safely in the mailbox, and so there is
possible use for DSNs after than point, and so all the work should
be done by the MTA.
--
Viktor.
Wietse Venema
2012-04-11 13:13:17 UTC
Permalink
Post by Viktor Dukhovni
I always thought that a final LMTP server only responds 2XX to an
MTA when the message is safely in the mailbox, [...]
I looked up some Sieve RFCs and found that the language implements
accept-then-bounce behavior:

- "reject" (RFC 3028) implements accept-then-bounce. It replies
with 2XX after sending a message disposition notification which
may contain non-ASCII text.

- "ereject" (RFC 5429) introduces reject-instead-of-accept behavior.
It either replies with 5XX plus ASCII-only text, or it replies
with 2XX and sends a message disposition notification. This RFC
also updates some details of the "reject" feature.

This means that the Postfix "lmtp_assume_final" feature is broken.
This feature (which has effect only when delivering to an LMTP
server that doesn't announce DSN support) assumes that a 2XX reply
implies final successful delivery.

http://www.iana.org/assignments/sieve-extensions/sieve-extensions.xml
has a list of Sieve extensions.

Wietse

Loading...