Message Priorities

[Mike Depot]

I'm looking for the ability to send messages of different priorities
to a Postfix server and have messages flagged as higher priority get
queued to go out first. I've scanned the archives a bit and found
other people have asked for this ability, but mostly by wanting
sendmail like support for a Precedence: header. From what I can
tell Wietse seemed to be against the idea due the inefficiency of
having to open and parse each message file in order to determine how
or when it should be queued.

I'm interested in finding a way to get that functionality in another
way. Rather than indicating the various message priorities with a
header in the message, how about running smtpd on a few different
ports, with each port meant to handle messages of a different
priority? The idea being that each port would have its own message
queue directory - and then by Postfix prioritizing the queues from
which the messages are sent from. Granted this is just an idea, but
it would be one way to get client defined message prioritization
without requiring the parsing of the individual message files, and
(I think) with minimal customization of Postfix.

I'd be interested in hearing other people's comments on the idea,
and meanwhile if anyone has setup similar functionality, perhaps in
a different way such as running separate instances of Postfix,
suggestions on how I could set up would be greatly appreciated.

Thanks!
Mike Depot

[Magnus Bäck]

On Tuesday, August 24, 2004 at 01:09 CEST,
Mike Depot <***@cmarket.com> wrote:

[...]

Post by Mike Depot
I'm interested in finding a way to get that functionality in another
way. Rather than indicating the various message priorities with a
header in the message, how about running smtpd on a few different
ports, with each port meant to handle messages of a different
priority?

The delivery is still scheduled by a single qmgr(8) per instance.

Post by Mike Depot
The idea being that each port would have its own message queue
directory - and then by Postfix prioritizing the queues from which the
messages are sent from. Granted this is just an idea, but it would be
one way to get client defined message prioritization without requiring
the parsing of the individual message files, and (I think) with
minimal customization of Postfix.

I'm pretty sure the customization would be non-trivial. If you have this
need, use multiple instances.

What actual problem are you trying to solve? A well-tuned system should
not have any significant delivery delays anyway.

[...]

--=20
Magnus B=E4ck
***@dsek.lth.se

[Wietse Venema]

Post by Mike Depot
I'm looking for the ability to send messages of different priorities
to a Postfix server and have messages flagged as higher priority get
queued to go out first. I've scanned the archives a bit and found

You're solving the wrong problem. Postfix should never have
significant transit delays. If the problem is lack of network
bandwidth, then reduce load or arrange for more capacity.

Postfix has a queue, but that is no reason to treat it as a printer
or a batch job processing system.

Wietse

[Victor Duchovni]

Post by Mike Depot
I'm interested in finding a way to get that functionality in another
way. Rather than indicating the various message priorities with a
header in the message, how about running smtpd on a few different
ports, with each port meant to handle messages of a different
priority? The idea being that each port would have its own message
queue directory - and then by Postfix prioritizing the queues from
which the messages are sent from. Granted this is just an idea, but
it would be one way to get client defined message prioritization
without requiring the parsing of the individual message files, and
(I think) with minimal customization of Postfix.

From message 7 in the thread: http://tinyurl.com/3okre

Yes, there is no mechanism for absolute priority that works when
the system is under duress. The first class passengers don't get
to be the first on the plane, when the highway to the airport is
congested with bumper-to-bumber traffic. They need to get to the
airport before the economy class boarding begins :-)

There is no way to implement absolute high priority queues in a single
Postfix queue manager. You need multiple incoming queues, and reserved
transport process pools for each higher than default priority level.
When enough of this is built, it looks like a very complex version
of multiple Postfix instances albeit with pre-emptive scheduling of
the higher priority queues.

So, the best way to approximate *source* "prioritization" in Postfix is
to in-fact run multiple Postfix instances with separate queues and queue
managers. Each instance will drain its queue in parallel to all other
queues, mail in a high priority queue will not queue behind a backlog of
mail in a lower priority queue. All the queues will have the same overall
priority, but the least busy queues will drain quickly. If you send only
a modest quantity of high priority mail, it will always be delivered
quickly. You could (negative) "nice" the master processes of high priority
Postfix instances to give them higher priority OS scheduling.

Sendmail does not in fact implement message priority in the way that
most users imagine. The first delivery attemt is FIFO just like Postfix,
only without concurrency limits (under load, much to the detriment of both
sender and recipient). The use of priority in Sendmail (<= 8.11) is when
processing deferred mail, and it only "works" because each queue runner
scans the whole backlog and performs deliveries one message at a time.
Often multiple queue runners overlap and some are processing the tail
of the queue while newly spawned runners process the head. Hardly a
strictly priority based scheduling system.

Priority of output is possible in Postfix (more so than in Sendmail),
via dedicated transports, but this only works if the incoming queue
(the highway to the airport) is not congested.

Post by Mike Depot
I'd be interested in hearing other people's comments on the idea,
and meanwhile if anyone has setup similar functionality, perhaps in
a different way such as running separate instances of Postfix,
suggestions on how I could set up would be greatly appreciated.

What problem are you really trying to solve? Does it really make sense in
this day of cheap fast machines to cram priority queueing into big complex
systems instead of dedicating more cheap iron to high priority traffic?

What should happen to low priority mail when there is a steady stream
of high priority mail? What should happen to deferred high priority
mail? ...

If your mail queue is congested solve that! On my systems all mail leaves
as soon as it arrives (or gets deferred because the remote system is
unreachable). There is no appreciable avoidable latency. Priorities are
for resource starved, saturated queues, don't let your queue get starved.

The existing Postfix scheduler is just fine. If bulk mail from lists is
clogging your queue, process bulk mail in a different Postfix instance,
or better-yet a different physical machine.