transport maps updating..

[brian moore]

Okay, the story is we're migrating a few thousand users from one machine
(running, ick, sendmail) to a brand spanking new postfix box.

The transition so far: set the new box to be MX and use a transport map
to send mail to the old machine except for a few addresses we're taking
over.... Ie, it looks like this:

efn.org slowsite:[clavin.efn.org]
***@efn.org local
***@efn.org local
***@efn.org local

etc. (slowsite mailer since they're using sendmail and it stops
accepting mail periodically so I have followed the 'bottlenecks'
advice on a high concurrency limit but low process limit transport
so that it doesn't get too backlogged.)

Now, here's the catch: when a new user is created (or one is pulled off
the existing server and moved here), I want to do the following steps:

1) set per-user transport to be 'defer'
2) sync the mailboxes
3) set perdition to use the local pop/imap servers
4) set per-user transport to the 'local' mailer

Basically make postfix the buffer during the transition of a user (or
managable size group of users). This should make it all happy and
good...

I have bit of perl that does:

tie %transport, 'DB_File', '/etc/postfix/transport-transition.db', O_CREAT|O_RDWR, 0666, $DB_HASH ;
$transport{$user . '@efn.org' } = 'local';

Which in theory should work. In fact, it does: the transport file is
updated... But postfix doesn't notice...

I'm lazy and didn't want to figure out why it didn't notice, so I forced
it into 'postmap -i' so that if there's some little trick to updating in
a way that postfix picks up, it should happen... but it doesn't. :/

samwise:/usr/local/sbin# ls -l /etc/postfix/transport-transition.db
-rw-r--r-- 1 root root 12288 May 26 11:03 /etc/postfix/transport-transition.db
samwise:/usr/local/sbin# postmap -q ***@efn.org /etc/postfix/transport-transition.db
local

but I see as recently as 11:27 that postfix has tried to send mail to
the slowsite transport.

I know that trivial-rewrite figures out the transport stuff, but it's
invoked at two stages: when the mail is first entered into the system
and it's placed in 'incoming', and again when qmgr figures out what to
do with it...

But...

samwise:/usr/local/sbin# ps aux | grep triv
postfix 21568 0.2 0.0 3448 1640 ? S 11:05 0:04 trivial-rewrite -n rewrite -t unix -u -c
postfix 21728 0.1 0.0 3316 1556 ? S 11:09 0:02 trivial-rewrite -n rewrite -t unix -u -c

so, both of those started after the last update to transport, so they
should both have the same version of the transport-transition.db file
that postmap -q sees....

But then why isn't qmgr figuring out to send these locally?

Is there a better 'trick' to get postfix to first forward all mail (on a
per-user basis), then hold it, then deliver it to a local address that
I could be using?

[Steve Crawford]

Post by brian moore
Okay, the story is we're migrating a few thousand users from one
machine (running, ick, sendmail) to a brand spanking new postfix
box.
The transition so far: set the new box to be MX and use a transport
map to send mail to the old machine except for a few addresses

...<snip>...

Post by brian moore
but I see as recently as 11:27 that postfix has tried to send mail
to the slowsite transport.

postfix reload??

Nearly every reference I see in the documentation related to transport
contains the phrase, "Execute the command postfix reload to make the
changes effective."

Cheers,
Steve

[brian moore]

Post by Steve Crawford

Post by brian moore
Okay, the story is we're migrating a few thousand users from one
machine (running, ick, sendmail) to a brand spanking new postfix
box.
The transition so far: set the new box to be MX and use a transport
map to send mail to the old machine except for a few addresses

...<snip>...

Post by brian moore
but I see as recently as 11:27 that postfix has tried to send mail
to the slowsite transport.

postfix reload??

ick, yeah, that seems to be the case, :/

Post by Steve Crawford
Nearly every reference I see in the documentation related to transport
contains the phrase, "Execute the command postfix reload to make the
changes effective."

Hrrm.. may have to install postfix-mysql for this map for a while.
I dislike depending on Big Monstrous SQL Server stuff for mail, but I
need to be able to update the map in more of a 'realtime' sense.

A mysql map, if I understand right, should allow that sort of real-time
change where I could put a user on defer, shuffle his mail, then put him
on local without the load of a 'postfix reload' every few minutes.

(gah, and did i mention sendmail sucks? the remote site is not scaling
well to having postfix spew at it.. :P)

[V***@MorganStanley.com]

Post by brian moore
tie %transport, 'DB_File', '/etc/postfix/transport-transition.db', O_CREAT|O_RDWR, 0666, $DB_HASH ;
Which in theory should work. In fact, it does: the transport file is
updated... But postfix doesn't notice...

This is not safe, because you are not following the Postfix locking
protocol for Berkeley DB files. You must update the map with
"postmap -ir".

Post by brian moore
I'm lazy and didn't want to figure out why it didn't notice, so I forced
it into 'postmap -i' so that if there's some little trick to updating in
a way that postfix picks up, it should happen... but it doesn't. :/
samwise:/usr/local/sbin# ls -l /etc/postfix/transport-transition.db
-rw-r--r-- 1 root root 12288 May 26 11:03 /etc/postfix/transport-transition.db
local
but I see as recently as 11:27 that postfix has tried to send mail to
the slowsite transport.

With enough mail in the queue, the queue manager may have messages in the
active queue that are resolved to the old transport setting for the
recipient. This is unavoidable. Transport selection happens in the queue
manager as messages are queued for a transport, not when messages are
allocated to a transport (it is impossible to schedule transport
allocation without making the transport selection decision as messages
entre the active queue).

Post by brian moore
I know that trivial-rewrite figures out the transport stuff, but it's
invoked at two stages: when the mail is first entered into the system
and it's placed in 'incoming'

No, the transport is not computed at that time. The lookups occur for
other reasons (recipient validation), but the data is not cached or saved.

Post by brian moore
and again when qmgr figures out what to do with it...

Yes, the transport for a recipient is cached while the message is in the
active queue. You must ensure that all mail for a recipient is drained
from the active queue before you can assume that the old transport setting
is no longer in place.

[V***@MorganStanley.com]

Post by brian moore
Hrrm.. may have to install postfix-mysql for this map for a while.
I dislike depending on Big Monstrous SQL Server stuff for mail, but I
need to be able to update the map in more of a 'realtime' sense.

This won't help.

Post by brian moore
A mysql map, if I understand right, should allow that sort of real-time
change where I could put a user on defer, shuffle his mail, then put him
on local without the load of a 'postfix reload' every few minutes.

The problem is that active messages sit around waiting for a delivery
agent and are immune to transport table changes until they are delivered
or deferred. A new transport value will only be seen after all currently
active messages leave the *active* queue in some fashion. Deferred
messages pick up the new transport value as they re-enter the active
queue.

[Gregory J Smith]

----- Original Message -----

Post by brian moore

Post by Steve Crawford
postfix reload??

ick, yeah, that seems to be the case, :/

I built a very simple script (RH9 Postfix 2.0.16) to rebuild the
hashes and do the reload so I only have to remember one
command when I change anything.

#!/bin/bash
# This simple script just rebuilds the alias tables,
# transport maps, etc. and reloads postfix.
/usr/bin/newaliases
/usr/sbin/postmap /etc/postfix/access
/usr/sbin/postmap /etc/postfix/sender_checks
/usr/sbin/postmap /etc/postfix/recipient_checks
/usr/sbin/postmap /etc/postfix/virtual
/usr/sbin/postmap /etc/postfix/transport
/etc/init.d/postfix reload
logger postfix_reload completed
sleep 1
tail -n 2 /var/log/maillog

The last bit just displays the last two line of the maillog so
you can see if any errors came up (mostly).

----------------------------------------------
Gregory J Smith
<***@smithonline.id.au>
----------------------------------------------

[George Vieira]

=20

Post by Gregory J Smith
I built a very simple script (RH9 Postfix 2.0.16) to rebuild the
hashes and do the reload so I only have to remember one
command when I change anything.
#!/bin/bash
# This simple script just rebuilds the alias tables,
# transport maps, etc. and reloads postfix.
/usr/bin/newaliases
/usr/sbin/postmap /etc/postfix/access
/usr/sbin/postmap /etc/postfix/sender_checks
/usr/sbin/postmap /etc/postfix/recipient_checks
/usr/sbin/postmap /etc/postfix/virtual
/usr/sbin/postmap /etc/postfix/transport
/etc/init.d/postfix reload
logger postfix_reload completed
sleep 1
tail -n 2 /var/log/maillog
=20
The last bit just displays the last two line of the maillog so
you can see if any errors came up (mostly).
=20
----------------------------------------------
Gregory J Smith
----------------------------------------------

How abou this Makefile I found from someone else.. It restarts Postfix
too

#all: client_restrictions.db virtual_alias.db access.db relay_domains.db
transport.db canonical.db relocated.db
# Makefile for /etc/postfix

TARGETS :=3D helo_checks.db client_restrictions.db virtual_alias.db
relay_domains.db access.db canonical.db relocated.db transport.db \
aliases.db .build.mark

all: $(TARGETS)
postfix reload

aliases.db: aliases
postalias aliases

.build.mark: main.cf master.cf header_checks
touch .build.mark

%.db: %
postmap $<

clean:
@rm -f *.db *~

[Robin Lynn Frank]

=2D----BEGIN PGP SIGNED MESSAGE-----
Hash: RIPEMD160

Post by Gregory J Smith
----- Original Message -----

Post by brian moore

Post by Steve Crawford
postfix reload??

ick, yeah, that seems to be the case, :/

I built a very simple script (RH9 Postfix 2.0.16) to rebuild the
hashes and do the reload so I only have to remember one
command when I change anything.
#!/bin/bash
# This simple script just rebuilds the alias tables,
# transport maps, etc. and reloads postfix.
/usr/bin/newaliases
/usr/sbin/postmap /etc/postfix/access
/usr/sbin/postmap /etc/postfix/sender_checks
/usr/sbin/postmap /etc/postfix/recipient_checks
/usr/sbin/postmap /etc/postfix/virtual
/usr/sbin/postmap /etc/postfix/transport
/etc/init.d/postfix reload
logger postfix_reload completed

Department of Redundancy Department. Unless your /etc/init.d/postfix diffe=
rs=20
substantially from mine, you will notice it runs postalias and postmaps =20
virtual and transport.

=2D --=20
BOFH excuse #352:

The cables are not the same length.
=2D----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Sed quis custodiet ipsos custodes?

iD8DBQFAtWl+o0pgX8xyW4YRA6MYAJoCW+V11eQjrRdis6BGS3CKoCnMjgCgzqYe
WO3LLIZzlwq0rOdoD5t7QgM=3D
=3DQ1En
=2D----END PGP SIGNATURE-----

[Craig Sanders]

Post by George Vieira
How abou this Makefile I found from someone else.. It restarts Postfix
too
#all: client_restrictions.db virtual_alias.db access.db relay_domains.db
transport.db canonical.db relocated.db
# Makefile for /etc/postfix
TARGETS := helo_checks.db client_restrictions.db virtual_alias.db
relay_domains.db access.db canonical.db relocated.db transport.db \
aliases.db .build.mark
all: $(TARGETS)
postfix reload
aliases.db: aliases
postalias aliases
.build.mark: main.cf master.cf header_checks
touch .build.mark
%.db: %
postmap $<
@rm -f *.db *~

three comments:

1. use a wildcard *.db - that way any .db file will be rebuilt as long
as it exists, no need to edit Makefile every time you add a new map.

create it by hand with postmap and a source file and from then on it
will be kept up-to-date with make.

2. .db and similar files should be built as an atomic operation (well,
emulated anyway). that means building it with a temporary name and
then renaming it after it is built. mv/rename is an atomic operation,
so the entire process is effectively atomic.

this is important because it means that at no point will postfix be
attempting to read a half-built .db file. it will either be accessing
the old version, or it will be accessing the complete & finished new
version. this is very much a Good Thing.

3. postfix can easily be reloaded or restarted from the Makefile - there
are fairly clear dependancies for when it should be reloaded and when
it should be restarted.



here's a (much-simplified & cruft-free) version of my /etc/postfix/Makefile.

with a well-designed Makefile, maintaining postfix is a matter of editing
whatever config or map files i need to change and then running 'make'. the
Makefile does all the repetitive and routine stuff to "activate" the changes.

---cut here---
#! /usr/bin/make -f

all: *.db /etc/aliases.db .stamp-restart .stamp-reload

# add other alias text files if you use any
/etc/aliases.db: /etc/aliases
newaliases

# postfix should be reloaded if main.cf and certain other config files
# are touched.
.stamp-reload: body_checks header_checks pcre.cf pcre-client.cf main.cf
/etc/init.d/postfix reload
touch .stamp-reload

# postfix should be restarted if master.cf is touched
.stamp-restart: master.cf
/etc/init.d/postfix restart
touch .stamp-restart .stamp-reload

# generic rule for generating a .db map file from the source txt file
# as an atomic operation.
%.db : %
mv $< $<.new
postmap $<.new
mv $<.new $<
mv $<.new.db $<.db

---cut here---

[brian moore]

Post by V***@MorganStanley.com
This is not safe, because you are not following the Postfix locking
protocol for Berkeley DB files. You must update the map with
"postmap -ir".

Well, that code's gone anyway. :P

Post by V***@MorganStanley.com
With enough mail in the queue, the queue manager may have messages in the
active queue that are resolved to the old transport setting for the
recipient. This is unavoidable. Transport selection happens in the queue
manager as messages are queued for a transport, not when messages are
allocated to a transport (it is impossible to schedule transport
allocation without making the transport selection decision as messages
entre the active queue).

Well, the remote site is.. slow. It periodically times out and load
gets too high so sendmail shuts down... so I have LOTS of stuff in the
queue. In the afternoon it can get to around 12k pieces of mail
backlogged to them. Needless to say getting users off that box will be
very very nice. I may even get daring and prioritize users based on how
much mail they get... move the 'high volume' people first so the mail
will get less backed up.

Post by V***@MorganStanley.com
Yes, the transport for a recipient is cached while the message is in the
active queue. You must ensure that all mail for a recipient is drained
from the active queue before you can assume that the old transport setting
is no longer in place.

Is there any decent way to do that?

For the simple case of 'new user added to system', that's not a concern:
I just add them to the transport map with 'local:' as the transport and
there's no worry since they don't have anything queued it's not a big
deal... Their mail will work instantly the way it should.

For the users-on-transition, I have to do the following:
1) set user to defer
2) rsync the two systems (a couple times just to be paranoid :))
3) set user to local, set perdition to local

Now, I will probably be doing this in 'lumps' of users instead of one at
a time... so the rsync time may cover the time in active. But it would
be nice to 'be sure'.

But I did get the mysql stuff in place, and that will at least make
adding new users possible. (Now to finish munging the passwd file and
get RADIUS set to here and I'll be able to create new users. :))

[V***@MorganStanley.com]

Post by brian moore
1) set user to defer
2) rsync the two systems (a couple times just to be paranoid :))
3) set user to local, set perdition to local

This is the wrong design. Set delivery to "defer" on the *target* system
without updating any transport tables, something along the lines of:

.forward:
"|exit 75"

Then copy mailbox to new system.
Then change .forward to send mail to new system.
Then change transport on hub to send directly to new system.

Presto magic, no need to synchronize the queue, mail sent to the wrong
system is forwarded to the right one.

[Rob Foehl]

Post by V***@MorganStanley.com

Post by brian moore
1) set user to defer
2) rsync the two systems (a couple times just to be paranoid :))
3) set user to local, set perdition to local

This is the wrong design. Set delivery to "defer" on the *target* system
"|exit 75"
Then copy mailbox to new system.
Then change .forward to send mail to new system.
Then change transport on hub to send directly to new system.
Presto magic, no need to synchronize the queue, mail sent to the wrong
system is forwarded to the right one.

I don't believe he was rsyncing the queue (at least, I *hope* he's not
rsyncing the queue), rather the mailboxes.. Unless I'm mistaken, as long
as the old system causes the new system to defer inbound mail for the
users in transition, the next time the new system attempts a delivery it
should pick up the newly applied local: transport and deliver properly,
and should require less administrative headache on the new box.

-Rob

[brian moore]

Post by Rob Foehl

Post by V***@MorganStanley.com
Presto magic, no need to synchronize the queue, mail sent to the wrong
system is forwarded to the right one.

I don't believe he was rsyncing the queue (at least, I *hope* he's not
rsyncing the queue), rather the mailboxes.. Unless I'm mistaken, as long
as the old system causes the new system to defer inbound mail for the
users in transition, the next time the new system attempts a delivery it
should pick up the newly applied local: transport and deliver properly,
and should require less administrative headache on the new box.

Nope, not gonna rsync the queue.. the remote system is running sendmail
so that would be a huge undertaking to even think of that.

The 'catch' is that this is all a very live system (users are being
added to the 'new' system) so I need a transport map on the new
system to mark which users are 'here' and which are 'there'...

So I need a way to suspend mail delivery for a given set of users
while syncing, then a way to make their new mailbox the only mailbox.

I do like the "drop a .forward in place on the way out", but
an 'exit 75' won't work, since I really don't want to figure out
how to do something as slick as postfix's per-user transport
on sendmail. I haven't used sendmail in a loong time and don't
want to go back to it.

I'll think about how to get the remote site to suspend/requeue
stuff and have it bounce back to postfix. I have another day before
I have to start moving mail. :/

[Rob Foehl]

On Thu, 27 May 2004, brian moore wrote:

[...]

Post by brian moore
The 'catch' is that this is all a very live system (users are being
added to the 'new' system) so I need a transport map on the new
system to mark which users are 'here' and which are 'there'...

Yes, in the interim you will.. Of course, you can make this part of your
life easier as well by generating a transport map for all the users that
are still on the old system and falling back to 'local' for everybody
else, so your only transport updates are deletions..

Post by brian moore
So I need a way to suspend mail delivery for a given set of users
while syncing, then a way to make their new mailbox the only mailbox.
I do like the "drop a .forward in place on the way out", but
an 'exit 75' won't work, since I really don't want to figure out
how to do something as slick as postfix's per-user transport
on sendmail. I haven't used sendmail in a loong time and don't
want to go back to it.

Victor's suggestion should work fine on the sendmail box; 4xx code will be
returned to the inbound (Postfix) system, and it'll defer the message.
Chances are you'll have removed the non-local transport entry for that
user by then (after syncing their mailbox).

-Rob

[Rob Foehl]

On Thu, 27 May 2004, Rob Foehl wrote:

[...]

Post by Rob Foehl
Chances are you'll have removed the non-local transport entry for that
user by then (after syncing their mailbox).

Ugh, long day. By 'by then', I of course meant by the next delivery
attempt on the Postfix system...

-Rob

[Craig Sanders]

Well, the remote site is.. slow. It periodically times out and load gets too
high so sendmail shuts down... so I have LOTS of stuff in the queue. In the
afternoon it can get to around 12k pieces of mail backlogged to them.
Needless to say getting users off that box will be very very nice. I may
even get daring and prioritize users based on how much mail they get... move
the 'high volume' people first so the mail will get less backed up.

wouldn't it make more sense to just build a replacement for the sendmail box,
but with postfix installed? then, the transition would be:

1. on the relay postfix box, defer all mail for the old sendmail box

2. duplicate ALL of the accounts from the sendmail box onto the new postfix box

3. rsync ALL of the mailboxes to the new postfix box.

4a. if mail is all that the sendmail box does, then decommission it and replace
it with the new postfix box.

4b. if the old sendmail box does other stuff as well, then just change the MX records
and/or transport table entries to the new box. and disable sendmail on the old
box.

5. flush the deferred mail.


alternatively, if you have a compiler and development tools on the sendmail
box:

1. on the relay box, defer all mail for the old sendmail box.
2. on the sendmail box, replace sendmail with postfix.
3. flush the deferred mail.



by doing something like either of these, you could have the entire migration
done in a few hours.



btw, in case you're wondering: yes, i have done similar things several times
before. it's not very difficult. the crucial thing is to plan it out in
advance so you don't run into any surprises while you're in the middle of doing
the work - with any migration, there is a point of no return. you want to make
that point as late in the process as possible, and you really don't want to run
into any unanticipated snags after you've reached that point. plan carefully.

craig