Conversion Debian Cyrus 2.1 to 2.2, experiences

[Paul van der Vlis]

Hello.

I did an upgrade from Cyrus 2.1 tot 2.2, both packages in Debian Etch.
The documentation is a bit poor, that's why I wrote about my
experiences. Maybe somebody else likes it.

And I would like to hear critics, because I have to upgrade another 8
machines... But I have no problems with the 2 machines I did this way.


First I stopped the services:
/etc/init.d/postfix stop
/etc/init.d/cyrus21 stop

check if it worked:
ps ax|grep cyr; ps ax|grep imapd; ps ax|grep pop3d
kill open processes.

close the firewall for pop3 and IMAP.
close local mailprogramms
close webmail

make backups:
cp -a /var/lib/cyrus /var/lib/cyrus-backup
cp -a /var/spool/sieve /var/spool/sieve-backup
cp -a /usr/lib/cyrus/ /usr/lib/cyrus-backup
cp -a /var/spool/cyrus /var/spool/cyrus-backup
The last one takes long...

remove packages:
apt-get remove cyrus21-common cyrus21-admin cyrus21-clients
libcyrus-imap-perl21
dpkg --get-selections | grep cyrus

backup config-files:
mv /etc/imapd.conf /etc/imapd.conf.backup
mv /etc/cyrus.conf /etc/cyrus.conf.backup

install packages:
apt-get install cyrus-imapd-2.2 cyrus-admin-2.2 cyrus-clients-2.2
libcyrus-imap-perl22 db4.2-util cyrus-pop3d-2.2

choose to overwrite cyrus.conf and imapd.conf (I wonder why this files
are still there).

You will see this and Cyrus will not start.
---------
Database backends have changed!

Comparison between /usr/lib/cyrus/cyrus-db-types.txt and
/usr/lib/cyrus/cyrus-db-types.active shows that database backends for
Cyrus IMAPd have been changed.


This means that those databases for which the database backends changed
might need to be converted manually to the new format, using the
cvt_cyrusdb(8) utility.
Please refer to /usr/share/doc/cyrus-common-2.2/README.Debian.database
for more information. Do not start cyrmaster until you have converted
the databases to the new format.
---------

edit /etc/imapd.conf, in my case I had to do this:
admins: cyrus
sasl_mech_list: PLAIN
sasl_minimum_layer: 0
sasl_pwcheck_method: saslauthd

then convert the databases (on one line):
find /var/lib/cyrus/ -name \*.db -print -exec /usr/bin/db4.2_upgrade {} \;

this was my output:
----------
/var/lib/cyrus/mailboxes.db
db_upgrade: /var/lib/cyrus/mailboxes.db: unrecognized file type
db_upgrade: DB->upgrade: /var/lib/cyrus/mailboxes.db: Invalid argument
/var/lib/cyrus/tls_sessions.db
/var/lib/cyrus/deliver.db
/var/lib/cyrus/db.backup1/mailboxes.db
db_upgrade: /var/lib/cyrus/db.backup1/mailboxes.db: unrecognized file type
db_upgrade: DB->upgrade: /var/lib/cyrus/db.backup1/mailboxes.db: Invalid
argument
/var/lib/cyrus/db.backup2/mailboxes.db
db_upgrade: /var/lib/cyrus/db.backup2/mailboxes.db: unrecognized file type
db_upgrade: DB->upgrade: /var/lib/cyrus/db.backup2/mailboxes.db: Invalid
argument
----------

So "mailboxes.db" did not work, but the other databases did.

Because I use sieve, I also had to convert this to bytecode:
su cyrus
/usr/lib/cyrus/upgrade/masssievec /usr/lib/cyrus/bin/sievec /etc/imapd.conf
exit

this was the output:
--------
you are using /var/spool/sieve as your sieve directory.
processing user ....
processing user ....
processing user ....
(...)
----------

Then I studied the texts in UPGRADE.Debian and README.Debian.database
and "upgrading from previous versions" from Cyrus, and I even wrote a
little script to convert mailboxes.db and seen.db, but in practice every
mailbox says: "Warning: apparently empty database converted.". So in
pratice I did nothing to convert the seen.db and the mailboxes.db (I
used the backup to go-back).

I will add the script on the end of this message, maybe it can be good
to mass-convert databases in another situation.

Then copy the cyrus-db-types.active out of the way (one line):
mv /usr/lib/cyrus/cyrus-db-types.active
/usr/lib/cyrus/cyrus-db-types.active.old

Reconfigure the package:
dpkg-reconfigure cyrus-common-2.2
It says:
-------
Stopping Cyrus IMAPd: .
Creating/updating cyrus user account...
The user `cyrus' is already a member of `sasl'.
Creating cyrus control directories in /var/lib/cyrus...
Creating partition spool /var/spool/cyrus/mail...
Creating partition spool /var/spool/cyrus/news...
Trying to optimize Cyrus partitions, edit /etc/default/cyrus2.2 to
disable...
Starting Cyrus IMAPd: cyrmaster.
---------

After this, Cyrus is running again.

I did a reconstruct (but it takes time, you can also do it when
everything is ready).
su cyrus
cd /var/spool/cyrus
/usr/sbin/cyrreconstruct -r user.*
exit

Then I tested the IMAP, without Postfix. Maybe you will need to open the
firewall for your IP. I did this, because I can go back to the backup
situation now. Once I start Postfix or open IMAP or POP3 for the public,
I cannot go back.

In first instance I found Cyrus 2.2 not so fast as 2.1, but after some
time testing it became better.

After the tests, start postfix, webmail, change your firewall.

and remove backups:
rm -r/var/lib/cyrus-backup
rm -r /var/spool/sieve-backup
rm -r /usr/lib/cyrus-backup
rm -r /var/spool/cyrus-backup
rm /etc/imapd.conf.backup
rm /etc/cyrus.conf.backup

With regards,
Paul van der Vlis.


the script I did not use, at the end. But maybe useable in another
situation:
---------
su cyrus
cd /
mv /var/lib/cyrus/mailboxes.db /var/lib/cyrus/mailboxes.db.old \
/usr/sbin/cvt_cyrusdb /var/lib/cyrus/mailboxes.db.old flat \
/var/lib/cyrus/mailboxes.db skiplist
chown cyrus:mail /var/lib/cyrus/mailboxes.db
rm -f /var/lib/cyrus/mailboxes.db.old

find /var/lib/cyrus/ -name \*.seen -print -exec mv {} {}.old \; \
-exec /usr/sbin/cvt_cyrusdb {}.old flat {} skiplist \; \
-exec chown cyrus:mail {} \;
find /var/lib/cyrus/ -name \*.seen.old -print -exec rm {} \;
--------

[Pascal Gienger]

Post by Paul van der Vlis
find /var/lib/cyrus/ -name \*.db -print -exec /usr/bin/db4.2_upgrade {} \;
db_upgrade: /var/lib/cyrus/mailboxes.db: unrecognized file type
So "mailboxes.db" did not work, but the other databases did.

Just a side note: I am pretty sure your mailboxes.db is a skiplist database
which is AFAIK the default for mailboxes.db in Cyrus IMAP 2.1 and 2.2. No
conversion is necessary.
Do you have any database type declarations in your imapd.conf?

Pascal

[Paul van der Vlis]

Post by Pascal Gienger

Post by Paul van der Vlis
find /var/lib/cyrus/ -name \*.db -print -exec /usr/bin/db4.2_upgrade {} \;
db_upgrade: /var/lib/cyrus/mailboxes.db: unrecognized file type
So "mailboxes.db" did not work, but the other databases did.

Just a side note: I am pretty sure your mailboxes.db is a skiplist
database which is AFAIK the default for mailboxes.db in Cyrus IMAP 2.1
and 2.2. No conversion is necessary.

I think that's correct, but I don't know for sure how to check the type.
The conversed machines are working fine.

The file /usr/lib/cyrus/cyrus-db-types.active of the old systems says:
DBENGINE BerkeleyDB3.2
DUPLICATE db3_nosync
MBOX skiplist
SEEN skiplist
SUBS flat
TLS db3_nosync

The file cyrus-db-types.txt is the same.

The new systems are working fine, and this is the cyrus-db-types.active:
ANNOTATION skiplist
DBENGINE BerkeleyDB4.2
DUPLICATE berkeley-nosync
MBOX skiplist
PTS berkeley
QUOTA quotalegacy
SEEN skiplist
SUBS flat
TLS berkeley-nosync

Post by Pascal Gienger
Do you have any database type declarations in your imapd.conf?

No, I did nothing special, and I did not found anything like that in my
imapd.conf.

Some of the systems did use Cyrus 1.5 before. The conversion was
difficult I can remember. But the file cyrus-db-types.active is the same
as the other old systems.

And I used everywhere Debian-packages.

With regards,
Paul van der Vlis.

[Gabor Gombas]

Post by Paul van der Vlis

Post by Pascal Gienger
Just a side note: I am pretty sure your mailboxes.db is a skiplist
database which is AFAIK the default for mailboxes.db in Cyrus IMAP 2.1
and 2.2. No conversion is necessary.

I think that's correct, but I don't know for sure how to check the type.
The conversed machines are working fine.

# file mailboxes.db
mailboxes.db: Cyrus skiplist DB

Gabor

[Paul van der Vlis]

Post by Gabor Gombas

Post by Paul van der Vlis

Post by Pascal Gienger
Just a side note: I am pretty sure your mailboxes.db is a skiplist
database which is AFAIK the default for mailboxes.db in Cyrus IMAP 2.1
and 2.2. No conversion is necessary.

I think that's correct, but I don't know for sure how to check the type.
The conversed machines are working fine.

# file mailboxes.db
mailboxes.db: Cyrus skiplist DB
Gabor

On an old server:
elo:/var/lib/cyrus# file mailboxes.db
mailboxes.db: Apple QuickTime movie (modified)
???
When I use "strings mailboxes.db" the first line says:
skiplist file
So I think it's a skiplist file.

Thanks for your help!
It still gives an correct answer on some other databases.


Met vriendelijke groet,
Paul van der Vlis.

[Gabor Gombas]

Post by Paul van der Vlis
cp -a /var/lib/cyrus /var/lib/cyrus-backup
cp -a /var/spool/sieve /var/spool/sieve-backup
cp -a /usr/lib/cyrus/ /usr/lib/cyrus-backup
cp -a /var/spool/cyrus /var/spool/cyrus-backup
The last one takes long...

Don't you have regular backups? If you don't, you should better start
doing them... Anyway, you can use rsync to make an initial copy while
the old service is still running and a much quicker update when the old
service is stopped.

Post by Paul van der Vlis
apt-get remove cyrus21-common cyrus21-admin cyrus21-clients
libcyrus-imap-perl21
dpkg --get-selections | grep cyrus
mv /etc/imapd.conf /etc/imapd.conf.backup
mv /etc/cyrus.conf /etc/cyrus.conf.backup

I'd do that _before_ removing the packages...

Post by Paul van der Vlis
apt-get install cyrus-imapd-2.2 cyrus-admin-2.2 cyrus-clients-2.2
libcyrus-imap-perl22 db4.2-util cyrus-pop3d-2.2
choose to overwrite cyrus.conf and imapd.conf (I wonder why this files
are still there).

Because you've used "apt-get remove" instead of "apt-get purge". See the
dpkg manual for the description of the difference between the two
operations. Note: "purge" may also remove the spool directory if you've
choosen to do so, so be careful.

Post by Paul van der Vlis
find /var/lib/cyrus/ -name \*.db -print -exec /usr/bin/db4.2_upgrade {} \;
----------
/var/lib/cyrus/mailboxes.db
db_upgrade: /var/lib/cyrus/mailboxes.db: unrecognized file type
db_upgrade: DB->upgrade: /var/lib/cyrus/mailboxes.db: Invalid argument
/var/lib/cyrus/tls_sessions.db
/var/lib/cyrus/deliver.db
/var/lib/cyrus/db.backup1/mailboxes.db
db_upgrade: /var/lib/cyrus/db.backup1/mailboxes.db: unrecognized file type
db_upgrade: DB->upgrade: /var/lib/cyrus/db.backup1/mailboxes.db: Invalid
argument
/var/lib/cyrus/db.backup2/mailboxes.db
db_upgrade: /var/lib/cyrus/db.backup2/mailboxes.db: unrecognized file type
db_upgrade: DB->upgrade: /var/lib/cyrus/db.backup2/mailboxes.db: Invalid
argument
----------
So "mailboxes.db" did not work, but the other databases did.

This was my recipe for a 2.1 -> 2.3 (from experimental) migration:

cd /var/lib/cyrus/db
db3_recover
cd /var/lib/cyrus
db4.X_upgrade deliver.db
rm tls*db
cd db
db4.X_checkpoint -1

(replace 'X' with the correct BDB version)

Gabor

[Paul van der Vlis]

Post by Gabor Gombas

Post by Paul van der Vlis
cp -a /var/lib/cyrus /var/lib/cyrus-backup
cp -a /var/spool/sieve /var/spool/sieve-backup
cp -a /usr/lib/cyrus/ /usr/lib/cyrus-backup
cp -a /var/spool/cyrus /var/spool/cyrus-backup
The last one takes long...

Don't you have regular backups? If you don't, you should better start
doing them...

I do have regular backups, but if I do something like this I like a way
back to the old situation, without the loss of the mail since the last
backup.

Post by Gabor Gombas
Anyway, you can use rsync to make an initial copy while
the old service is still running and a much quicker update when the old
service is stopped.

Correct, that's better.

Post by Gabor Gombas

Post by Paul van der Vlis
apt-get remove cyrus21-common cyrus21-admin cyrus21-clients
libcyrus-imap-perl21
dpkg --get-selections | grep cyrus
mv /etc/imapd.conf /etc/imapd.conf.backup
mv /etc/cyrus.conf /etc/cyrus.conf.backup

I'd do that _before_ removing the packages...

Without --purge, the configfiles are not removed.
But maybe your way is better.

Post by Gabor Gombas

Post by Paul van der Vlis
apt-get install cyrus-imapd-2.2 cyrus-admin-2.2 cyrus-clients-2.2
libcyrus-imap-perl22 db4.2-util cyrus-pop3d-2.2
choose to overwrite cyrus.conf and imapd.conf (I wonder why this files
are still there).

Because you've used "apt-get remove" instead of "apt-get purge".

I did also a "mv /etc/imapd.conf /etc/imapd.conf.backup" etc.

Post by Gabor Gombas
cd /var/lib/cyrus/db
db3_recover
cd /var/lib/cyrus
db4.X_upgrade deliver.db
rm tls*db
cd db
db4.X_checkpoint -1
(replace 'X' with the correct BDB version)

Thanks for the information!

With regards,
Paul van der Vlis.