IMAP not seeing old mail present on filesystem

Post by Chris Pepper
More importantly, I don't know how to make the old messages
accessible to my users via IMAP (I can give them the files, but that's
quite awkward). chk_cyrus agrees with IMAP clients about message counts
(very low). I have tried reconstruct with various combinations of
"-rfx", and "quota -f", but not found any way to make it show the old
messages.
Any suggestions?

You probably need to run cyrreconstruct on each user mailbox.

Chris Pepper

2010-10-04 11:27:07 UTC

Post by Patrick Goetz

You probably need to run cyrreconstruct on each user mailbox.

On my system it's /usr/lib/cyrus-imapd/reconstruct, and I have. No joy,
alas.

Chris

Simon Matter

2010-10-04 12:36:52 UTC

Post by Chris Pepper
All,
I have run a small mail service based on Cyrus IMAP for a few
years. The (CentOS 5) server I've used for the past couple years failed
last week. I brought up a new CentOS 5 system on a new Linux server,
installed cyrus-imapd-2.3.7-7.el5_4.3, mounted the old /var disk
(actually one of 2 mdadm submirrors), and copied /var/spool/imap over to
the new /var FS. It's running CentOS' cyrus-imapd-2.3.7-7.el5_4.3 RPM.

Did you also recover /var/lib/imap from the old server and make sure the
configs are the same?

Simon

Post by Chris Pepper
Unfortunately, for many users (all but me?), mail clients (at least
Apple's Mail.app and SquirrelMail) don't show any messages from before
the migration in INBOX. When I grope around in /var/spool/imap/user, I
see the old message (with high numbers) and the new messages (starting a
new sequence from 1).
I know there are last-message counters in the cyrus.* files, so
would resetting those have caused IMAP to start storing new messages
from 1. and vanish the old ones? I don't know how those could have been
cleared for at least 6 users simultaneously. Presumably it happened for
all, and I just fixed it for myself somehow.
More importantly, I don't know how to make the old messages
accessible to my users via IMAP (I can give them the files, but that's
quite awkward). chk_cyrus agrees with IMAP clients about message counts
(very low). I have tried reconstruct with various combinations of
"-rfx", and "quota -f", but not found any way to make it show the old
messages.
Any suggestions?
Thanks,
Chris
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

Chris Pepper

2010-10-04 13:06:41 UTC

Simon,

I did recover /var/lib/imap (although a bit later, FWIW) and
/etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter
& adjusted prefork numbers -- the rest all matched.

Is there other configuration I should check? Both systems are 64-bit
CentOS 5, so the db4 installations should be fully compatible.

Thanks,

Chris

Did you also recover /var/lib/imap from the old server and make sure the
configs are the same?
Simon

Simon Matter

2010-10-04 13:17:13 UTC

Post by Chris Pepper
Simon,
I did recover /var/lib/imap (although a bit later, FWIW) and

I think "a bit later" is your problem. But I think reconstruct should fix
this.
BTW, did you check the subscription status of mailboxes? Maybe the folders
are there but not subscribed, and that's why some users can't see them?

Post by Chris Pepper
/etc/imapd.conf. I just now compared /etc/cyrus.conf and added squatter
& adjusted prefork numbers -- the rest all matched.
Is there other configuration I should check? Both systems are 64-bit
CentOS 5, so the db4 installations should be fully compatible.

If both the old and the new system are basically the same I don't think
there is anything missing.

Simon

Post by Chris Pepper
Thanks,
Chris

Did you also recover /var/lib/imap from the old server and make sure the
configs are the same?
Simon

Chris Pepper

2010-10-04 13:37:55 UTC

Post by Chris Pepper
Simon,
I did recover /var/lib/imap (although a bit later, FWIW) and

Simon,

No, users see the folders, just not old messages. For most (all?)
INBOXes but my own, new messages started arriving as 1. and continued
from there. Users can see the new mail, but not the old. This makes me
think it's not an internal permissions problem, because they see the
mailboxes and (some) mail in them. All file permissions I checked appear
correct

"reconstruct -rfx" doesn't help. Is there anything else to try?

Thanks,

Chris

If both the old and the new system are basically the same I don't think
there is anything missing.
Simon

Post by Chris Pepper
Thanks,
Chris

Did you also recover /var/lib/imap from the old server and make sure the
configs are the same?
Simon

Bron Gondwana

2010-10-04 13:47:09 UTC

A script to find each file and IMAP append it? Just thinking outside the box here!

Post by Chris Pepper
Simon,
I did recover /var/lib/imap (although a bit later, FWIW) and

Simon,
No, users see the folders, just not old messages. For most (all?)
INBOXes but my own, new messages started arriving as 1. and continued
from there. Users can see the new mail, but not the old. This makes me
think it's not an internal permissions problem, because they see the
mailboxes and (some) mail in them. All file permissions I checked appear
correct
"reconstruct -rfx" doesn't help. Is there anything else to try?
Thanks,
Chris

If both the old and the new system are basically the same I don't think
there is anything missing.
Simon

Post by Chris Pepper
Thanks,
Chris

Did you also recover /var/lib/imap from the old server and make sure the
configs are the same?
Simon

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Patrick Goetz

2010-10-04 14:47:35 UTC

Post by Bron Gondwana
A script to find each file and IMAP append it? Just thinking outside the box here!

How would you do the IMAP append? Using a Perl::IMAP function?

This isn't necessarily a concern for this list, but a few days ago I
upgraded a site from cyrus 2.1.16 to cyrus 2.3.16 by using imapsync to
transfer mail from the old server to the new one. This worked great
(i.e. all metadata seems to have been preserved), however the old server
was still collecting mail for the few hours it took to imapsync all the
users (note that this is very slows and probably not appropriate for
larger sites). My plan was to swap the servers and then do a final
imapsync from the old IMAP server to the new one. For some reason, for
some users the additional set of new messages is copied to the new
server, and others they aren't. I couldn't figure out what the
difference is, and don't want to spend too much more time on this
because of the small number of messages in play, so I need some way to
transfer a couple of dozen messages "by hand". In general, though
imapsync seems to be a great way to "clean up" cyrus folders when
switching servers. Had I known I would have sync problem later, I would
have just taken the old server off line before syncing the messages.

Before someone suggests that I should have just copied /var/lib/cyrus
and the messages over to the new server, I didn't trust this because I
couldn't get anyone to confirm the database files I had on the old
2.1.16 server -- the filenames were mostly not the same as this set:

annotation_db
duplicate_db
mboxlist_db
ptscache_db
quota_db
seenstate_db
tlscache_db

and the old server experienced a couple of devastating crashes which
required me to cyrreconstruct all the user.user mailboxes a couple of
times. It seemed pretty clear that the db files on the old server were
a mess, and upgrading from anywhere from libdb2 to libdb4.3 to libdb4.8
seemed sketchy as well. The old server is such an old debian system
that apt-cache depends doesn't seem to work any more because the package
servers have changed. (Yeah, I could probably find the source packages
somewhere, but then there are a lot of things I could do given infinite
time.)

Patrick Goetz

2010-10-04 14:23:35 UTC

Post by Chris Pepper
No, users see the folders, just not old messages. For most (all?)
INBOXes but my own, new messages started arriving as 1. and continued
from there. Users can see the new mail, but not the old. This makes me
think it's not an internal permissions problem, because they see the
mailboxes and (some) mail in them. All file permissions I checked appear
correct
"reconstruct -rfx" doesn't help. Is there anything else to try?

I wasn't clear about whether the old install was completely gone or
could still be booted. If you can still start cyrus on the old server,
you could try imapsync to transfer mail to the new one.

Chris Pepper

2010-10-04 15:17:59 UTC

Post by Patrick Goetz

I wasn't clear about whether the old install was completely gone or
could still be booted. If you can still start cyrus on the old server,
you could try imapsync to transfer mail to the new one.

Old system is not bootable, unfortunately.

FYI: I have 943 directories & 298,409 mail files, so manually fixing things isn't feasible.

Thanks for all the suggestions!

Chris

--
Chris Pepper: <http://cbio.mskcc.org/>
<http://www.extrapepperoni.com/>

André Schild

2010-10-04 15:26:20 UTC

I wasn't clear about whether the old install was completely gone or

Post by Patrick Goetz
could still be booted. If you can still start cyrus on the old server,
you could try imapsync to transfer mail to the new one.

Old system is not bootable, unfortunately.
FYI: I have 943 directories& 298,409 mail files, so manually fixing things isn't feasible.
Thanks for all the suggestions!

Are the location of the "old" mails correct for the new system ?
Means: When you deliver a new mail, does it appear in the same folder as
the old ones ?

Another thing: Did you make sure the reconstruct runs under the correct
user account ? (Probably cyrus)
and the file owner/groups are correct for the new system ? (Not root)

André

Patrick Goetz

2010-10-04 15:28:05 UTC

Post by Patrick Goetz
I wasn't clear about whether the old install was completely gone or
could still be booted. If you can still start cyrus on the old server,
you could try imapsync to transfer mail to the new one.

Old system is not bootable, unfortunately.
FYI: I have 943 directories & 298,409 mail files, so manually fixing things isn't feasible.

This seems like a long shot, but could you temporarily set up another
machine with the old version of CentOS, copy /var/lib/cyrus and
/var/spool/cyrus to this machine as is from the old server, and then run
imapsync?

The other option is as Bron suggested, using some kind of IMAP function
to append "lost" messages to the spool. I'm pretty that this will
result in all the metadata being lost (i.e. replied to and forwarded
flags, etc.) (And moreover, I don't know to do this, so can't advise.)

Bron Gondwana

2010-10-04 15:36:15 UTC

Old system is not bootable, unfortunately.
FYI: I have 943 directories & 298,409 mail files,
so manually fixing things isn't feasible.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Well, of course not. It there were 5 directories and 20 mail
files I wouldn't consider doing it manually - I'd write a script
to automate it and then sit back and drink coffee. Manually is
how you do the first one to find out how it's done.

And then the second one to make sure the process you settled on
after stack of trial and error is repeatable. Maybe a third one
if you screwed up number 2.

After that, you automate the process you've decided on and let
the rest happen automatically. It always amazes me to see
admins repeating themselves manually over and over for a
frequent task.

I'd love to see the output of your reconstruct command (including
syslog with the logging level turned up) to see why it's not
finding the files. And maybe an 'ls -la' of one of the imap
directories with this issue as well. I'm away on a "team building"
exercise for the next couple of days Oslo time - but I can certainly
look at it afterwards. I've been travelling (from Australia) which
is why I haven't been looking in on this earlier... it's a strange
set of symptoms.

If you have a small folder that you don't mind sharing, a tar.gz of
the entire folder contents (including the metadata files) would be
fantastic, because then I could check the contents of the .index
and .cache files as well - and maybe even try a reconstruct on a
testbed here.

But file permissions are the most interesting - I'm wondering if
reconstruct is unable to read the directory correctly or unable
to read the old files.

Bron.

Chris Pepper

2010-10-04 17:20:22 UTC

Post by Bron Gondwana

Old system is not bootable, unfortunately.
FYI: I have 943 directories& 298,409 mail files,
so manually fixing things isn't feasible.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Well, of course not. It there were 5 directories and 20 mail
files I wouldn't consider doing it manually - I'd write a script
to automate it and then sit back and drink coffee. Manually is
how you do the first one to find out how it's done.
And then the second one to make sure the process you settled on
after stack of trial and error is repeatable. Maybe a third one
if you screwed up number 2.
After that, you automate the process you've decided on and let
the rest happen automatically. It always amazes me to see
admins repeating themselves manually over and over for a
frequent task.
I'd love to see the output of your reconstruct command (including
syslog with the logging level turned up) to see why it's not
finding the files. And maybe an 'ls -la' of one of the imap
directories with this issue as well. I'm away on a "team building"
exercise for the next couple of days Oslo time - but I can certainly
look at it afterwards. I've been travelling (from Australia) which
is why I haven't been looking in on this earlier... it's a strange
set of symptoms.
If you have a small folder that you don't mind sharing, a tar.gz of
the entire folder contents (including the metadata files) would be
fantastic, because then I could check the contents of the .index
and .cache files as well - and maybe even try a reconstruct on a
testbed here.
But file permissions are the most interesting - I'm wondering if
reconstruct is unable to read the directory correctly or unable
to read the old files.

Bron,

Unfortunately I don't know how to write a script to do this.

The old system was CentOS 5/x64, just like the new system. I believe it had the same cyrus-imapd & db4 RPMs, as I had patched a few weeks ago, but I cannot confirm.

I will send you a tarball of this directory directly. I will try "reconstruct -rfx" with debug logging enabled in syslog tonight.

Thanks,

Chris

Post by Bron Gondwana
total 1756
-rw------- 1 cyrus mail 2931 May 20 12:56 158.
-rw------- 1 cyrus mail 2052 Jun 21 08:33 159.
-rw------- 1 cyrus mail 1536 Aug 11 17:15 162.
-rw------- 1 cyrus mail 2221 Aug 13 08:15 163.
-rw------- 1 cyrus mail 413395 Aug 14 18:40 164.
-rw------- 1 cyrus mail 79756 Aug 21 13:57 165.
-rw------- 1 cyrus mail 296759 Aug 22 15:18 166.
-rw------- 1 cyrus mail 156064 Sep 4 09:43 167.
-rw------- 1 cyrus mail 75003 Sep 4 10:06 168.
-rw------- 1 cyrus mail 164035 Sep 6 09:51 169.
-rw------- 1 cyrus mail 156853 Sep 7 12:24 170.
-rw------- 1 cyrus mail 340068 Sep 16 17:44 171.
drwx------ 2 cyrus mail 4096 Oct 1 21:31 Junk
drwx------ 2 cyrus mail 4096 Oct 2 23:40 Drafts
drwx------ 2 cyrus mail 4096 Oct 2 23:40 Sent
drwx------ 2 cyrus mail 4096 Oct 2 23:40 Apple Mail To Do
drwx------ 2 cyrus mail 4096 Oct 2 23:40 Deleted Messages
-rw------- 1 cyrus mail 1148 Oct 2 23:44 1.
drwx------ 2 cyrus mail 4096 Oct 3 00:09 Trash
drwx------ 2 cyrus mail 4096 Oct 3 13:16 Sent Messages
-rw------- 1 cyrus mail 179 Oct 3 13:39 cyrus.header
-rw------- 1 cyrus mail 176 Oct 4 09:04 cyrus.index
-rw------- 1 cyrus mail 640 Oct 4 09:04 cyrus.cache

--
Chris Pepper: <http://cbio.mskcc.org/>
<http://www.extrapepperoni.com/>

Chris Pepper

2010-10-04 21:26:19 UTC

Thanks, all, for the generous help. Bron asked about output from reconstruct, which never provided any. It turns out that I was using the wrong delimiters (., per chk_cyrus output, rather than /), and reconstruct wasn't even trying. It looks like all the missing mail is accessible again.

Thanks again!

Chris

--
Chris Pepper: <http://cbio.mskcc.org/>
<http://www.extrapepperoni.com/>

Bron Gondwana

2010-10-04 21:34:32 UTC

Post by Chris Pepper
Thanks, all, for the generous help. Bron asked about output from reconstruct, which never provided any. It turns out that I was using the wrong delimiters (., per chk_cyrus output, rather than /), and reconstruct wasn't even trying. It looks like all the missing mail is accessible again.

Excellent! Glad to be of help :)

(and yeah, the internal vs external naming of folders is
something that annoys me and I'd like to tidy up)

Bron.

Shuvam Misra

2010-10-05 05:54:32 UTC