Discussion:
Strange bug in cyrus (missing folders with reconstruct !)
Denis BUCHER
2009-12-17 11:01:27 UTC
Permalink
Dear all,

When doing /usr/sbin/cyrreconstruct -rf user.psmith I have a very very
strange behavior !

Most of folders are successfully reconstructed except one or two that
are ignored ???

Example for a folder called "CLIENT". This folder only contains
subfolders, and no messages, could this be the cause of the bug ?

It seems to be the same problem as described in :
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-May/029171.html
But they offer no "real" solution...

Thanks a lot for any help (the users are complaining and I'm stuck at
this problem)

Denis
Patrick Boutilier
2009-12-17 11:09:07 UTC
Permalink
Post by Denis BUCHER
Dear all,
When doing /usr/sbin/cyrreconstruct -rf user.psmith I have a very very
strange behavior !
Most of folders are successfully reconstructed except one or two that
are ignored ???
Example for a folder called "CLIENT". This folder only contains
subfolders, and no messages, could this be the cause of the bug ?
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-May/029171.html
But they offer no "real" solution...
Thanks a lot for any help (the users are complaining and I'm stuck at
this problem)
What version of Cyrus?

Why is the reconstruct being run? Was the mailbox corrupted or are you
trying to inject messages like the person in the above link?
Post by Denis BUCHER
Denis
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Denis BUCHER
2009-12-17 11:45:17 UTC
Permalink
Dear Patrick,
Post by Patrick Boutilier
Post by Denis BUCHER
When doing /usr/sbin/cyrreconstruct -rf user.psmith I have a very very
strange behavior !
Most of folders are successfully reconstructed except one or two that
are ignored ???
Example for a folder called "CLIENT". This folder only contains
subfolders, and no messages, could this be the cause of the bug ?
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-May/029171.html
But they offer no "real" solution...
Thanks a lot for any help (the users are complaining and I'm stuck at
this problem)
What version of Cyrus?
name : Cyrus IMAPD
version : v2.2.13-Debian-2.2.13-14+lenny3 2006/12/19 19:32:59
vendor : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os : Linux
os-version : 2.6.26-2-amd64
environment: Built w/Cyrus SASL 2.1.22
Running w/Cyrus SASL 2.1.22
Built w/Sleepycat Software: Berkeley DB 4.2.52: (December
3, 2003)
Running w/Sleepycat Software: Berkeley DB 4.2.52: (December
3, 2003)
Built w/OpenSSL 0.9.8g 19 Oct 2007
Running w/OpenSSL 0.9.8g 19 Oct 2007
CMU Sieve 2.2
TCP Wrappers
NET-SNMP
mmap = shared
lock = fcntl
nonblock = fcntl
idle = poll
Post by Patrick Boutilier
Why is the reconstruct being run? Was the mailbox corrupted or are you
trying to inject messages like the person in the above link?
Good question, the answer is simple, we simply migrated from an old
server to a new one. So in fact it is neither of the two case but
another one :

We had a completely working set of users accounts and folders.
We moved it to the new server.
To make them work we launched the reconstruct.
It works perfectly except for some folders...

Denis
Patrick Boutilier
2009-12-17 12:35:00 UTC
Permalink
Post by Denis BUCHER
Dear Patrick,
Post by Patrick Boutilier
Post by Denis BUCHER
When doing /usr/sbin/cyrreconstruct -rf user.psmith I have a very very
strange behavior !
Most of folders are successfully reconstructed except one or two that
are ignored ???
Example for a folder called "CLIENT". This folder only contains
subfolders, and no messages, could this be the cause of the bug ?
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-May/029171.html
But they offer no "real" solution...
Thanks a lot for any help (the users are complaining and I'm stuck at
this problem)
What version of Cyrus?
name : Cyrus IMAPD
version : v2.2.13-Debian-2.2.13-14+lenny3 2006/12/19 19:32:59
vendor : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os : Linux
os-version : 2.6.26-2-amd64
environment: Built w/Cyrus SASL 2.1.22
Running w/Cyrus SASL 2.1.22
Built w/Sleepycat Software: Berkeley DB 4.2.52: (December
3, 2003)
Running w/Sleepycat Software: Berkeley DB 4.2.52: (December
3, 2003)
Built w/OpenSSL 0.9.8g 19 Oct 2007
Running w/OpenSSL 0.9.8g 19 Oct 2007
CMU Sieve 2.2
TCP Wrappers
NET-SNMP
mmap = shared
lock = fcntl
nonblock = fcntl
idle = poll
Post by Patrick Boutilier
Why is the reconstruct being run? Was the mailbox corrupted or are you
trying to inject messages like the person in the above link?
Good question, the answer is simple, we simply migrated from an old
server to a new one. So in fact it is neither of the two case but
We had a completely working set of users accounts and folders.
We moved it to the new server.
To make them work we launched the reconstruct.
It works perfectly except for some folders...
How did you move the mail from one server to the other?

Does the CLIENT folder have at least a cyrus.index file in it?
Post by Denis BUCHER
Denis
Denis BUCHER
2009-12-17 13:28:20 UTC
Permalink
Hello Patrick,
Post by Patrick Boutilier
Post by Denis BUCHER
Post by Patrick Boutilier
Post by Denis BUCHER
When doing /usr/sbin/cyrreconstruct -rf user.psmith I have a very very
strange behavior !
Most of folders are successfully reconstructed except one or two that
are ignored ???
Example for a folder called "CLIENT". This folder only contains
subfolders, and no messages, could this be the cause of the bug ?
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-May/029171.html
But they offer no "real" solution...
Thanks a lot for any help (the users are complaining and I'm stuck at
this problem)
What version of Cyrus?
name : Cyrus IMAPD
version : v2.2.13-Debian-2.2.13-14+lenny3 2006/12/19 19:32:59
vendor : Project Cyrus
support-url: http://asg.web.cmu.edu/cyrus
os : Linux
os-version : 2.6.26-2-amd64
environment: Built w/Cyrus SASL 2.1.22
Running w/Cyrus SASL 2.1.22
Built w/Sleepycat Software: Berkeley DB 4.2.52: (December
3, 2003)
Running w/Sleepycat Software: Berkeley DB 4.2.52: (December
3, 2003)
Built w/OpenSSL 0.9.8g 19 Oct 2007
Running w/OpenSSL 0.9.8g 19 Oct 2007
CMU Sieve 2.2
TCP Wrappers
NET-SNMP
mmap = shared
lock = fcntl
nonblock = fcntl
idle = poll
Post by Patrick Boutilier
Why is the reconstruct being run? Was the mailbox corrupted or are you
trying to inject messages like the person in the above link?
Good question, the answer is simple, we simply migrated from an old
server to a new one. So in fact it is neither of the two case but
We had a completely working set of users accounts and folders.
We moved it to the new server.
To make them work we launched the reconstruct.
It works perfectly except for some folders...
How did you move the mail from one server to the other?
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...

Denis
Michael Menge
2009-12-17 13:35:32 UTC
Permalink
Hi,
Post by Denis BUCHER
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
In cyrus it is possible to create subfolders of notexsisting folders.
You may whant to use allowallsubscribe: 1 in imapd.conf


Allow subscription to nonexistent mailboxes. This option is typi-
cally used on backend servers in a Murder so that users can sub-
scribe to mailboxes that don't reside on their "home" server.
This option can also be used as a workaround for IMAP clients
which don't play well with nonexistent or unselectable mailboxes
(eg. Microsoft Outlook).


Regards

Michael Menge


--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
UniversitÀt TÌbingen Fax.: (49) 7071/29-5912
Zentrum fÃŒr Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
WÀchterstraße 76
72074 TÃŒbingen
Brian Awood
2009-12-17 14:07:30 UTC
Permalink
Post by Denis BUCHER
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Normally reconstruct doesn't recurse through directories that aren't
mailboxes unless you use the "-p partition" option. If you don't have a
partition defined, try "-p default".

-Brian
Denis BUCHER
2009-12-17 14:14:28 UTC
Permalink
Hello Brian,
Post by Brian Awood
Post by Denis BUCHER
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Normally reconstruct doesn't recurse through directories that aren't
mailboxes unless you use the "-p partition" option. If you don't have a
partition defined, try "-p default".
This looks intersting, but this is what I get :

$ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS

user.psmith.CLIENTS does not appear to be a mailbox (no
/var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header).

Denis
Brian Awood
2009-12-17 17:31:31 UTC
Permalink
Post by Denis BUCHER
Hello Brian,
Post by Brian Awood
Post by Denis BUCHER
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Normally reconstruct doesn't recurse through directories that aren't
mailboxes unless you use the "-p partition" option. If you don't
have a partition defined, try "-p default".
$ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS
user.psmith.CLIENTS does not appear to be a mailbox (no
/var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header).
Sorry about that, my recollection of -p functionality was incorrect. It
will add mailboxes whose parent directory isn't a mailbox, but you still
have to specify the path to the mailbox. So for example if you
know /var/spool/cyrus/mail/c/user/psmith/CLIENTS/submailboxname is valid,
then;
/usr/sbin/cyrreconstruct -p default user/psmith/CLIENTS/submailboxname
should add "submailboxname" even if "CLIENTS" is just a directory without
cyrus metadata. The "-rf" functionality is slightly different because it
needs a valid mailbox to start from.
If you have a lot of these cases, you might be able to automate this by
running "ctl_mboxlist -v" and cutting out the mailboxes that are listed
as being on the filesystem but not in the database.

-Brian
Denis BUCHER
2009-12-17 23:38:47 UTC
Permalink
Hello Brian,
Post by Brian Awood
Post by Denis BUCHER
Post by Brian Awood
Post by Denis BUCHER
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Normally reconstruct doesn't recurse through directories that aren't
mailboxes unless you use the "-p partition" option. If you don't
have a partition defined, try "-p default".
$ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS
user.psmith.CLIENTS does not appear to be a mailbox (no
/var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header).
Sorry about that, my recollection of -p functionality was incorrect. It
will add mailboxes whose parent directory isn't a mailbox, but you still
have to specify the path to the mailbox. So for example if you
know /var/spool/cyrus/mail/c/user/psmith/CLIENTS/submailboxname is valid,
then;
/usr/sbin/cyrreconstruct -p default user/psmith/CLIENTS/submailboxname
should add "submailboxname" even if "CLIENTS" is just a directory without
cyrus metadata.
Correct ! It worked (I tried with a folder)
Post by Brian Awood
The "-rf" functionality is slightly different because it
needs a valid mailbox to start from.
If you have a lot of these cases, you might be able to automate this by
running "ctl_mboxlist -v" and cutting out the mailboxes that are listed
as being on the filesystem but not in the database.
Do you mean something like /usr/sbin/ctl_mboxlist -d -x -p default ?
I didn't find how to list folder present in system but not it database...

Denis
Brian Awood
2009-12-18 02:48:06 UTC
Permalink
Post by Denis BUCHER
Hello Brian,
Correct ! It worked (I tried with a folder)
Post by Brian Awood
The "-rf" functionality is slightly different because it
needs a valid mailbox to start from.
If you have a lot of these cases, you might be able to automate this
by running "ctl_mboxlist -v" and cutting out the mailboxes that are
listed as being on the filesystem but not in the database.
Do you mean something like /usr/sbin/ctl_mboxlist -d -x -p default ?
I didn't find how to list folder present in system but not it
database...
Denis
No, at least in cyrus 2.3 just "/usr/sbin/ctl_mboxlist -v" will scan the
entire filesystem and look for mailboxes that are not in the mailbox
database or vis-versa.
eg.
# /usr/sbin/ctl_mboxlist -v
'user.ddjdjdj.498' has a directory '/var/spool/imap/L/user/ddjdjdj/498'
but no DB entry

depending on how many mailboxes you have, it may take quite a while to
run. But you should be able to take that output and use your favorite
text processing utility to take the first field and pass it to
reconstruct.

-Brian

Patrick Boutilier
2009-12-17 16:32:36 UTC
Permalink
On 12/17/2009 09:28 AM, Denis BUCHER wrote:
<snip>
Post by Denis BUCHER
Post by Patrick Boutilier
How did you move the mail from one server to the other?
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Is it even possible to create a folder (even an empty one) that doesn't
have the cyrus.* files in it ?

I am pretty sure that if you just put an empty cyrus.header file in
CLIENT then your reconstruct command will see the the folder when run
with -f
Post by Denis BUCHER
Denis
Denis BUCHER
2009-12-17 23:30:17 UTC
Permalink
Post by Patrick Boutilier
<snip>
Post by Denis BUCHER
Post by Patrick Boutilier
How did you move the mail from one server to the other?
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Is it even possible to create a folder (even an empty one) that doesn't
have the cyrus.* files in it ?
Yes it must be, because I really took the files "as is" from the old
server...
Post by Patrick Boutilier
I am pretty sure that if you just put an empty cyrus.header file in
CLIENT then your reconstruct command will see the the folder when run
with -f
Good idea, but :

$ touch /var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header
$ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS
user.psmith.CLIENTS: Mailbox has an invalid format

If only ;-))

Denis
Patrick Boutilier
2009-12-17 23:34:26 UTC
Permalink
Post by Denis BUCHER
Post by Patrick Boutilier
<snip>
Post by Denis BUCHER
Post by Patrick Boutilier
How did you move the mail from one server to the other?
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Is it even possible to create a folder (even an empty one) that doesn't
have the cyrus.* files in it ?
Yes it must be, because I really took the files "as is" from the old
server...
Post by Patrick Boutilier
I am pretty sure that if you just put an empty cyrus.header file in
CLIENT then your reconstruct command will see the the folder when run
with -f
$ touch /var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header
$ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS
user.psmith.CLIENTS: Mailbox has an invalid format
If only ;-))
Copy a real cyrus.header file in perhaps? I know there is a way to trick
reconstruct. :-)
Post by Denis BUCHER
Denis
Bron Gondwana
2009-12-18 00:08:58 UTC
Permalink
Post by Patrick Boutilier
Copy a real cyrus.header file in perhaps? I know there is a way to trick
reconstruct. :-)
The downside here is that you'll wind up with multiple folders with the same uniqueid,
which is going to cause REALLY CONFUSING seen state handling.

Bron.
--
Bron Gondwana
***@fastmail.fm
Denis BUCHER
2009-12-18 00:55:42 UTC
Permalink
Post by Patrick Boutilier
Post by Denis BUCHER
Post by Patrick Boutilier
<snip>
Post by Denis BUCHER
Post by Patrick Boutilier
How did you move the mail from one server to the other?
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Is it even possible to create a folder (even an empty one) that doesn't
have the cyrus.* files in it ?
Yes it must be, because I really took the files "as is" from the old
server...
Post by Patrick Boutilier
I am pretty sure that if you just put an empty cyrus.header file in
CLIENT then your reconstruct command will see the the folder when run
with -f
$ touch /var/spool/cyrus/mail/c/user/psmith/CLIENTS/cyrus.header
$ /usr/sbin/cyrreconstruct -p default -rf user.psmith.CLIENTS
user.psmith.CLIENTS: Mailbox has an invalid format
If only ;-))
Copy a real cyrus.header file in perhaps? I know there is a way to trick
reconstruct. :-)
Yes, my last idea ressort was to copy an email to the folder that
contains only subfolders...

Let's try :
cp /var/spool/cyrus/mail/t/user/test2007/277.
/var/spool/cyrus/mail/p/user/psmith/CLIENTS/1.

Doesn't work... Strange...

Denis
Patrick Boutilier
2009-12-17 23:50:39 UTC
Permalink
Post by Patrick Boutilier
<snip>
Post by Denis BUCHER
Post by Patrick Boutilier
How did you move the mail from one server to the other?
I think I copied with scp.
Post by Patrick Boutilier
Does the CLIENT folder have at least a cyrus.index file in it?
No, the CLIENT folder has only subfolders...
Is it even possible to create a folder (even an empty one) that doesn't
have the cyrus.* files in it ?
It is possible if you create a folder called "Layer1/Layer2" for
instance. I always used to create Layer1 then create Layer2 in Layer1.
Never knew you could create it as a single folder.
Post by Patrick Boutilier
I am pretty sure that if you just put an empty cyrus.header file in
CLIENT then your reconstruct command will see the the folder when run
with -f
Post by Denis BUCHER
Denis
----
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Bron Gondwana
2009-12-18 00:12:03 UTC
Permalink
Post by Denis BUCHER
Dear all,
When doing /usr/sbin/cyrreconstruct -rf user.psmith I have a very very
strange behavior !
Most of folders are successfully reconstructed except one or two that
are ignored ???
Example for a folder called "CLIENT". This folder only contains
subfolders, and no messages, could this be the cause of the bug ?
http://lists.andrew.cmu.edu/pipermail/info-cyrus/2008-May/029171.html
But they offer no "real" solution...
Thanks a lot for any help (the users are complaining and I'm stuck at
this problem)
Sorry I didn't look into this earlier. Do you still have the original filesystem that you
copied these folders from? Look in there for the original cyrus.header and cyrus.index
files, and copy them into place - then reconstruct. That way you'll get the flags for
the messages that are there, and possibly even the seen state.

Copying a cyrus.header from somewhere else really isn't a good idea due to the way
cyrus does unique mailbox IDs and then uses them for seen state handling.

Regards,

Bron.
--
Bron Gondwana
***@fastmail.fm
Loading...