Subscriptions and sieve filters lost after upgrade to Cyrus 3.0

[Martin Svec]

Hello,

after upgrade from Cyrus 2.5.7 to 3.0.6, we noticed that some users lost their IMAP subscriptions
and all sieve filters. After some investigation, we've found that this happens if the user's name
contains dot(s). For sieve directories and .sub files, version 2.5.x uses names with dots replaced
with "^". But 3.0.x series expects the names with dots.

Note that we've altnamespace and unixhierarchysep turned _on_ in both versions, so no namespace
transformations should be needed.

Names in 2.5.x:
/var/imap/sieve/domain/s/somedomain.com/n/name^with^dots/defaultbc
/var/imap/domain/s/somedomain.com/user/n/name^with^dots.sub

Names in 3.0.x:
/var/imap/sieve/domain/s/somedomain.com/n/name.with.dots/defaultbc
/var/imap/domain/s/somedomain.com/user/n/name.with.dots.sub

Surprisingly, other metadata files like .lock files and quota files still use the format with "^"
characters.

I've found no information regarding this incompatible change in 3.0 upgrade documentation. Is this
an expected behavior or a bug?

Best regards

Martin Svec

[ellie timoney]

Hi,

Fixed Issue #2303: lmtpd(8) now looks for sieve scripts in the same directories as the rest of cyrus (including timsieved(8)).
Please note that if you had manually installed sieve scripts to the directories lmtpd used to look in, these will need to be moved.

It sounds like your setup was taking the bug into account (using the '^' paths that lmtpd expected), and since lmtpd has been fixed to use dots, you're now seeing the reverse of the problem.

You should be able to rename the name^with^dots paths to name.with.dots and things will work normally for you.

This behaviour will probably change again in a future major release, but when it does so it will do so correctly, and with upgrade path documentation! The ^ notation is what _should_ be used, eventually, but the implementation that slipped into 3.0 was incomplete and incorrect, and the fix was to back it out and restore the older behaviour.

Sorry about the inconvenience,

ellie

Hello,
after upgrade from Cyrus 2.5.7 to 3.0.6, we noticed that some users lost
their IMAP subscriptions
and all sieve filters. After some investigation, we've found that this
happens if the user's name
contains dot(s). For sieve directories and .sub files, version 2.5.x
uses names with dots replaced
with "^". But 3.0.x series expects the names with dots.
Note that we've altnamespace and unixhierarchysep turned _on_ in both
versions, so no namespace
transformations should be needed.
/var/imap/sieve/domain/s/somedomain.com/n/name^with^dots/defaultbc
/var/imap/domain/s/somedomain.com/user/n/name^with^dots.sub
/var/imap/sieve/domain/s/somedomain.com/n/name.with.dots/defaultbc
/var/imap/domain/s/somedomain.com/user/n/name.with.dots.sub
Surprisingly, other metadata files like .lock files and quota files
still use the format with "^"
characters.
I've found no information regarding this incompatible change in 3.0
upgrade documentation. Is this
an expected behavior or a bug?
Best regards
Martin Svec
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[Martin Svec]

Hi Ellie,

Post by ellie timoney
Hi,

Fixed Issue #2303: lmtpd(8) now looks for sieve scripts in the same directories as the rest of cyrus (including timsieved(8)).
Please note that if you had manually installed sieve scripts to the directories lmtpd used to look in, these will need to be moved.

It sounds like your setup was taking the bug into account (using the '^' paths that lmtpd expected), and since lmtpd has been fixed to use dots, you're now seeing the reverse of the problem.

I don't think #2303 is related to our problem. Issue #2303 was a sieve path inconsistency between
LMTP and the rest of Cyrus between 3.0.0-beta2 and 3.0.6. We've upgraded directly from 2.5.7 to
3.0.6. Also, in our case subscription file names are affected too.

Post by ellie timoney
You should be able to rename the name^with^dots paths to name.with.dots and things will work normally for you.

Of course, we already renamed them using a simple script. However, I'm curious if this breakage
between 2.5 and 3.0 series is well known and documented. Let me repeat again: all subscription and
sieve paths in previous Cyrus versions had dots translated to "^" for years. Since 3.0 Cyrus uses
"." instead of "^" and so the upgrade leads to missing subscriptions and sieve scripts. This is a
significant incompatible change that should be documented in 3.0 Upgrade Guide. I'm also afraid if
there're no more similar "silent" changes that we didn't find yet.

Post by ellie timoney
This behaviour will probably change again in a future major release, but when it does so it will do so correctly, and with upgrade path documentation! The ^ notation is what _should_ be used, eventually, but the implementation that slipped into 3.0 was incomplete and incorrect, and the fix was to back it out and restore the older behaviour.

If "^" notation should be the right one, why 3.0 series changed "^" notation to "." notation in
subscription and sieve paths (and only in these ones)? Shouldn't it be consider a bug in 3.0 series
and fixed it back to old "^" notation everywhere?

Thank you
Martin

Post by ellie timoney
Sorry about the inconvenience,
ellie

Hello,
after upgrade from Cyrus 2.5.7 to 3.0.6, we noticed that some users lost
their IMAP subscriptions
and all sieve filters. After some investigation, we've found that this
happens if the user's name
contains dot(s). For sieve directories and .sub files, version 2.5.x
uses names with dots replaced
with "^". But 3.0.x series expects the names with dots.
Note that we've altnamespace and unixhierarchysep turned _on_ in both
versions, so no namespace
transformations should be needed.
/var/imap/sieve/domain/s/somedomain.com/n/name^with^dots/defaultbc
/var/imap/domain/s/somedomain.com/user/n/name^with^dots.sub
/var/imap/sieve/domain/s/somedomain.com/n/name.with.dots/defaultbc
/var/imap/domain/s/somedomain.com/user/n/name.with.dots.sub
Surprisingly, other metadata files like .lock files and quota files
still use the format with "^"
characters.
I've found no information regarding this incompatible change in 3.0
upgrade documentation. Is this
an expected behavior or a bug?
Best regards
Martin Svec
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[Michael Menge]

Hi,

Post by Martin Svec
Hi Ellie,

Hi,
This was an lmtpd bug in earlier versions of 3.0. It was fixed in

Fixed Issue #2303: lmtpd(8) now looks for sieve scripts in the
same directories as the rest of cyrus (including timsieved(8)).
Please note that if you had manually installed sieve scripts to
the directories lmtpd used to look in, these will need to be moved.

It sounds like your setup was taking the bug into account (using
the '^' paths that lmtpd expected), and since lmtpd has been fixed
to use dots, you're now seeing the reverse of the problem.

I don't think #2303 is related to our problem. Issue #2303 was a
sieve path inconsistency between
LMTP and the rest of Cyrus between 3.0.0-beta2 and 3.0.6. We've
upgraded directly from 2.5.7 to
3.0.6. Also, in our case subscription file names are affected too.

You should be able to rename the name^with^dots paths to
name.with.dots and things will work normally for you.

Of course, we already renamed them using a simple script. However,
I'm curious if this breakage
between 2.5 and 3.0 series is well known and documented. Let me
repeat again: all subscription and
sieve paths in previous Cyrus versions had dots translated to "^"
for years. Since 3.0 Cyrus uses
"." instead of "^" and so the upgrade leads to missing subscriptions
and sieve scripts. This is a
significant incompatible change that should be documented in 3.0
Upgrade Guide. I'm also afraid if
there're no more similar "silent" changes that we didn't find yet.

Just to make sure, you have seen

Post by Martin Svec
defaults have changed in imapd.conf(5). Implications are outlined in
the Note in User Namespace Mode and Switching the Alternative
Namespace. Please also see “Sieve Scripts,” below.
unixhierarchysep: on
altnamespace: on

https://www.cyrusimap.org/3.0/imap/download/upgrade.html#copy-config-files-and-update



--------------------------------------------------------------------------------
M.Menge Tel.: [phone]
Universität Tübingen Fax.: [phone]
Zentrum für Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailm

[Martin Svec]

Hi,

Post by Michael Menge
Hi,

Post by Martin Svec
Hi Ellie,

Hi,
This was an lmtpd bug in earlier versions of 3.0.  It was fixed in 3.0.6, and this was noted in

Fixed Issue #2303: lmtpd(8) now looks for sieve scripts in the same directories as the rest of
cyrus (including timsieved(8)).
Please note that if you had manually installed sieve scripts to the directories lmtpd used to
look in, these will need to be moved.

It sounds like your setup was taking the bug into account (using the '^' paths that lmtpd
expected), and since lmtpd has been fixed to use dots, you're now seeing the reverse of the
problem.

I don't think #2303 is related to our problem. Issue #2303 was a sieve path inconsistency between
LMTP and the rest of Cyrus between 3.0.0-beta2 and 3.0.6. We've upgraded directly from 2.5.7 to
3.0.6. Also, in our case subscription file names are affected too.

You should be able to rename the name^with^dots paths to name.with.dots and things will work
normally for you.

Of course, we already renamed them using a simple script. However, I'm curious if this breakage
between 2.5 and 3.0 series is well known and documented. Let me repeat again: all subscription and
sieve paths in previous Cyrus versions had dots translated to "^" for years. Since 3.0 Cyrus uses
"." instead of "^" and so the upgrade leads to missing subscriptions and sieve scripts. This is a
significant incompatible change that should be documented in 3.0 Upgrade Guide. I'm also afraid if
there're no more similar "silent" changes that we didn't find yet.

Just to make sure, you have seen

Post by Martin Svec
Important config options: unixhierarchysep: and altnamespace: defaults have changed in
imapd.conf(5). Implications are outlined in the Note in User Namespace Mode and Switching the
Alternative Namespace. Please also see “Sieve Scripts,” below.
   unixhierarchysep: on
   altnamespace: on

https://www.cyrusimap.org/3.0/imap/download/upgrade.html#copy-config-files-and-update

Yes, I read it. We had both unixhierarchysep and altnamespace explicitly turned _on_ in 2.5.x. So,
there should be no implications during upgrade to 3.0 for us.

Martin


----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andr