Mailbox inaccessible after setaclmailbox with invalid identifier

[Sven Schwedas]

After running `setaclmailbox some/mailbox "group:name with spaces" all`,
all attempts to access the mailbox in any way either result in an
"Invalid identifier" error message, or can't even find the mailbox.

deleteaclmailbox with exactly the same parameters as set doesn't help
(same "invalid identifier" error); cyrreconstruct doesn't even find the
mailbox. Deliveries are deferred as well.

Any ideas how I can recover the mailbox without rolling back the whole
server to an older backup?

(Cyrus 2.5.10 on Debian stretch, if that matters.)

[Pedro silva]

Hello,

I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.

I have cyrus 2.4.17

Does any one know how or if this is possible in cyrus (or sieve)?

I wasn't able to find any documentation about using external programs
with sieve.

Thanks,

Pedro Silva

[Sven Schwedas]

Post by Pedro silva
Hello,
I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.
I have cyrus 2.4.17
Does any one know how or if this is possible in cyrus (or sieve)?

If you want to (read only) process data already received to train your
spam filter, just use the individual mail files in cyrus' spool.

If you want to act on mails being received and possibly delete/redirect
them, you normally want hook into your MTA, not cyrus/sieve.

[Pedro silva]

Hello Sven,

Thanks for your answer.

Actually I would like to set up a system were a user would move his
messages to a spam folder, and by doing so, the message would be sent to
the spam learning system.

Ive seen something like this in other distributions (eg.:
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just wondering
if the same is possible with cyrus or cyrus sieve.

Best regards,

Pedro Silva

Post by Sven Schwedas

Post by Pedro silva
Hello,
I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.
I have cyrus 2.4.17
Does any one know how or if this is possible in cyrus (or sieve)?

If you want to (read only) process data already received to train your
spam filter, just use the individual mail files in cyrus' spool.
If you want to act on mails being received and possibly delete/redirect
them, you normally want hook into your MTA, not cyrus/sieve.
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[David Lang]

several years ago I used a system that subscribed to the various folders and
watched for messages to appear. Unfortunantly, that software was abandoned and I
haven't gone looking for a replacement in the last several years.

but it did work very well, at very little load on the system (fastmail was a
little concerned at seeing many connections from me, but once they learned that
they were mostly idle watching for new message notifications, they relaxed a
bit)

David Lang

On Tue, 29 May

Date: Tue, 29 May 2018 10:16:37 +0100
Subject: Re: Cyrus (Sieve) and external programs
Hello Sven,
Thanks for your answer.
Actually I would like to set up a system were a user would move his
messages to a spam folder, and by doing so, the message would be sent to
the spam learning system.
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just wondering
if the same is possible with cyrus or cyrus sieve.
Best regards,
Pedro Silva

Post by Sven Schwedas

Post by Pedro silva
Hello,
I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.
I have cyrus 2.4.17
Does any one know how or if this is possible in cyrus (or sieve)?

If you want to (read only) process data already received to train your
spam filter, just use the individual mail files in cyrus' spool.
If you want to act on mails being received and possibly delete/redirect
them, you normally want hook into your MTA, not cyrus/sieve.
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[Pedro silva]

Hei Adam,

Thanks.

This is a great starting point.

I'll try to get it to work with my setup.

Again, thanks.

Pedro Silva

I wrote a script for this purpose in Python 3, for use with rspamd. I
only recently got it working, so it may be buggy, and it also may need
to be modified for your spam system (and your python location), but I
think it should at least provide a good start for you. Note that
using "#!/usr/bin/env python3" at the top does not work. Perhaps
someone can translate this into Perl and include it in Cyrus contrib.
event_notifier: external
notify_external: /path/to/autolearn
# Just make sure \Junk isn't included here
event_exclude_specialuse: \Drafts \Sent
On Tue, 29 May 2018 16:35:53 -0700 (PDT)
several years ago I used a system that subscribed to the various
folders and watched for messages to appear. Unfortunantly, that
software was abandoned and I haven't gone looking for a replacement
in the last several years.
but it did work very well, at very little load on the system
(fastmail was a little concerned at seeing many connections from me,
but once they learned that they were mostly idle watching for new
message notifications, they relaxed a bit)
David Lang
On Tue, 29 May
Date: Tue, 29 May 2018 10:16:37 +0100
Info-cyrus
Subject: Re: Cyrus (Sieve) and external programs
Hello Sven,
Thanks for your answer.
Actually I would like to set up a system were a user would move his
messages to a spam folder, and by doing so, the message would be
sent to the spam learning system.
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just
wondering if the same is possible with cyrus or cyrus sieve.
Best regards,
Pedro Silva
Hello,
I'm trying to set up a spam learning system, and I would like to
pipe email places in (for example) the SPAM folder to an external
program.
I have cyrus 2.4.17
Does any one know how or if this is possible in cyrus (or
sieve)?
If you want to (read only) process data already received to train
your spam filter, just use the individual mail files in cyrus'
spool.
If you want to act on mails being received and possibly
delete/redirect them, you normally want hook into your MTA, not
cyrus/sieve.
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[Pedro silva]

Niels,

sa-learn would be the way to go, but my spamassassin is not on the same
server as my mailboxes. I tried solution with a common mailbox and
something like fetchmail on my spamassassin machine, but could not get
it to work as I intended.

You are right about learning based on user experience, but as it is a
score based learning I would like to try to see how it goes. Maybe I'll
come to regret it but well...

Best regards,

Pedro Silva

Post by Pedro silva
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just
wondering if the same is possible with cyrus or cyrus sieve.

Just for reading mail files into i.e. spamassassin efficently periodically (in a
small to "large" amount of mailboxes) you could do things (by i.e. cron)
- for a "few hundreds" mailboxes (as long as the shell did not complain a to
spam-learn /var/spool/imap/user/*/SpamLearn/*.
for i.e. a cyrus imap Folder called "SpamLearn" (i.e. auto-created in every
mailbox). You possibly have to adapt the whole path to your config/setup.
spam-learn /var/spool/imap/user/$user/Spam/*.
In a 2nd step you may delete the Spam folder content by a little imap script
to avoid re-reading yet processed mails. "spam-learn" is a script / program
which reads emails direcftly fom files.
seems a bit ugly, but works very efficient on filesystem level. It sepends from
the filesystem layer how powerful this could be for you.
- If you want to avoid cron and process "instantly" you may use the notifier
principle as described from Pedro to start such a script for a single mailbox
if /at least one) mail is placed in that Folder..
- It should be possible to to use inotify or similiar filesystem notification
features to start such a script too.
I'm personally do not like "spam training" by users as this often leads to
bad / not valuable results, as many typical end users (by my experience)
misunderstand what "spam" (technically) is (and/or change their mind about).
I.e. they use "forwarded" "spam" fom any external mail sevice/pop, regular
list mail fom a company (which all is a valid mail). Such things leads to
false positives or (if less weighted) just not useful processing overhead.
But this may depend fom the practical application (i do not use Spam marking
("Spam Folder") - just bouncing - as checking Spam Folders costs the end user
additional work time if they rely on email....
good luck.
best regards,
niels.
--
---
Niels Dettenbach
Syndicat IT & Internet
http://www.syndicat.com
PGP: https://syndicat.com/pub_key.asc
---
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[Adam Tauno Williams]

Post by Pedro silva
sa-learn would be the way to go, but my spamassassin is not on the
same server as my mailboxes. I tried solution with a common mailbox
and something like fetchmail on my spamassassin machine, but could
not get it to work as I intended.
You are right about learning based on user experience, but as it is a
score based learning I would like to try to see how it goes. Maybe
I'll come to regret it but well...

The fetchmail method works.

In the spamd home directory create a .fetchmailrc file like:

poll aleph.wmmi.net protocol IMAP:
user {user} with password {secret}

Where {user} is a system account with access to the user.{user}.SPAM
folders.

Then you want to run:
fetchmail --verbose --keep --all --norewrite  --folder
'user.{user}.SPAM' --mda '/usr/bin/sa-learn --spam' AS THE "spamd"
USER for each user.

It works for smallish systems. Easily enough adapted to learn from
one centralized folder provided your mail client has an easy way to get
SPAM reported messages into that folder.

[Pedro silva]

Hello Adam,

I've tried with fetchmail before, but coudn't get it to work, you
explanation is a big help with that method, I'll try it.

Thanks.

Pedro Silva

Post by Adam Tauno Williams

Post by Pedro silva
sa-learn would be the way to go, but my spamassassin is not on the
same server as my mailboxes. I tried solution with a common mailbox
and something like fetchmail on my spamassassin machine, but could
not get it to work as I intended.
You are right about learning based on user experience, but as it is a
score based learning I would like to try to see how it goes. Maybe
I'll come to regret it but well...

The fetchmail method works.
user {user} with password {secret}
Where {user} is a system account with access to the user.{user}.SPAM
folders.
fetchmail --verbose --keep --all --norewrite --folder
'user.{user}.SPAM' --mda '/usr/bin/sa-learn --spam' AS THE "spamd"
USER for each user.
It works for smallish systems. Easily enough adapted to learn from
one centralized folder provided your mail client has an easy way to get
SPAM reported messages into that folder.
--
Meetings Coordinator, Michigan Association of Railroad Passengers
537 Shirley St NE Grand Rapids, MI 49503-1754 Phone: [phone]
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[David Lang]

FYI, the software I mentioned below is popfile, which seems to have stopped
major development in 2011, but someone is still fixing it to work with later
releases

http://getpopfile.org/

Even if you don't use this, you can look at how it works to build a replacement.

David Lang

Post by David Lang
several years ago I used a system that subscribed to the various folders and
watched for messages to appear. Unfortunantly, that software was abandoned
and I haven't gone looking for a replacement in the last several years.
but it did work very well, at very little load on the system (fastmail was a
little concerned at seeing many connections from me, but once they learned
that they were mostly idle watching for new message notifications, they
relaxed a bit)
David Lang

Date: Tue, 29 May 2018 10:16:37 +0100
Info-cyrus
Subject: Re: Cyrus (Sieve) and external programs
Hello Sven,
Thanks for your answer.
Actually I would like to set up a system were a user would move his
messages to a spam folder, and by doing so, the message would be sent to
the spam learning system.
https://wiki.dovecot.org/HowTo/AntispamWithSieve) I was just wondering
if the same is possible with cyrus or cyrus sieve.
Best regards,
Pedro Silva

Post by Sven Schwedas

Post by Pedro silva
Hello,
I'm trying to set up a spam learning system, and I would like to pipe
email places in (for example) the SPAM folder to an external program.
I have cyrus 2.4.17
Does any one know how or if this is possible in cyrus (or sieve)?

If you want to (read only) process data already received to train your
spam filter, just use the individual mail files in cyrus' spool.
If you want to act on mails being received and possibly delete/redirect
them, you normally want hook into your MTA, not cyrus/sieve.
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

[Bron Gondwana]

If you use cyr_dbtool you can edit the offending line in mailboxes db.

Clearly there's a bug here too, we should be sanitising group names! Or
at least storing them in a reversible format

Post by Sven Schwedas
After running `setaclmailbox some/mailbox "group:name with
spaces" all`,> all attempts to access the mailbox in any way either result in an
"Invalid identifier" error message, or can't even find the mailbox.
deleteaclmailbox with exactly the same parameters as set doesn't help> (same "invalid identifier" error); cyrreconstruct doesn't even
find the> mailbox. Deliveries are deferred as well.
Any ideas how I can recover the mailbox without rolling back the whole> server to an older backup?
(Cyrus 2.5.10 on Debian stretch, if that matters.)
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
* signature.asc 1k (application/pgp-signature)

--
Bron Gondwana, CEO, FastMail Pty Ltd
***@fastmailteam.com

[Bron Gondwana]

https://github.com/cyrusimap/cyrus-imapd/issues/2378 to track the issue.
Sorry about the short comment earlier on my phone as I woke up - meant
to save it as a draft and add some detail!
# /usr/cyrus/bin/cyr_dbtool
Usage: /usr/cyrus/bin/cyr_dbtool [-C altconfig] <db file> <db backend> <action> [<key>] [<value>]Usable Backends: flat, skiplist, quotalegacy, twoskip

Actions:
* show [<prefix>]
* get <key>
* set <key> <value>
* delete <key>
* dump - internal format dump
* consistent - check consistency
* repack - repack/checkpoint the DB (if supported)
* damage - start a commit then die during
* batch - read from stdin and execute commands
You may omit key or key/value and specify one per line on stdin
keys are terminated by tab or newline, values are terminated by newline
=====

<db file> will need to be your mailboxes.db (e.g. /var/imap/mailboxes.db
on many systems). You'll have to run it as the cyrus user. You can use
"show" to dump the whole file and grep for the individual mailbox, or
"get" on the mailbox key directly to get the one line. Then "set" to
set it with a value with that group name removed.
You'll probably want to do another setacl on that mailbox with something
safe afterwards, just so the cyrus.header file gets cleaned up as well.
It contains a value which is a cache of the value in mailboxes.db.
Cheers,

Bron.

If you use cyr_dbtool you can edit the offending line in mailboxes db.>
Clearly there's a bug here too, we should be sanitising group names!
Or at least storing them in a reversible format>

Post by Sven Schwedas
After running `setaclmailbox some/mailbox "group:name with
spaces" all`,>> all attempts to access the mailbox in any way either result in an
"Invalid identifier" error message, or can't even find the mailbox.
deleteaclmailbox with exactly the same parameters as set doesn't help>> (same "invalid identifier" error); cyrreconstruct doesn't even
find the>> mailbox. Deliveries are deferred as well.
Any ideas how I can recover the mailbox without rolling back
the whole>> server to an older backup?
(Cyrus 2.5.10 on Debian stretch, if that matters.)
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
* signature.asc 1k (application/pgp-signature)>

--
Bron Gondwana, CEO, FastMail Pty Ltd
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
Bron Gondwana, CEO, FastMail Pty Ltd
***@fastmailteam.com

[ellie timoney]

Hi Sven,

Are you able to provide a sample mailboxes.db entry (using cyr_dbtool as
Bron describes below) for a mailbox which is affected by this?
I'm trying to determine if the cause is bad data getting into the
database, or good data that's being misread for some reason. An example
would help a lot!
Cheers,

ellie

Post by Bron Gondwana
https://github.com/cyrusimap/cyrus-imapd/issues/2378 to track
the issue.>
Sorry about the short comment earlier on my phone as I woke up - meant
to save it as a draft and add some detail!>
# /usr/cyrus/bin/cyr_dbtool
Usage: /usr/cyrus/bin/cyr_dbtool [-C altconfig] <db file> <db backend>
<action> [<key>] [<value>]> Usable Backends: flat, skiplist, quotalegacy, twoskip
* show [<prefix>]
* get <key>
* set <key> <value>
* delete <key>
* dump - internal format dump
* consistent - check consistency
* repack - repack/checkpoint the DB (if supported)
* damage - start a commit then die during
* batch - read from stdin and execute commands
You may omit key or key/value and specify one per line on stdin
keys are terminated by tab or newline, values are terminated by
newline>
=====
<db file> will need to be your mailboxes.db (e.g.
/var/imap/mailboxes.db on many systems). You'll have to run it as the
cyrus user. You can use "show" to dump the whole file and grep for
the individual mailbox, or "get" on the mailbox key directly to get
the one line. Then "set" to set it with a value with that group name
removed.>
You'll probably want to do another setacl on that mailbox with
something safe afterwards, just so the cyrus.header file gets cleaned
up as well. It contains a value which is a cache of the value in
mailboxes.db.>
Cheers,
Bron.

Post by Bron Gondwana
If you use cyr_dbtool you can edit the offending line in
mailboxes db.>>
Clearly there's a bug here too, we should be sanitising group names!
Or at least storing them in a reversible format>>

Post by Sven Schwedas
After running `setaclmailbox some/mailbox "group:name with
spaces" all`,>>> all attempts to access the mailbox in any way either result in an
"Invalid identifier" error message, or can't even find the mailbox.>>>
deleteaclmailbox with exactly the same parameters as set
doesn't help>>> (same "invalid identifier" error); cyrreconstruct doesn't even
find the>>> mailbox. Deliveries are deferred as well.
Any ideas how I can recover the mailbox without rolling back
the whole>>> server to an older backup?
(Cyrus 2.5.10 on Debian stretch, if that matters.)
--
Mit freundlichen GrÌßen, / Best Regards,
Sven Schwedas, Systemadministrator
TAO Digital | Teil der TAO Beratungs- & Management GmbH
Lendplatz 45 | FN 213999f/Klagenfurt, FB-Gericht Villach
A8020 Graz | https://www.tao-digital.at
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
* signature.asc 1k (application/pgp-signature)>>

--
Bron Gondwana, CEO, FastMail Pty Ltd
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus

--
Bron Gondwana, CEO, FastMail Pty Ltd
----
Cyrus Home Page: http://www.cyrusimap.org/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus