Discussion:
master-master replication
Michael Menge
2018-09-12 11:53:00 UTC
Permalink
Hi,
Hello!
I have two servers with
cyrus-imapd cyrus-imapd-2.5.8-13.3.el7.centos.kolab_16.x86_64
One server as master and second as replica.
All worked fine when users login on master server, but when I
temporary move users on replica I found some trouble
Messages synchronisation from replica to master goes fine if
sync_client sees a mismatch on the master, but if user create folder
on replica it isn't sync on master.
Instead of it folder is unsubscribes from the master server and
removed from both server
grep UNSUB maillog
example.com!user.user.foldername
Why is it happend ?
When I tried manual sync from replica to master server, folder was
subscribed. Is it possible that both servers will be master and
replica in same time.
Cyrus does not support master-master replication. Because of
CONDSTORE (https://tools.ietf.org/html/rfc4551) Cyrus is able
to handle messages on a master-master setup, but the information
about folder operations is not tracked and so cyrus is unable to
distinguish is a folder was subscribed on one server, or if a folder
was unsubscribed on the other. Same is true for folder
creation/rename/deletion

The master-master replication is a wanted feature but even the 3.0.x
does not support it. I am not sure about the upcoming 3.1.x or master
branch.


Regards,

Michael



--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://list
Bron Gondwana
2018-09-12 11:56:18 UTC
Permalink
Yes! This is on our roadmap, and I really hope to land it before we release 3.2.

The subscriptions are a particularly tricky part of it, because there's currently no change information in the subscriptions database, but I'll make sure that gets added so we can tell if it's a subscription add or subscription remove!

I'm really looking forward to proper master/master safety too :)

Cheers,

Bron.
Hello!
I have two servers with cyrus-imapd cyrus-imapd-2.5.8-13.3.el7.centos.kolab_16.x86_64
One server as master and second as replica.
All worked fine when users login on master server, but when I temporary move users on replica I found some trouble
Messages synchronisation from replica to master goes fine if sync_client sees a mismatch on the master, but if user create folder on replica it isn't sync on master.
Instead of it folder is unsubscribes from the master server and removed from both server
grep UNSUB maillog
Why is it happend ?
When I tried manual sync from replica to master server, folder was subscribed.
Is it possible that both servers will be master and replica in same time.
Thank you.
--
Evgeniy Kononov
----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
--
Bron Gondwana, CEO, FastMail Pty Ltd
***@fastmailteam.com
Michael Menge
2018-09-13 08:22:12 UTC
Permalink
Hi,

This setup is NOT SUPPORTED and WILL BREAK if the replication process
is triggered
from the wrong server (user is active on both servers, user switched
from one server
to the other while the sync-log file is still processed, after split
brain) and
some mailboxes have been subscribed, renamed created deleted.....

Also there is the risk of a race condition with subscriptions, if a
user subscribes
to multiple folders, the first will trigger a sync from A to B, but as
the folder
is subscribed on B it will trigger a sync from B to A, witch can undo the next
folder subscription.

These are only some cases that came to my mind. There will be more
cases and it
will be hard to debug. So DON'T DO IT!

What we do is, that we have distributed our users between multiple
instances, and each server is running one instance as master and one other
as replic. In case of failure or maintenance we stop the master instance, and
promote the corresponding replic and configure them so that they will sync
them back. If the old master is up to date we switch them back.

We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses
multiple
instances on one server with nginx frontends

Regards,

Michael
Sorry! Previous message was sent by mistake.
For example, I can configure both servers as follows.
Server A.
-----------------
/etc/cyrus.conf
START {
...
syncclient       cmd="sync_client -r"
...
}
SERVICES {
...
syncserver       cmd="sync_server" listen="csync"
...
}
/etc/imapd.conf
...
sync_host: SERVER-B
sync_authname: admin
sync_password: password
sync_log: 1
sync_repeat_interval: 30
sync_timeout: 600
sync_shutdown_file: /var/lib/imap/syncstop And the same on server B.
-----------------
/etc/cyrus.conf
START {
...
syncclient       cmd="sync_client -r"
...
}
SERVICES {
...
syncserver       cmd="sync_server" listen="csync"
...
}
/etc/imapd.conf
...
sync_host: SERVER-A
sync_authname: admin
sync_password: password
sync_log: 1
sync_repeat_interval: 30
sync_timeout: 600
sync_shutdown_file: /var/lib/imap/syncstop
Both server will be as master and as slave in one time.
Will there be any problems with this configuration?
Thank you. --
Evgeniy Kononov
--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen
Michael Menge
2018-09-13 14:26:12 UTC
Permalink
Hi!
Thank you for reply.
Users can connect to only one server at a time. I move the master
server to another hardware and at this time it is necessary for
users to use the mail.
If this is not a secure configuration, then can I just run
"sync_client -A" from the master server, and then switch users to a
replica?
After that, swap the roles of master-replica between the servers? I'm right ?
We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses 
multiple
instances on one server with nginx frontends
Can you give an example of the configuration?
Sure,

first of some background Infos:

We recently switched from Cyrus 2.4.20 on SLES 11 SP4 to Cyrus 3.0.8
on RHEL 7.5 so consult
the man pages for your version.

Our Mailserver are running as 6 KVM VMs (RHEV) with 20 GB Ram, 8 Cores each on
two locations. We have a total of ~44000 accounts, ~457000 Mailboxes,
and 2x6.5 TB Mails

Each server is running 3-4 instances. One frontend, two backend/replic
and on one of the servers the cyrus mupdate master. Each Server on one
location is paired with one server on the other location for replication
so in normal operation one backend on server A replicates to a replic on
server B and the backend on server B replicates to the replica on server A.

Keepalived and ipvs loadbalancer distribute the the load to the
frontend servers.
We use a private subnet for our backend and replic und mupdate instances and a
service ip address for the frontends.

We move the ip address with the role, so that ma01.mail.localhost on server A
replicate to sl01.mail.localhost on server B. But if we need to switch
to the replic
we will start it with ma01.mail.localhost on server B

Keeping the master instance for mailbox on the same IP is important,
because updating the
location for all mailboxes in the mupdate master would take to long.
(the mupdate protocol
knows nothing about replication)


The main trick to run multiple instances on one server is to use
different cyrus.conf
and imapd.conf files for each instance. We use cyrus_INSTANCE.conf and
imapd_INSTANCE.conf
where INSTANCE is replaced by mu for mupdate, fe for the frontend, be
for the first
backend/replic and re of the second backend/replic

The choosing of "be" and "re" was not the best as it is easily
confused with the role
in wich each of these instances can run.

The masterproces is started with "master -C /etc/imapd_INSTANCE.conf
-M /etc/cyrus_INSTANCE.conf -p /var/run/cyrus_instance.pid"
and in the cyrus_INSTANCE.conf you must also use "-C
/etc/imapd_INSTANCE.conf" service, start and event
"cmd" so that the correct conf file is used. For services you also
have to configure "listen="
so that each instance has its own ip to listen on as only one process
can listen on 0.0.0.0 for each port.
In the imapd_INSTANC.conf many directories must be configured.

We generate the conf files from templates. Where TYPE = INSTANCES
Here are the main parts of our templates


========== Cyrus Master ============
# cyrus_@@TYPE@@.conf
# Template MD5SUM: @@MD5SUM@@

START {
@@TYPE@@recover cmd="ctl_cyrusdb -r -C /etc/imapd_@@TYPE@@.conf"
@@TYPE@@mupdatepush cmd="ctl_mboxlist -m -a -C /etc/imapd_@@TYPE@@.conf"
@@TYPE@@idled cmd="idled -C /etc/imapd_@@TYPE@@.conf"
}

SERVICES {
@@TYPE@@imap cmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024
@@TYPE@@imaps cmd="imapd -U 50 -s -C
/etc/imapd_@@TYPE@@.conf" listen="@@HOSTNAME@@:imaps" prefork=1
maxfds=1024
@@TYPE@@pop3 cmd="pop3d -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:pop3" prefork=1 maxfds=1024
@@TYPE@@pop3s cmd="pop3d -s -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:pop3s" prefork=1 maxfds=1024
@@TYPE@@sieve cmd="timsieved -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:sieve" prefork=0 maxfds=1024
@@TYPE@@lmtp cmd="lmtpd -U 5 -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:lmtp" prefork=1 maxfds=1024
@@TYPE@@lmtpunix cmd="lmtpd -U 5 -C /etc/imapd_@@TYPE@@.conf"
listen="/srv/cyrus-@@TYPE@@/socket/lmtp" prefork=1 maxfds=1024
}

EVENTS {
@@TYPE@@checkpoint cmd="ctl_cyrusdb -c -C
/etc/imapd_@@TYPE@@.conf" period=30
@@TYPE@@delprune cmd="cyr_expire -E 3 -X 60 -D 60 -C
/etc/imapd_@@TYPE@@.conf" at=0100
@@TYPE@@tlsprune cmd="tls_prune -C /etc/imapd_@@TYPE@@.conf" at=0430
@@TYPE@@squatter cmd="squatter -C /etc/imapd_@@TYPE@@.conf -i" at=2200
}

======= Cyrus Replic ==============
# cyrus_@@TYPE@@.conf
# Template MD5SUM: @@MD5SUM@@

START {
@@TYPE@@recover cmd="ctl_cyrusdb -r -C /etc/imapd_@@TYPE@@.conf"
}

SERVICES {
@@TYPE@@syncserver cmd="sync_server -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:csync" prefork=1 maxfds=1024
@@TYPE@@imap cmd="imapd -U 50 -C /etc/imapd_@@TYPE@@.conf"
listen="@@HOSTNAME@@:imap" prefork=1 maxfds=1024
}

EVENTS {
@@TYPE@@checkpoint cmd="ctl_cyrusdb -c -C
/etc/imapd_@@TYPE@@.conf" period=30
@@TYPE@@delprune cmd="cyr_expire -E 3 -X 60 -D 60 -C
/etc/imapd_@@TYPE@@.conf" at=0100
}

===============


Configuration for Backend/Failover Instance
# Template MD5SUM: @@MD5SUM@@
servername: @@HOSTNAME@@
configdirectory: /srv/cyrus-@@TYPE@@
partition-default: /srv/cyrus-@@TYPE@@
partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock
dav archivecache
archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
archive_enabled: 1
proc_path: /srv/tmpfs/proc-@@TYPE@@
mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
defaultpartition: ssd
admins: XXX

mupdate_server: @@MUPDATEHOSTNAME@@
mupdate_port: 3905
mupdate_authname: XXX
mupdate_password: XXX
proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

allowallsubscribe: 1

sync_host: @@SYNCHOST@@
sync_authname: XXX
sync_password: XXX
sync_port: 2005
guid_mode: sha1
sync_log: 1
sync_shutdown_file: /srv/cyrus-@@TYPE@@/sync/shutdown

sievedir: /srv/cyrus-@@TYPE@@/sieve
sieve_extensions: fileinto reject vacation imapflags notify include
envelope body relational regex subaddress copy
sieve_maxscriptsize: 150

syslog_prefix: @@TYPE@@

============== Imapd Replic ===============
# Configuration for Slave (Replica) Instance
# Template MD5SUM: @@MD5SUM@@
servername: @@HOSTNAME@@
configdirectory: /srv/cyrus-@@TYPE@@
partition-default: /srv/cyrus-@@TYPE@@
partition-ssd: /srv/cyrus-@@TYPE@@/ssd-part
metapartition-ssd: /srv/cyrus-ssd-@@TYPE@@/meta/ssd-part
metapartition_files: header index cache expunge squat annotations lock
dav archivecache
archivepartition-ssd: /srv/cyrus-hdd-@@TYPE@@/archive/ssd-part
archive_enabled: 1

proc_path: /srv/tmpfs/proc-@@TYPE@@
mboxname_lockpath: /srv/tmpfs/lock-@@TYPE@@
defaultpartition: ssd
admins: XXX

allowusermoves: 1
allowallsubscribe: 1

proxy_authname: XXX
proxy_password: XXX
proxyservers: XXX

sievedir: /srv/cyrus-@@TYPE@@/sieve
sieve_extensions: fileinto reject vacation imapflags notify include
envelope body relational regex subaddress copy
sieve_maxscriptsize: 150

sasl_pwcheck_method: saslauthd
sasl_mech_list: plain login
allowanonymouslogin: no
syslog_prefix: @@TYPE@@
=================================

The sync client is started as own service

I hope it helps

Regards

Michael
Best regards.
Четверг, 13 сентября 2018, 13:22 +05:00 от Michael Menge
Hi,
This setup is NOT SUPPORTED and WILL BREAK if the replication process
is triggered
from the wrong server (user is active on both servers, user switched
from one server
to the other while the sync-log file is still processed, after split
brain) and
some mailboxes have been subscribed, renamed created deleted.....
Also there is the risk of a race condition with subscriptions, if a
user subscribes
to multiple folders, the first will trigger a sync from A to B, but as
the folder
is subscribed on B it will trigger a sync from B to A, witch can undo the next
folder subscription.
These are only some cases that came to my mind. There will be more
cases and it
will be hard to debug. So DON'T DO IT!
What we do is, that we have distributed our users between multiple
instances, and each server is running one instance as master and one other
as replic. In case of failure or maintenance we stop the master instance, and
promote the corresponding replic and configure them so that they will sync
them back. If the old master is up to date we switch them back.
We use cyrus aggregator aka cyrus murder, and AFAIK fastmail also uses
multiple
instances on one server with nginx frontends
Regards,
    Michael
--------------------------------------------------------------------------------
M.Menge Tel.: (49) 7071/29-70316
Universität Tübingen Fax.: (49) 7071/29-5912
Zentrum für Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

----
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.

Loading...