Help! Cyrus 2.5.11 segmentation fault

[Marco Chesi]

Hello,
we have a Cyrus 2.5.11 on Debian 7 using murder (2 frontends, 3
backends, 1 mupdate, about  5000 mailboxes) in production environment,
hosted by a VMware cluster 6.5.

Suddenly, ALL mailboxes have becomed inaccessible.

In the log, we found many messages like this:

   master[5965]: process type:SERVICE name:imaps
path:/usr/cyrus/bin/proxyd age:82.255s pid:5987 signaled to death by
signal 11 (Segmentation fault)

I tried many times to restart the cyrus service on all servers without
success.

Users are authenticated correctly, error occurs when they try to access
the mailbox (trying with a manual connection using telnet, client dies
on a "SELECT INBOX" command after a successfull login)

This is the output of one core dump:

    Core was generated by `proxyd -s'.
Program terminated with signal 11, Segmentation fault.
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
32        ../sysdeps/x86_64/multiarch/../strlen.S: File o directory non
esistente.
(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
#1  0x00007f58caffa656 in xstrdup (str=0x0) at lib/xmalloc.c:95
#2  0x00007f58cb574b93 in parse_capability (s=***@entry=0x2223580,
str=<optimized out>) at imap/backend.c:282
#3  0x00007f58cb576362 in parse_capability (str=<optimized out>,
s=0x2223580) at imap/backend.c:279
#4  backend_login (noauth=6486368, auth_status=0x0, cb=0x2233b20,
userid=0x2235360 "xxxx", ret=0x2223580) at imap/backend.c:833
#5  backend_connect (ret_backend=***@entry=0x0, server=0xc
<Address 0xc out of bounds>, ***@entry=0x2233d90 "imap2",
prot=0x7ffccb275a40, ***@entry=0x62f840,
    userid=***@entry=0x2235360 "xxxx", cb=0x2233b20, ***@entry=0x0,
auth_status=***@entry=0x0, logfd=***@entry=-1) at
imap/backend.c:1027
#6  0x0000000000425915 in proxy_findserver (server=0x2233d90 "imap2",
prot=***@entry=0x62f840, userid=0x2235360 "xxxx", cache=0x62fb70,
current=0x62fb78, inbox=0x62fb80, clientin=0x21ed470)
    at imap/proxy.c:173
#7  0x0000000000417a0d in proxy_findinboxserver (userid=<optimized out>)
at imap/imap_proxy.c:129
#8  0x0000000000412f63 in cmd_list (tag=0x221ad00 "6",
listargs=***@entry=0x7ffccb276a60) at imap/imapd.c:6728
#9  0x0000000000420afc in cmdloop () at imap/imapd.c:1638
#10 0x0000000000424f4e in service_main (argc=<optimized out>,
argv=<optimized out>, envp=***@entry=0x7ffccb27a9e0) at imap/imapd.c:984
#11 0x0000000000416962 in main (argc=<optimized out>, argv=<optimized
out>, envp=0x7ffccb27a9e0) at master/service.c:606
(gdb)

Any suggestions or support?


Greetings,
Marco

[Michael Menge]

Hi,

Post by Marco Chesi
Hello,
we have a Cyrus 2.5.11 on Debian 7 using murder (2 frontends, 3
backends, 1 mupdate, about  5000 mailboxes) in production
environment, hosted by a VMware cluster 6.5.
Suddenly, ALL mailboxes have becomed inaccessible.
   master[5965]: process type:SERVICE name:imaps
path:/usr/cyrus/bin/proxyd age:82.255s pid:5987 signaled to death by
signal 11 (Segmentation fault)
I tried many times to restart the cyrus service on all servers
without success.
Users are authenticated correctly, error occurs when they try to
access the mailbox (trying with a manual connection using telnet,
client dies on a "SELECT INBOX" command after a successfull login)
    Core was generated by `proxyd -s'.
Program terminated with signal 11, Segmentation fault.
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
32        ../sysdeps/x86_64/multiarch/../strlen.S: File o directory
non esistente.
(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:32
#1  0x00007f58caffa656 in xstrdup (str=0x0) at lib/xmalloc.c:95
str=<optimized out>) at imap/backend.c:282
#3  0x00007f58cb576362 in parse_capability (str=<optimized out>,
s=0x2223580) at imap/backend.c:279
#4  backend_login (noauth=6486368, auth_status=0x0, cb=0x2233b20,
userid=0x2235360 "xxxx", ret=0x2223580) at imap/backend.c:833
#6  0x0000000000425915 in proxy_findserver (server=0x2233d90
cache=0x62fb70, current=0x62fb78, inbox=0x62fb80, clientin=0x21ed470)
    at imap/proxy.c:173
#7  0x0000000000417a0d in proxy_findinboxserver (userid=<optimized
out>) at imap/imap_proxy.c:129
#8  0x0000000000412f63 in cmd_list (tag=0x221ad00 "6",
#9  0x0000000000420afc in cmdloop () at imap/imapd.c:1638
#10 0x0000000000424f4e in service_main (argc=<optimized out>,
imap/imapd.c:984
#11 0x0000000000416962 in main (argc=<optimized out>,
argv=<optimized out>, envp=0x7ffccb27a9e0) at master/service.c:606
(gdb)
Any suggestions or support?

As far as i can tell is that the frontend imapd-proxy process is
receiving the SIGSEGV
after it connected and authenticate to the backend. The imapd-proxy is
parsing the
the capability output of the backend.

What is strange is that the parse_cpability is called with a NULL
pointer (str=0x0)
which is then used in xstrdup.

So a check to ensure that str != NULL would help to prevent the
SIGSEGV, but a am
not sure what should be done in the str == NULL case.


The other question is why is str == NULL in the first place?
Can you try to manual connect to the backend using telnet, and check
the CAPABILITY output.


And as always the frustrating question: Did something change on your backends?

Regards,

Michael Menge

--------------------------------------------------------------------------------
M.Menge Tel.: [phone]
Universität Tübingen Fax.: [phone]
Zentrum für Datenverarbeitung mail:
***@zdv.uni-tuebingen.de
Wächterstraße 76
72074 Tübingen

[Wolfgang Breyha]

Hi!

Sorry for the late answer... most likely you already fixed your issue...

Post by Marco Chesi
Hello,
we have a Cyrus 2.5.11 on Debian 7 using murder (2 frontends, 3 backends, 1
mupdate, about  5000 mailboxes) in production environment, hosted by a
VMware cluster 6.5.
Suddenly, ALL mailboxes have becomed inaccessible.
   master[5965]: process type:SERVICE name:imaps path:/usr/cyrus/bin/proxyd
age:82.255s pid:5987 signaled to death by signal 11 (Segmentation fault)
I tried many times to restart the cyrus service on all servers without
success.
Users are authenticated correctly, error occurs when they try to access the
mailbox (trying with a manual connection using telnet, client dies on a
"SELECT INBOX" command after a successfull login)

Actually that's the point when your frontend connects to your backend to
access the mailbox. So, you've troubles with backend authentication. Most
likely the proxy user is somehow broken.

At this point you're hit by an quite old bug...

2.5.11 misses the patch for this issue which was committed (backported from
3.0 where it was applied in 2016) in November 2017. Don't know why no
release was done for that yet.

see:
https://github.com/cyrusimap/cyrus-imapd/issues/40
At the bottom you'll find the two patches which fix this issue.

If you're able to rebuild new packages include these patches. If not,
you've to convince the debian people to provide an update with them.

But most likely the issue goes away if you fix your backend authentication
anyway.

Greetings, Wolfgang