Elasticsearch run as a service dies on first request

Elastic Stack Elasticsearch
1

I'm running elasticsearch on Centos using the *serivce elasticsearch
start. *The service happily stays running until it receives its first
request causing it to die.
service elasticsearch status returns

elasticsearch dead but pid file exists

This same problem doesn't exist when I run elasticsearch with
./bin/elasticsearch

With logging set to trace I see no stacktrace in the logs or any indication
the service went down.

Where else can I look to find why elasticsearch will not accept request
running as a service?

More info:
Elasticsearch
version:{
number: "1.3.4",
build_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
build_timestamp: "2014-09-30T09:07:17Z",
build_snapshot: false,
lucene_version: "4.9"

CentOS 6.5

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Lack of permissions on filesystem access? Unexpected directory.

Something like dtrace/truss/strace might be useful here.
On 03/11/2014 1:19 pm, "Jef Statham" jef.statham@gmail.com wrote:

I'm running elasticsearch on Centos using the *serivce elasticsearch
start. *The service happily stays running until it receives its first
request causing it to die.
service elasticsearch status returns

elasticsearch dead but pid file exists

This same problem doesn't exist when I run elasticsearch with
./bin/elasticsearch

With logging set to trace I see no stacktrace in the logs or any
indication the service went down.

Where else can I look to find why elasticsearch will not accept request
running as a service?

More info:
Elasticsearch
version:{
number: "1.3.4",
build_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
build_timestamp: "2014-09-30T09:07:17Z",
build_snapshot: false,
lucene_version: "4.9"

CentOS 6.5

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for the strace suggestion, this is what my trace returns. I'm
looking into what futex is now.

[root@localhost elasticsearch]# sudo service elasticsearch start

Starting elasticsearch: [ OK ]

[root@localhost elasticsearch]# sudo service elasticsearch status

elasticsearch (pid 16887) is running...

[root@localhost elasticsearch]# strace -p 16887

\Process 16887 attached - interrupt to quit

futex(0x7f97ed4279d0, FUTEX_WAIT, 16889, NUL <unfinished ...>

+++ killed by SIGKILL +++

[root@localhost elasticsearch]# sudo service elasticsearch start

Starting elasticsearch: [ OK ]

[root@localhost elasticsearch]# sudo service elasticsearch status

elasticsearch (pid 16958) is running...

[root@localhost elasticsearch]# strace -p 16958

Process 16958 attached - interrupt to quit

futex(0x7f0a7e8089d0, FUTEX_WAIT, 16960, NULL <unfinished ...>

+++ killed by SIGKILL +++

-Jef Statham

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!

On Mon, Nov 3, 2014 at 2:01 PM, Alexandre Rafalovitch arafalov@gmail.com
wrote:

Lack of permissions on filesystem access? Unexpected directory.

Something like dtrace/truss/strace might be useful here.
On 03/11/2014 1:19 pm, "Jef Statham" jef.statham@gmail.com wrote:

I'm running elasticsearch on Centos using the *serivce elasticsearch
start. *The service happily stays running until it receives its first
request causing it to die.
service elasticsearch status returns

elasticsearch dead but pid file exists

This same problem doesn't exist when I run elasticsearch with
./bin/elasticsearch

With logging set to trace I see no stacktrace in the logs or any
indication the service went down.

Where else can I look to find why elasticsearch will not accept request
running as a service?

More info:
Elasticsearch
version:{
number: "1.3.4",
build_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45",
build_timestamp: "2014-09-30T09:07:17Z",
build_snapshot: false,
lucene_version: "4.9"

CentOS 6.5

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj8sWJPb2ATE8o%2Bpf36GNSF%3Dws7%3DkBo4pR%2BJmfjiXZ8rjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

I was actually thinking more about using trace to monitor file system
access (not just open, access as well).

Regards,
Alex.

On 3 November 2014 14:08, Jef Statham jef.statham@gmail.com wrote:

Thanks for the strace suggestion, this is what my trace returns. I'm looking
into what futex is now.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: Sign Up | LinkedIn

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

These debugging commands are neat I'm learning as we go, here's some file
access stats:

[root@localhost ~]# service elasticsearch start

Starting elasticsearch: [ OK ]

[root@localhost ~]# service elasticsearch status

elasticsearch (pid 17129) is running...

[root@localhost ~]# strace -f -e trace=file -p 17129

Process 17129 attached with 35 threads - interrupt to quit

[pid 17163] lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share", {st_mode=S_IFDIR|0755, st_size=4096, ...})
= 0

[pid 17163] lstat("/usr/share/elasticsearch", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share/elasticsearch/lib", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar",
{st_mode=S_IFREG|0777, st_size=2507813, ...}) = 0

[pid 17163]
stat("/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.class",
0x7f1b301331c0) = -1 ENOENT (No such file or directory)

[pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
O_RDONLY) = 95

[pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
O_RDONLY) = 95

) = ?

[pid 17172] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17172 leader 17129

[pid 17171] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17171 leader 17129

[pid 17170] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17170 leader 17129

[pid 17169] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17169 leader 17129

[pid 17168] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17168 leader 17129

[pid 17167] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17167 leader 17129

[pid 17166] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17166 leader 17129

[pid 17165] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17165 leader 17129

[pid 17162] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17162 leader 17129

[pid 17161] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17161 leader 17129

[pid 17160] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17160 leader 17129

[pid 17159] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17159 leader 17129

[pid 17158] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17158 leader 17129

[pid 17157] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17157 leader 17129

[pid 17156] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17156 leader 17129

[pid 17155] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17155 leader 17129

[pid 17154] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17154 leader 17129

[pid 17153] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17153 leader 17129

[pid 17152] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17152 leader 17129

[pid 17142] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17142 leader 17129

[pid 17164] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17164 leader 17129

[pid 17163] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17163 leader 17129

[pid 17141] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17141 leader 17129

[pid 17140] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17140 leader 17129

[pid 17139] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17139 leader 17129

[pid 17138] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17138 leader 17129

[pid 17137] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17137 leader 17129

[pid 17136] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17136 leader 17129

[pid 17135] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17135 leader 17129

[pid 17134] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17134 leader 17129

[pid 17133] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17133 leader 17129

[pid 17132] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17132 leader 17129

[pid 17131] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17131 leader 17129

[pid 17151] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17151 leader 17129

+++ killed by SIGKILL +++

-Jef Statham

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!

On Mon, Nov 3, 2014 at 3:43 PM, Alexandre Rafalovitch arafalov@gmail.com
wrote:

I was actually thinking more about using trace to monitor file system
access (not just open, access as well).

Regards,
Alex.

On 3 November 2014 14:08, Jef Statham jef.statham@gmail.com wrote:

Thanks for the strace suggestion, this is what my trace returns. I'm
looking
into what futex is now.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: Sign Up | LinkedIn

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj_52g5E%2BAocsT998jYrXHpVUcDEKv9qb0Vj%2Ba9qHxtvyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Updating my command, could there be an issue with the write lock on the
events-production index?

[root@localhost ~]# strace -f -e trace=file,read,write -p 17197

[pid 17231]
stat("/var/lib/elasticsearch/elasticsearch/nodes/0/indices/events-production/2/index/write.lock",
{st_mode=S_IFREG|0644, st_size=0, ...}) = 0

[pid 17231] write(26, "[2014-11-03 16:18:31,244][DEBUG]"..., 138) = 138

[pid 17231]
stat("/var/lib/elasticsearch/elasticsearch/nodes/0/indices/events-production/2/translog",
{st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

[pid 17231] write(26, "[2014-11-03 16:18:31,257][DEBUG]"..., 316) = 316

[pid 17231] open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 142

[pid 17231] read(142, "0\n", 8192) = 2

[pid 17231] write(26, "[2014-11-03 16:18:31,273][DEBUG]"..., 179) = 179

[pid 17231] write(26, "[2014-11-03 16:18:31,274][DEBUG]"..., 116) = 116

[pid 17231] write(26, "[2014-11-03 16:18:31,275][DEBUG]"..., 207) = 207

) = ?

[pid 17232] +++ killed by SIGKILL +++

-Jef Statham

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!

On Mon, Nov 3, 2014 at 4:16 PM, Jef Statham jef.statham@gmail.com wrote:

These debugging commands are neat I'm learning as we go, here's some file
access stats:

[root@localhost ~]# service elasticsearch start

Starting elasticsearch: [ OK ]

[root@localhost ~]# service elasticsearch status

elasticsearch (pid 17129) is running...

[root@localhost ~]# strace -f -e trace=file -p 17129

Process 17129 attached with 35 threads - interrupt to quit

[pid 17163] lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share", {st_mode=S_IFDIR|0755, st_size=4096, ...})
= 0

[pid 17163] lstat("/usr/share/elasticsearch", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share/elasticsearch/lib", {st_mode=S_IFDIR|0777,
st_size=4096, ...}) = 0

[pid 17163] lstat("/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar",
{st_mode=S_IFREG|0777, st_size=2507813, ...}) = 0

[pid 17163]
stat("/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.class",
0x7f1b301331c0) = -1 ENOENT (No such file or directory)

[pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
O_RDONLY) = 95

[pid 17151] open("/usr/java/jdk1.8.0_25/jre/lib/amd64/server/libjvm.so",
O_RDONLY) = 95

) = ?

[pid 17172] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17172 leader 17129

[pid 17171] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17171 leader 17129

[pid 17170] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17170 leader 17129

[pid 17169] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17169 leader 17129

[pid 17168] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17168 leader 17129

[pid 17167] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17167 leader 17129

[pid 17166] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17166 leader 17129

[pid 17165] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17165 leader 17129

[pid 17162] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17162 leader 17129

[pid 17161] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17161 leader 17129

[pid 17160] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17160 leader 17129

[pid 17159] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17159 leader 17129

[pid 17158] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17158 leader 17129

[pid 17157] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17157 leader 17129

[pid 17156] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17156 leader 17129

[pid 17155] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17155 leader 17129

[pid 17154] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17154 leader 17129

[pid 17153] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17153 leader 17129

[pid 17152] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17152 leader 17129

[pid 17142] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17142 leader 17129

[pid 17164] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17164 leader 17129

[pid 17163] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17163 leader 17129

[pid 17141] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17141 leader 17129

[pid 17140] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17140 leader 17129

[pid 17139] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17139 leader 17129

[pid 17138] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17138 leader 17129

[pid 17137] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17137 leader 17129

[pid 17136] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17136 leader 17129

[pid 17135] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17135 leader 17129

[pid 17134] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17134 leader 17129

[pid 17133] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17133 leader 17129

[pid 17132] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17132 leader 17129

[pid 17131] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17131 leader 17129

[pid 17151] +++ killed by SIGKILL +++

PANIC: handle_group_exit: 17151 leader 17129

+++ killed by SIGKILL +++

-Jef Statham

Without vices there would be no virtues.
It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring!

On Mon, Nov 3, 2014 at 3:43 PM, Alexandre Rafalovitch arafalov@gmail.com
wrote:

I was actually thinking more about using trace to monitor file system
access (not just open, access as well).

Regards,
Alex.

On 3 November 2014 14:08, Jef Statham jef.statham@gmail.com wrote:

Thanks for the strace suggestion, this is what my trace returns. I'm
looking
into what futex is now.

Personal: http://www.outerthoughts.com/ and @arafalov
Solr resources and newsletter: http://www.solr-start.com/ and @solrstart
Solr popularizers community: Sign Up | LinkedIn

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj9a7LHywxVy6HQAgxLkG%2BKm_e2X1tVjpnRiMKtHsun9og%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

These look like debug messages. Do you know where they go and what
they say. Might be where the clue is. Otherwise, I am not sure. The
usual debugging method is to try comparing failing instance with a
successful one (on file access sequence). You could try that with a
fresh local copy of the ES and see.

Regards,
Alex.

On 3 November 2014 16:27, Jef Statham jef.statham@gmail.com wrote:

[pid 17231] write(26, "[2014-11-03 16:18:31,273][DEBUG]"..., 179) = 179
[pid 17231] write(26, "[2014-11-03 16:18:31,274][DEBUG]"..., 116) = 116
[pid 17231] write(26, "[2014-11-03 16:18:31,275][DEBUG]"..., 207) = 207

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G-h%2BVbvaZJU53pP6Z8abh8OuLb1%3Dqh1Qz3ORXXFTiDSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.