Segfault in ffi_prep_closure_loc

Elastic Stack Elasticsearch
1

hello guys,

i tryed to install elasticsearch 1.3.1 today on a dev machine and got a
problem related to mlockall that i have never seen before and cant wrap my
head around.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f09617e040a, pid=3143, tid=139678674036480

JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build

1.7.0_65-b17)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode

linux-amd64 compressed oops)

Problematic frame:

C [jna1731590185807699933.tmp+0x1240a] ffi_prep_closure_loc+0x1a

this is on a vanilla install with just a few custom settings (mainly
related to /tmp mounted noexec), sysctl and limit settings are ok (the full
log is attached).

has anybody seen this before or has any clue why this might be happening ?

best regards, markus

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Can you try with Java 7u60?

Jörg

On Tue, Aug 12, 2014 at 11:58 AM, Markus Burger m4rkus.burger@gmail.com
wrote:

hello guys,

i tryed to install elasticsearch 1.3.1 today on a dev machine and got a
problem related to mlockall that i have never seen before and cant wrap my
head around.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f09617e040a, pid=3143, tid=139678674036480

JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build

1.7.0_65-b17)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode

linux-amd64 compressed oops)

Problematic frame:

C [jna1731590185807699933.tmp+0x1240a] ffi_prep_closure_loc+0x1a

this is on a vanilla install with just a few custom settings (mainly
related to /tmp mounted noexec), sysctl and limit settings are ok (the full
log is attached).

has anybody seen this before or has any clue why this might be happening ?

best regards, markus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG50%2B_CfH%2BnHpVLajRT63-4c1eCoi_MWgR5u7FHf871gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

sadly exactly the same error accours :confused:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fae3d43440a, pid=21440, tid=140387243357952

JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build

1.7.0_60-b19)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode

linux-amd64 compressed oops)

Problematic frame:

C [jna8009017922749255641.tmp+0x1240a] ffi_prep_closure_loc+0x1a

Am Dienstag, 12. August 2014 12:12:00 UTC+2 schrieb Jörg Prante:

Can you try with Java 7u60?

Jörg

On Tue, Aug 12, 2014 at 11:58 AM, Markus Burger <m4rkus...@gmail.com
<javascript:>> wrote:

hello guys,

i tryed to install elasticsearch 1.3.1 today on a dev machine and got a
problem related to mlockall that i have never seen before and cant wrap my
head around.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f09617e040a, pid=3143, tid=139678674036480

JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build

1.7.0_65-b17)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode

linux-amd64 compressed oops)

Problematic frame:

C [jna1731590185807699933.tmp+0x1240a] ffi_prep_closure_loc+0x1a

this is on a vanilla install with just a few custom settings (mainly
related to /tmp mounted noexec), sysctl and limit settings are ok (the full
log is attached).

has anybody seen this before or has any clue why this might be happening ?

best regards, markus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5f339f21-df3e-406c-9947-68443df5d307%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

OK. Can you check if your mem lock setting for the S user allows to
reserve locked memory?

ulimit -l

This is by default on RHEL 64kb which should be changed to unlimited.

Jörg

On Tue, Aug 12, 2014 at 1:30 PM, Markus Burger m4rkus.burger@gmail.com
wrote:

sadly exactly the same error accours :confused:

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007fae3d43440a, pid=21440, tid=140387243357952

JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build

1.7.0_60-b19)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode

linux-amd64 compressed oops)

Problematic frame:

C [jna8009017922749255641.tmp+0x1240a] ffi_prep_closure_loc+0x1a

Am Dienstag, 12. August 2014 12:12:00 UTC+2 schrieb Jörg Prante:

Can you try with Java 7u60?

Jörg

On Tue, Aug 12, 2014 at 11:58 AM, Markus Burger m4rkus...@gmail.com
wrote:

hello guys,

i tryed to install elasticsearch 1.3.1 today on a dev machine and got a
problem related to mlockall that i have never seen before and cant wrap my
head around.

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f09617e040a, pid=3143, tid=139678674036480

JRE version: Java(TM) SE Runtime Environment (7.0_65-b17) (build

1.7.0_65-b17)

Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode

linux-amd64 compressed oops)

Problematic frame:

C [jna1731590185807699933.tmp+0x1240a] ffi_prep_closure_loc+0x1a

this is on a vanilla install with just a few custom settings (mainly
related to /tmp mounted noexec), sysctl and limit settings are ok (the full
log is attached).

has anybody seen this before or has any clue why this might be happening
?

best regards, markus

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3ee4f06b-e207-4b4b-9b49-f908260d852e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5f339f21-df3e-406c-9947-68443df5d307%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/5f339f21-df3e-406c-9947-68443df5d307%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEVze7Do%3DXFtso%3Dazm-1WVKNx%3D%3DkhPACEVmOsb%2BvMAp_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE-up9o89szdPG6F%3DphZcLB3c2NpYchh%3DDqPeuHD5WxPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

And

rpm -q libffi

gives

libffi-3.0.5-3.2.el6.x86_64

?

Jörg

On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger m4rkus.burger@gmail.com
wrote:

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEWcVL8V%2BiAOn9am22h-B5hnQ%3D6aXswz_YwrqSeLgX-NA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

8

exactly...

[root@dev ~]# rpm -q libffi
libffi-3.0.5-3.2.el6.x86_64

markus

Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:

And

rpm -q libffi

gives

libffi-3.0.5-3.2.el6.x86_64

?

Jörg

On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger <m4rkus...@gmail.com
<javascript:>> wrote:

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

9

Same configuration runs here, on bare metal. Maybe you run a VM?

Just a shot in the dark, it looks like a bug in libffi, and there are more
recent versions like 3.0.5, so if you feel like hacking this bug, I would
try if libffi 3.1 from GitHub - atgreen/libffi: A portable foreign-function interface library. gives a different
picture.

Jörg

On Tue, Aug 12, 2014 at 2:59 PM, Markus Burger m4rkus.burger@gmail.com
wrote:

exactly...

[root@dev ~]# rpm -q libffi
libffi-3.0.5-3.2.el6.x86_64

markus

Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:

And

rpm -q libffi

gives

libffi-3.0.5-3.2.el6.x86_64

?

Jörg

On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger m4rkus...@gmail.com
wrote:

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEHzP8kKyjO9yGgydpCes0Ko7wdey6gUe5GMsGELteHLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

10

Yeah its a Vmware VM but i have a couple of ES 1.1 sitting in another
Cluster without Issues, are there known issues with running ES in a VM ?

Thanks Jörg! I'll update libffi the next couple of days

markus

Am Dienstag, 12. August 2014 15:12:16 UTC+2 schrieb Jörg Prante:

Same configuration runs here, on bare metal. Maybe you run a VM?

Just a shot in the dark, it looks like a bug in libffi, and there are more
recent versions like 3.0.5, so if you feel like hacking this bug, I would
try if libffi 3.1 from GitHub - atgreen/libffi: A portable foreign-function interface library. gives a
different picture.

Jörg

On Tue, Aug 12, 2014 at 2:59 PM, Markus Burger <m4rkus...@gmail.com
<javascript:>> wrote:

exactly...

[root@dev ~]# rpm -q libffi
libffi-3.0.5-3.2.el6.x86_64

markus

Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:

And

rpm -q libffi

gives

libffi-3.0.5-3.2.el6.x86_64

?

Jörg

On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger m4rkus...@gmail.com
wrote:

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6e894da-6ebe-4f30-9a7b-7daf76b4b4d4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

11

There are so many VMs but, together with native code paths like JNA +
mlockall + libffi etc. the chance that something odd breaks down the road
is always there...

With Vmware VM and mlockall, not everything can work like on bare metal.
Memory management strongly depends on the host OS settings. The guest can
not control the host resource management, memory pinning, swap etc. Beside
this: bug is bug, libffi should not break with SIGSEGV, in any case.

Jörg

On Tue, Aug 12, 2014 at 3:29 PM, Markus Burger m4rkus.burger@gmail.com
wrote:

Yeah its a Vmware VM but i have a couple of ES 1.1 sitting in another
Cluster without Issues, are there known issues with running ES in a VM ?

Thanks Jörg! I'll update libffi the next couple of days

markus

Am Dienstag, 12. August 2014 15:12:16 UTC+2 schrieb Jörg Prante:

Same configuration runs here, on bare metal. Maybe you run a VM?

Just a shot in the dark, it looks like a bug in libffi, and there are
more recent versions like 3.0.5, so if you feel like hacking this bug, I
would try if libffi 3.1 from GitHub - atgreen/libffi: A portable foreign-function interface library. gives a
different picture.

Jörg

On Tue, Aug 12, 2014 at 2:59 PM, Markus Burger m4rkus...@gmail.com
wrote:

exactly...

[root@dev ~]# rpm -q libffi
libffi-3.0.5-3.2.el6.x86_64

markus

Am Dienstag, 12. August 2014 14:51:45 UTC+2 schrieb Jörg Prante:

And

rpm -q libffi

gives

libffi-3.0.5-3.2.el6.x86_64

?

Jörg

On Tue, Aug 12, 2014 at 2:19 PM, Markus Burger m4rkus...@gmail.com
wrote:

[root@dev limits.d]# runuser -s /bin/bash elasticsearch -c 'ulimit -l'
unlimited
[root@dev limits.d]# sysctl vm.max_map_count
vm.max_map_count = 262144

markus

Am Dienstag, 12. August 2014 13:58:30 UTC+2 schrieb Jörg Prante:

"S User" --> "ES user"

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/044b2715-c566-476b-ba8f-035cb4cf0af6%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8485ac49-7d23-47f2-b28f-0b475052d293%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b6e894da-6ebe-4f30-9a7b-7daf76b4b4d4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/b6e894da-6ebe-4f30-9a7b-7daf76b4b4d4%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFz8H38HgFp-5ssBTebgey2Jxg%3DB4R533qOjrMTB6paew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.