JVM crashes on SunOS

Elastic Stack Elasticsearch
1

Hi,

I'm experiencing JVM crashes while running ElasticSearch on Joyent's
SmartOS (an OpenSolaris fork).

I was originally using mmapfs, but switched to niofs after seeing crashes
related to MMapDirectory (and reading
http://stackoverflow.com/questions/8224843/jvm-crashes-on-lucene-datainput-readvint).
However, I'm still getting crashes that hint at MMap being a problem. I
only have one index on this cluster, and started with a clean install and
data directory before creating the niofs index.

Here are my index settings:

curl localhost:9200/_settings?pretty=true
{
"foo" : {
"settings" : {
"index.store" : "niofs",
"index.number_of_replicas" : "2",
"index.number_of_shards" : "80",
"index.version.created" : "190199"
}
}
}

These are the problematic frames:

for f in hs_err_pid*; do sed '9!d' $f; done | sort | uniq -c | sort -n
1 # v ~StubRoutines::jint_disjoint_arraycopy
2 # v ~StubRoutines::jshort_disjoint_arraycopy
3 # J
org.apache.lucene.index.TermBuffer.read(Lorg/apache/lucene/store/IndexInput;Lorg/apache/lucene/index/FieldInfos;)V
3 # J org.apache.lucene.store.DataInput.readVLong()J
3 # J
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache$BloomFilterLoader.run()V
6 # J
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte()B
12 # v ~StubRoutines::jbyte_disjoint_arraycopy
23 # J org.apache.lucene.index.SegmentTermEnum.next()Z

These are the JVM versions I've tried:

JRE version: 6.0_26-b03

Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode

solaris-amd64 )

JRE version: 7.0_03-b04

Java VM: Java HotSpot(TM) 64-Bit Server VM (22.1-b02 mixed mode

solaris-amd64 )

These are the ElasticSearch versions I've tried:
0.18.7
0.19.0
0.19.1

The frequency of crashes went down significantly after switching to niofs,
but still occur regularly. Are these crashes still mmap related? Have I not
set the index store correctly?

Thanks!

2

It seems like mmapfs is still being used, its because the proper setting
for it is index.store.type. You can simply set it in the elasticsearch.yml
and it will be applied to all indices created. Hopefully those failure will
go away with Lucene 3.6 when its out.

On Wed, Apr 11, 2012 at 10:42 PM, Danny Coates dannycoates@gmail.comwrote:

Hi,

I'm experiencing JVM crashes while running Elasticsearch on Joyent's
SmartOS (an OpenSolaris fork).

I was originally using mmapfs, but switched to niofs after seeing crashes
related to MMapDirectory (and reading
java - JVM crashes on Lucene DataInput.readVInt - Stack Overflow).
However, I'm still getting crashes that hint at MMap being a problem. I
only have one index on this cluster, and started with a clean install and
data directory before creating the niofs index.

Here are my index settings:

curl localhost:9200/_**settings?pretty=true
{
"foo" : {
"settings" : {
"index.store" : "niofs",
"index.number_of_replicas" : "2",
"index.number_of_shards" : "80",
"index.version.created" : "190199"
}
}
}

These are the problematic frames:

for f in hs_err_pid*; do sed '9!d' $f; done | sort | uniq -c | sort -n
1 # v ~StubRoutines::jint_disjoint_arraycopy
2 # v ~StubRoutines::jshort_disjoint_arraycopy
3 # J
org.apache.lucene.index.TermBuffer.read(Lorg/apache/lucene/store/IndexInput;Lorg/apache/lucene/index/FieldInfos;)V
3 # J org.apache.lucene.store.DataInput.readVLong()J
3 # J
org.elasticsearch.index.cache.bloom.simple.SimpleBloomCache$BloomFilterLoader.run()V
6 # J
org.apache.lucene.store.MMapDirectory$MMapIndexInput.readByte()B
12 # v ~StubRoutines::jbyte_disjoint_arraycopy
23 # J org.apache.lucene.index.SegmentTermEnum.next()Z

These are the JVM versions I've tried:

JRE version: 6.0_26-b03

Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode

solaris-amd64 )

JRE version: 7.0_03-b04

Java VM: Java HotSpot(TM) 64-Bit Server VM (22.1-b02 mixed mode

solaris-amd64 )

These are the Elasticsearch versions I've tried:
0.18.7
0.19.0
0.19.1

The frequency of crashes went down significantly after switching to niofs,
but still occur regularly. Are these crashes still mmap related? Have I not
set the index store correctly?

Thanks!

3

Thank you! This works great. No more crashes.

Is there a definitive resource with all the possible settings listed out
(besides source code)? In this case I
used http://www.elasticsearch.org/guide/reference/index-modules/store.html
to find the proper value, but mislead myself to setting it to the wrong
property.

It would also have helped me if I could have seen output like, "index.store
is a user defined property", somewhere. Then I wouldn't have believed that
I set it correctly. Does such logging exist?

This isn't the first time I've had trouble with settings. A definitive list
would help immensely.

Thanks again,
Danny

4

The guide lists the relevant settings for each section, the funny thing is
that the guide store section doesn't properly list this setting how to set
it..., will fix it.

On Sun, Apr 15, 2012 at 7:20 PM, Danny Coates dannycoates@gmail.com wrote:

Thank you! This works great. No more crashes.

Is there a definitive resource with all the possible settings listed out
(besides source code)? In this case I used
Elasticsearch Platform — Find real-time answers at scale | Elastic to
find the proper value, but mislead myself to setting it to the wrong
property.

It would also have helped me if I could have seen output like,
"index.store is a user defined property", somewhere. Then I wouldn't have
believed that I set it correctly. Does such logging exist?

This isn't the first time I've had trouble with settings. A definitive
list would help immensely.

Thanks again,
Danny