OOME when using mapping

Anurag_Phadke · July 5, 2011, 4:12am

We are running a 5-node ElasticSearch 0.16.2 cluster, avg. size 20kb /
document. However, I get an OutOfMemoryException on trying to insert
1million JSONS with the following mapping.

curl -XPUT 'http://localhost:9200/socorro/' -d '
index :
number_of_shards : 30
number_of_replicas : 2
'

curl -XPUT 'http://localhost:9200/socorro/crash_reports/_mapping' -d '{
"crash_reports": {
"date_formats": [
"yyyy-MM-dd HH:mm:ss.SSSSSS"
],
"properties": {
"completeddatetime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"date_processed": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"client_crash_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"build_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"startedDateTime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"signature": {
"type": "multi_field",
"fields": {
"signature": {
"type": "string",
"index": "analyzed"
},
"full": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}'

No OOME are thrown if the custom mappings aren't added to the index.
Any idea what might be going wrong here?

-anurag

kimchy · July 5, 2011, 11:00am

Do you get the OOM right when you start to index, or after a time? Is there a chance that you can provide a link for me to download the heap dump generated?

On Tuesday, July 5, 2011 at 7:12 AM, Anurag wrote:

We are running a 5-node Elasticsearch 0.16.2 cluster, avg. size 20kb /
document. However, I get an OutOfMemoryException on trying to insert
1million JSONS with the following mapping.

curl -XPUT 'http://localhost:9200/socorro/' -d '
index :
number_of_shards : 30
number_of_replicas : 2
'

curl -XPUT 'http://localhost:9200/socorro/crash_reports/_mapping' -d '{
"crash_reports": {
"date_formats": [
"yyyy-MM-dd HH:mm:ss.SSSSSS"
],
"properties": {
"completeddatetime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"date_processed": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"client_crash_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"build_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"startedDateTime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"signature": {
"type": "multi_field",
"fields": {
"signature": {
"type": "string",
"index": "analyzed"
},
"full": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}'

No OOME are thrown if the custom mappings aren't added to the index.
Any idea what might be going wrong here?

-anurag

Anurag_Phadke · July 5, 2011, 5:00pm

Shay,
I don't get the error when starting the index, it happens after
inserting about 100k jsons.
Here's the dump:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "HashSessionScavenger-0"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "New I/O client worker #1-4"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]transport_client_worker-pool-13-thread-8"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]discovery#multicast#received-pool-16-thread-1"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]transport_client_worker-pool-13-thread-2"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

On Tue, Jul 5, 2011 at 4:00 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Do you get the OOM right when you start to index, or after a time? Is there
a chance that you can provide a link for me to download the heap dump
generated?

On Tuesday, July 5, 2011 at 7:12 AM, Anurag wrote:

We are running a 5-node Elasticsearch 0.16.2 cluster, avg. size 20kb /
document. However, I get an OutOfMemoryException on trying to insert
1million JSONS with the following mapping.

curl -XPUT 'http://localhost:9200/socorro/' -d '
index :
number_of_shards : 30
number_of_replicas : 2
'

curl -XPUT 'http://localhost:9200/socorro/crash_reports/_mapping' -d '{
"crash_reports": {
"date_formats": [
"yyyy-MM-dd HH:mm:ss.SSSSSS"
],
""properties": {
"completeddatetime": {
""format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
""type": "date"
},
"date_processed": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"client_crash_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"build_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
""type": "date"
},
"startedDateTime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
}},
"signature": {
"type": "multi_field",
"fields": {
""signature": {
"type": "string",
"index": "analyzed"
}},
"full": {
""type": "string",
"index": "not_analyzed"
}
}
}
}
}
}'

No OOME are thrown if the custom mappings aren't added to the index.
Any idea what might be going wrong here?

-anurag

kimchy · July 6, 2011, 1:24am

Thats the exceptions, not the heap dump... . How are you running the indexing? Is it against an embedded instance? Do you have enough memory? Are you using bulk indexing, if so, whats your bulk size?

On Tuesday, July 5, 2011 at 8:00 PM, Anurag wrote:

Shay,
I don't get the error when starting the index, it happens after
inserting about 100k jsons.
Here's the dump:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "HashSessionScavenger-0"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "New I/O client worker #1-4"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]transport_client_worker-pool-13-thread-8"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]discovery#multicast#received-pool-16-thread-1"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]transport_client_worker-pool-13-thread-2"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

On Tue, Jul 5, 2011 at 4:00 AM, Shay Banon <shay.banon@elasticsearch.com (mailto:shay.banon@elasticsearch.com)> wrote:

Do you get the OOM right when you start to index, or after a time? Is there
a chance that you can provide a link for me to download the heap dump
generated?

On Tuesday, July 5, 2011 at 7:12 AM, Anurag wrote:

We are running a 5-node Elasticsearch 0.16.2 cluster, avg. size 20kb /
document. However, I get an OutOfMemoryException on trying to insert
1million JSONS with the following mapping.

curl -XPUT 'http://localhost:9200/socorro/' -d '
index :
number_of_shards : 30
number_of_replicas : 2
'

curl -XPUT 'http://localhost:9200/socorro/crash_reports/_mapping' -d '{
"crash_reports": {
"date_formats": [
"yyyy-MM-dd HH:mm:ss.SSSSSS"
],
""properties": {
"completeddatetime": {
""format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
""type": "date"
},
"date_processed": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"client_crash_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"build_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
""type": "date"
},
"startedDateTime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
}},
"signature": {
"type": "multi_field",
"fields": {
""signature": {
"type": "string",
"index": "analyzed"
}},
"full": {
""type": "string",
"index": "not_analyzed"
}
}
}
}
}
}'

No OOME are thrown if the custom mappings aren't added to the index.
Any idea what might be going wrong here?

-anurag

Anurag_Phadke · July 6, 2011, 4:31am

Will try to get u the heap dump by tomorrow
Indexing is against an embedded instance, xms as 2g and Xmx as 8g. Using bulk indexing with 100 jsons, each approximately 20kb in size.

On Jul 5, 2011, at 6:24 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Thats the exceptions, not the heap dump... . How are you running the indexing? Is it against an embedded instance? Do you have enough memory? Are you using bulk indexing, if so, whats your bulk size?
On Tuesday, July 5, 2011 at 8:00 PM, Anurag wrote:

Shay,
I don't get the error when starting the index, it happens after
inserting about 100k jsons.
Here's the dump:
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "HashSessionScavenger-0"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "New I/O client worker #1-4"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]transport_client_worker-pool-13-thread-8"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]discovery#multicast#received-pool-16-thread-1"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "elasticsearch[Madame
MacEvil]transport_client_worker-pool-13-thread-2"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

On Tue, Jul 5, 2011 at 4:00 AM, Shay Banon shay.banon@elasticsearch.com wrote:

Do you get the OOM right when you start to index, or after a time? Is there
a chance that you can provide a link for me to download the heap dump
generated?

On Tuesday, July 5, 2011 at 7:12 AM, Anurag wrote:

We are running a 5-node Elasticsearch 0.16.2 cluster, avg. size 20kb /
document. However, I get an OutOfMemoryException on trying to insert
1million JSONS with the following mapping.

curl -XPUT 'http://localhost:9200/socorro/' -d '
index :
number_of_shards : 30
number_of_replicas : 2
'

curl -XPUT 'http://localhost:9200/socorro/crash_reports/_mapping' -d '{
"crash_reports": {
"date_formats": [
"yyyy-MM-dd HH:mm:ss.SSSSSS"
],
""properties": {
"completeddatetime": {
""format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
""type": "date"
},
"date_processed": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"client_crash_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
},
"build_date": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
""type": "date"
},
"startedDateTime": {
"format": "yyyy-MM-dd HH:mm:ss.SSSSSS",
"type": "date"
}},
"signature": {
"type": "multi_field",
"fields": {
""signature": {
"type": "string",
"index": "analyzed"
}},
"full": {
""type": "string",
"index": "not_analyzed"
}
}
}
}
}
}'

No OOME are thrown if the custom mappings aren't added to the index.
Any idea what might be going wrong here?

-anurag