# Memory leak while indexing data

**URL:** <https://discuss.elastic.co/t/memory-leak-while-indexing-data/3050>\
**Category:** Elasticsearch\
**Created:** [June 29, 2010, 12:41pm UTC](https://discuss.elastic.co/t/memory-leak-while-indexing-data/3050 "2010-06-29T12:41:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zaharije](https://avatars.discourse-cdn.com/v4/letter/z/ebca7d/32.png) [@Zaharije](https://discuss.elastic.co/u/Zaharije)\
**Post date:** [June 29, 2010, 12:41pm UTC](https://discuss.elastic.co/t/memory-leak-while-indexing-data/3050/1 "2010-06-29T12:41:17Z")

</div>

Hi

we are doing single index update via python script (there is sample code at  
the end of email) but after few millions of documents we are getting  
slowdown and "Long running GC". It seems that we are having similar problem  
as  
[http://elasticsearch-users.115913.n3.nabble.com/lack-of-memory-td762199.html#a762199](http://elasticsearch-users.115913.n3.nabble.com/lack-of-memory-td762199.html#a762199),  
but i did not found any solution in that mail.

we are having 8 node cluster (8GB of ram). Also we are using 24 shards and 2  
replicas. Actual numbers is rough estimate for 1bilion of documents but test  
failed far from that number (also question is, is that large number of  
shards x replicas too large? and how ES handles per-shard index - does it  
need to be whole in memory?).

We used following configuration:

cluster:  
name: ES-test

discovery:  
type: jgroups  
jgroups:  
config: tcp  
bind\_port: 7800  
bind\_addr: katta  
tcpping:  
initial\_hosts:  
katta[7800],k00[7800],k01[7800],k02[7800],k03[7800],k04[7800],k06[7800],k07[7800]

gateway.fs.location: /search-sharing  
gateway.type: fs

NFS is used for gateway.

Here is some dump from DEBUG log:

[11:18:39,215][INFO][cluster.metadata] [Gaia] Index [users0]:  
Update mapping [id120071] (dynamic)  
[11:18:52,367][WARN][monitor.jvm] [Gaia] Long GC collection  
occurred, took [13s], breached threshold [10s][11:19:04,513][WARN  
][monitor.jvm] [Gaia] Long GC collection occurred, took [12s],  
breached threshold [10s]  
[11:19:11,453][WARN][jgroups.FC] Received two credit  
requests from k07-63227 without any intervening messages; sen  
ding 1981561 credits[11:19:23,380][WARN][monitor.jvm] [Gaia]  
Long GC collection occurred, took [18.7s], breached threshold  
[10s][11:19:23,380][DEBUG][action.index] [Gaia] [users0][4],  
Node[katta-26574], [P], S[STARTED]: Failed to execute [[users  
0][id660070][4729da78-2392-4c41-9534-957be5ba1984], source[{"air\_class":  
"coach", "count": "true", "anual\_income": 4314, "hotel":  
"starwood", "zipcode": 94365, "sex": "female", "net\_worth":  
64362}]]java.lang.NullPointerException  
at  
org.elasticsearch.index.mapper.xcontent.XContentNumberFieldMapper$CachedNumericTokenStream.close(XContentNumberFieldMap  
per.java:216)  
at  
org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:196)  
at  
org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)  
at  
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)  
at  
org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752)  
at  
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:191)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.innerCreate(InternalIndexShard.java:222)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:210)  
at  
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:127)  
at  
org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)  
at  
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)  
at  
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)  
at  
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(T  
ransportShardReplicationOperationAction.java:252)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:619)  
[11:19:28,071][WARN][jgroups.FD] I was suspected by  
k04-49735; ignoring the SUSPECT message and sending back a HEA  
RTBEAT\_ACK[11:19:42,368][WARN][monitor.jvm] [Gaia] Long GC  
collection occurred, took [18.7s], breached threshold [10s]

And after this there are more and more of the same message.

Here is sample of app which is doing indexing.

################################################################################  
def createDocuments(a\_index, a\_type, a\_cfg):  
dGen = dataGenerator.dataGenerator()  
dGen.init('data/fields.txt')  
nodes = getNodeList(a\_cfg)  
node = dGen.getRandomListMember(nodes)  
address = node['address']  
port = node['port']  
url = address + ':' + str(port)  
numOfDoc = int(a\_cfg.getNode('create/indexes/numOfDocuments').text)  
numOfRet = int(a\_cfg.getNode('create/indexes/numOfRetries').text)

```
idx = ElasticSearch(url)

success = 0
errors = 0
startTime = time.time()
for i in range (0, numOfDoc):
    data = getRandomData(dGen)
    numOfRetries = 0
    while 1:
        try:
            ret = idx.index(data, a_index, a_type)
            if ret.has_key('ok') and ret['ok'] == True:
                success += 1
            else:
                #print 'Error: ' + str(ret)
                errors += 1
            break
        except:
            #print 'An error has occuerd, retrying....'
            if numOfRetries == numOfRet:
                #print 'Unable to recover after ' + str(numOfRet) + '

```

retries.'  
break  
numOfRetries += 1  
pass

```
endTime = time.time()
totalTime = endTime - startTime
print 'Generated: ' + str(success) + ' records, errors: ' + str(errors)

```

- ', time: ' + time.strftime('%M:%S', time.localtime(totalTime))

## Thanks Zaharije

View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Memory-leak-while-indexing-data-tp930459p930459.html](http://elasticsearch-users.115913.n3.nabble.com/Memory-leak-while-indexing-data-tp930459p930459.html)  
Sent from the ElasticSearch Users mailing list archive at [Nabble.com](http://Nabble.com).

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [June 29, 2010, 10:33pm UTC](https://discuss.elastic.co/t/memory-leak-while-indexing-data/3050/2 "2010-06-29T22:33:31Z")

</div>

For some reason this got into the google groups spam filter, but I see you  
managed to post it, so answer in the other thread...

On Tue, Jun 29, 2010 at 3:41 PM, Zaharije [pasalic.zaharije@gmail.com](mailto:pasalic.zaharije@gmail.com)wrote:

> Hi
> 
> we are doing single index update via python script (there is sample code at  
> the end of email) but after few millions of documents we are getting  
> slowdown and "Long running GC". It seems that we are having similar problem  
> as
> 
> [http://elasticsearch-users.115913.n3.nabble.com/lack-of-memory-td762199.html#a762199](http://elasticsearch-users.115913.n3.nabble.com/lack-of-memory-td762199.html#a762199)  
> ,  
> but i did not found any solution in that mail.
> 
> we are having 8 node cluster (8GB of ram). Also we are using 24 shards and  
> 2  
> replicas. Actual numbers is rough estimate for 1bilion of documents but  
> test  
> failed far from that number (also question is, is that large number of  
> shards x replicas too large? and how ES handles per-shard index - does it  
> need to be whole in memory?).
> 
> We used following configuration:
> 
> cluster:  
> name: ES-test
> 
> discovery:  
> type: jgroups  
> jgroups:  
> config: tcp  
> bind\_port: 7800  
> bind\_addr: katta  
> tcpping:  
> initial\_hosts:
> 
> katta[7800],k00[7800],k01[7800],k02[7800],k03[7800],k04[7800],k06[7800],k07[7800]
> 
> gateway.fs.location: /search-sharing  
> gateway.type: fs
> 
> NFS is used for gateway.
> 
> Here is some dump from DEBUG log:
> 
> [11:18:39,215][INFO][cluster.metadata] [Gaia] Index [users0]:  
> Update mapping [id120071] (dynamic)  
> [11:18:52,367][WARN][monitor.jvm] [Gaia] Long GC collection  
> occurred, took [13s], breached threshold [10s][11:19:04,513][WARN  
> ][monitor.jvm] [Gaia] Long GC collection occurred, took  
> [12s],  
> breached threshold [10s]  
> [11:19:11,453][WARN][jgroups.FC] Received two credit  
> requests from k07-63227 without any intervening messages; sen  
> ding 1981561 credits[11:19:23,380][WARN][monitor.jvm] [Gaia]  
> Long GC collection occurred, took [18.7s], breached threshold  
> [10s][11:19:23,380][DEBUG][action.index] [Gaia] [users0][4],  
> Node[katta-26574], [P], S[STARTED]: Failed to execute [[users  
> 0][id660070][4729da78-2392-4c41-9534-957be5ba1984], source[{"air\_class":  
> "coach", "count": "true", "anual\_income": 4314, "hotel":  
> "starwood", "zipcode": 94365, "sex": "female", "net\_worth":  
> 64362}]]java.lang.NullPointerException  
> at
> 
> org.elasticsearch.index.mapper.xcontent.XContentNumberFieldMapper$CachedNumericTokenStream.close(XContentNumberFieldMap  
> per.java:216)  
> at
> 
> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:196)  
> at
> 
> org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:246)  
> at
> 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:774)  
> at
> 
> org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:752)  
> at  
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1932)  
> at
> 
> org.elasticsearch.index.engine.robin.RobinEngine.create(RobinEngine.java:191)  
> at
> 
> org.elasticsearch.index.shard.service.InternalIndexShard.innerCreate(InternalIndexShard.java:222)  
> at
> 
> org.elasticsearch.index.shard.service.InternalIndexShard.create(InternalIndexShard.java:210)  
> at
> 
> org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:127)  
> at
> 
> org.elasticsearch.action.index.TransportIndexAction.shardOperationOnPrimary(TransportIndexAction.java:56)  
> at
> 
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:328)  
> at
> 
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.access$400(TransportShardReplicationOperationAction.java:198)  
> at
> 
> org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(T  
> ransportShardReplicationOperationAction.java:252)  
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:619)  
> [11:19:28,071][WARN][jgroups.FD] I was suspected by  
> k04-49735; ignoring the SUSPECT message and sending back a HEA  
> RTBEAT\_ACK[11:19:42,368][WARN][monitor.jvm] [Gaia] Long GC  
> collection occurred, took [18.7s], breached threshold [10s]
> 
> And after this there are more and more of the same message.
> 
> Here is sample of app which is doing indexing.
> 
> ################################################################################  
> def createDocuments(a\_index, a\_type, a\_cfg):  
> dGen = dataGenerator.dataGenerator()  
> dGen.init('data/fields.txt')  
> nodes = getNodeList(a\_cfg)  
> node = dGen.getRandomListMember(nodes)  
> address = node['address']  
> port = node['port']  
> url = address + ':' + str(port)  
> numOfDoc = int(a\_cfg.getNode('create/indexes/numOfDocuments').text)  
> numOfRet = int(a\_cfg.getNode('create/indexes/numOfRetries').text)
> 
> ```
> idx = ElasticSearch(url)
> 
> success = 0
> errors = 0
> startTime = time.time()
> for i in range (0, numOfDoc):
> data = getRandomData(dGen)
> numOfRetries = 0
> while 1:
> try:
> ret = idx.index(data, a_index, a_type)
> if ret.has_key('ok') and ret['ok'] == True:
> success += 1
> else:
> #print 'Error: ' + str(ret)
> errors += 1
> break
> except:
> #print 'An error has occuerd, retrying....'
> if numOfRetries == numOfRet:
> #print 'Unable to recover after ' + str(numOfRet) + '
> 
> ```
> 
> retries.'  
> break  
> numOfRetries += 1  
> pass
> 
> ```
> endTime = time.time()
> totalTime = endTime - startTime
> print 'Generated: ' + str(success) + ' records, errors: ' + str(errors)
> 
> ```
> 
> - ', time: ' + time.strftime('%M:%S', time.localtime(totalTime))
> 
> ## Thanks Zaharije
> 
> View this message in context:  
> [http://elasticsearch-users.115913.n3.nabble.com/Memory-leak-while-indexing-data-tp930459p930459.html](http://elasticsearch-users.115913.n3.nabble.com/Memory-leak-while-indexing-data-tp930459p930459.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:22am UTC](https://discuss.elastic.co/t/memory-leak-while-indexing-data/3050/3 "2017-07-06T04:22:54Z")

</div>