Garbage collection issue

Elastic Stack Elasticsearch
1

I am trying to bulk index data of size 6GB , but garbage collection is taking long time so process is getting killed.

cluster.name: Elastic-Demo
node.name: es-node-1
node.master: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 178.90.91.223
hosts.resolve_timeout: 15
http.port: 9200
discovery.seed_hosts: ["178.90.91.223","178.90.91.146","178.90.91.90"]
cluster.initial_master_nodes: ["178.90.91.223"]

Heap size is 1g.

the logs

[2019-06-04T13:28:31,570][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9751] overhead, spent [1.2s] collecting in the last [1.3s]
[2019-06-04T13:28:37,024][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][young][9752][913] duration [5.3s], collections [1]/[5.4s], total [5.3s]/[29.6m], memory [2.7gb]->[2.9gb]/[3.9gb], all_pools {[young] [3.6mb]->[86.8mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [2.7gb]->[2.8gb]/[3.8gb]}
[2019-06-04T13:28:37,025][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9752] overhead, spent [5.3s] collecting in the last [5.4s]
[2019-06-04T13:28:39,261][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][young][9754][914] duration [1.2s], collections [1]/[1.2s], total [1.2s]/[29.6m], memory [2.9gb]->[2.9gb]/[3.9gb], all_pools {[young] [100.5mb]->[1.4mb]/[133.1mb]}{[survivor] [16.6mb]->[16.6mb]/[16.6mb]}{[old] [2.8gb]->[2.9gb]/[3.8gb]}
[2019-06-04T13:28:39,261][WARN ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9754] overhead, spent [1.2s] collecting in the last [1.2s]
[2019-06-04T13:28:40,262][INFO ][o.e.m.j.JvmGcMonitorService] [es-node-1] [gc][9755] overhead, spent [289ms] collecting in the last [1s]

How do i solve this

Thanks.

2

What's your heap size? What version are you on? What sort of data is it? How many shards+indices and how big are they? How are you sending it to Elasticsearch?

3

heap size: 1g
es version : 7.1.1
the data is basically an LDIF file which i convert to json and it has large number of fields probably hundred.(6GB file)

  • index                shard prirep state   docs  store ip            node

  • .kibana_1            0     r      STARTED    8   49kb 178.90.91.90  es-node-3

  • .kibana_1            0     p      STARTED    8   49kb 178.90.91.146 es-node-2

  • .kibana_task_manager 0     p      STARTED    2 12.8kb 178.90.91.146 es-node-2

  • .kibana_task_manager 0     r      STARTED    2 12.8kb 178.90.91.223 es-node-1

  • ldap-listener-stat   2     p      STARTED 5290  1.2mb 178.90.91.90  es-node-3

  • ldap-listener-stat   2     r      STARTED 5290  1.3mb 178.90.91.146 es-node-2

  • ldap-listener-stat   1     p      STARTED 5217  1.2mb 178.90.91.146 es-node-2

  • ldap-listener-stat   1     r      STARTED 5217  1.3mb 178.90.91.223 es-node-1

  • ldap-listener-stat   0     p      STARTED 5155  1.2mb 178.90.91.90  es-node-3

  • ldap-listener-stat   0     r      STARTED 5155  1.2mb 178.90.91.223 es-node-1

i have no other indices, the index that i was trying to bulk index into has been deleted.

I have a python script which takes LDIF file converts to json and index using Parallel_bulk.

4

Hi @warkolm,
Can you suggest anything.

5

How large are the documents? What bulk size are you using?

6 
if __name__ == '__main__':
    pb = parallel_bulk(es, insert_data("user.ldif"),chunk_size=10000, thread_count=8, queue_size=16,request_timeout=10000)
    deque(pb, maxlen=0)

This is the main function and has info you needed

7

The file is of 6GB, so the insert data function converts it to json .

Thanks

8

Right, but one big document or multiple?

9

multiple documents

10

Try using smaller bulk sizes and see if that helps. Ever larger bulk sizes foes not necessarily give better performance.

11

I tried reducing chunk size to 1000, and it still doesn't work.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.