Data/write/bulk [s][r] - Some task running from more than 70days

Elastic Stack Elasticsearch
1

Hi All,

I wanted to understand what is the below task that is executing from past 76 days. Is it expected to run for such a long time or is there any issue with my elasticsearch cluster.
Also, we frequently face long GC pauses, was suspecting if this could be one of the reason for the issue.

[root@es-node-15 ~]# curl -XGET "localhost:9200/_cat/tasks?v"
action task_id parent_task_id type start_time timestamp running_time ip node
indices:data/write/bulk[s][r] -u1mvyirSXSO9pJKfwBODQ:154872000 IF-7DJwzTi2X0Sly5Pnk5g:253428804 netty 1550677531716 21:15:31 76.6d 10.0.47.162 es-warm-node-16

Can anyone please help me out?

Thanks,
Vikash

2

Which version are you using?

3

Hi David,

We are using Elasticsearch version : "5.4.2" and "lucene_version" : "6.5.1"

Thanks

4

In which case I would suspect #36770, which fixes a race condition that can lead to these long-running tasks. The fix is in versions ≥6.5.5.

Each such task only takes a small amount of heap, and prevents the parent tasks from completing too. The parent tasks will be holding onto a whole bulk request, which is normally not that large (kilobytes rather than gigabytes). It does seem more likely to trigger this race condition when under load (heavy GC etc) but it's not normally the cause of the GC pressure.

5

Hey David,

Thanks for the response. Also, may I know what could be the reason of the frequent and long GC pauses. As the long gc pauses, makes the elastic node inactive and which further makes my logstash machine think that there is dead es- instance.

We use Logstash as consumers from kafka cluster. Because of the long GC pauses, logstash stops taking any load and thus there is a lag in the kafka consumer.

Kafka-version : 0.10.1.1
Logstash version: 5.2.2
Elasticsearch version : 5.4.2

Can you please provide some insights on this?

Thanks,
Vikash

6

Normally this is because Elasticsearch is overloaded in some way, but I'm not the best person to answer about this.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.