Any idea what is causing this error Elastic 2.4.2

Elastic Stack Elasticsearch
1

Some of our queries keep failing since we upgraded from 1.7.1 to 2.4.2, any idea what is causing these errors?

[2016-12-01 20:21:36,064][ERROR][marvel.agent ] [ElasticSearch-10] background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)
at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)
at java.lang.Thread.run(Thread.java:745)
Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk]; nested: ElasticsearchException[failure in bulk execution:
[0]: index [.marvel-es-1-2016.12.01], type [node_stats], id [AVi8DKi8ZUbrVfwneDzJ], message [RemoteTransportException[[ElasticSearch-18][10.10.0.112:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[ElasticSearch-18][10.10.0.112:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@5efc8094 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@328300fe[Running, pool size = 8, active threads = 8, queued tasks = 60, completed tasks = 160799220]]];]];
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:106)
... 3 more

2

Please help !!!

3

hi,

No guaranty, but I think it is because there is too many requests sended to the server (maybe a big bulk) and as the server is too "small" it drops some queries.

4

It is a 20 nodes cluster AWS r3.2xlarge, plenty of computing power

5

There is something wrong with threadpool and queue sizes, are your bulk very large ?

Some doc about threadpool : https://www.elastic.co/guide/en/elasticsearch/reference/5.0/modules-threadpool.html

6

That's Marvel, so what does it say about threadpool capacity?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.