Tuning the ES performance

If nProbe is sending to Elasticsearch directly you will likely need to ask them.

Is Elasticsearch overloaded? Are you using the Monitoring functionality to see what is happening?

now seems like ES overload.
No, how to monitor it ?

https://www.elastic.co/guide/en/x-pack/5.6/xpack-monitoring.html :slight_smile:

When I install x-pack in ES it show the error:
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
but I have install the java jdk to the environment variable.

image

Does you collection pipeline use bulk requests to index into Elasticsearch? What does disk I/O and iowait look like on the Elasticsearch host?

Continuing the discussion from this parallel thread: How to tune elasticsearch performance

sorry, How could I know whether I use bulk requests to index in ES.
disk I/O and IO wait do you mean this :

yes I use /_bulk

What is the size of your bulk requests? How large are your documents?

sorry, where can I find the information.

That will be decided by the application ingesting data into Elasticsearch, so you will need to look there. You may also be able to find out by looking at the network traffic.

Does the ES can tune the performance or the only way is to improve my device (such as CPU or RAM)

Indexing individual documents generally results in dramatically lower indexing throughput compared to using bulk requests, so I would recommend ensuring that you are using bulk requests of an appropriate size before starting to try and tune Elasticsearch.

yes, I thought I am using bulk requests
this is the Application command to pass the flow to ES:
ntopng /c -F "es;ntopng;ntopng-%Y.%m.%d;http://192.168.0.157:9200/_bulk;"

I do not know ntopng, so do not know how this indexes data nor what its performance characteristics are.

I ask the ntopng engineer, and they told me I should tune my ES java VM to increase ES ingestion speed.
But I don't know how to tune it.

Did he tell you what bulk size they use?

What kind of storage do you have in your cluster? What does disk I/O and iowait look like during indexing?

From what I have seen so far it does not look like Elasticsearch is struggling or being the bottleneck.

I had a quick look at the ntopng code, and it looks to me (it has been a while since I used C++, so I could be reading it wrong...) like they send requests that are no larger than 16 kB in size. If that is the case, it could mean quite small bulk requests (especially if events are not very small), which could be inefficient, especially if indexing is not multi-threaded.

sorry, may I ask you how to index using multi-threaded?