Tuning the ES performance

Elastic Stack Elasticsearch
1

I am using ES to ingest flow data from other application, but I found that the speed isn't high enough, so sometimes it would drop flow.
How can I change the setting of ES performance ?
I have seen the elasticsearch.yml, jvm.options and log4j2.properties, but there seems no setting about ES performance.

Thanks a lot.

1 Like
2

The speed of which parts exactly?

3

the speed of ES receive data from other application.

4

How are you indexing into Elasticsearch? What version are you on? What hardware? Are you using bulk?

5

I connect ES with nProbe to collect flow data.
version 5.6
2 core CPU, 8G RAM
image
receive speed about : [362.6 Flows/s]
could I speed it up?
what is bulk ><

6

If nProbe is sending to Elasticsearch directly you will likely need to ask them.

Is Elasticsearch overloaded? Are you using the Monitoring functionality to see what is happening?

7

now seems like ES overload.
No, how to monitor it ?

8

https://www.elastic.co/guide/en/x-pack/5.6/xpack-monitoring.html :slight_smile:

9

When I install x-pack in ES it show the error:
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME
but I have install the java jdk to the environment variable.

10

image

11

Does you collection pipeline use bulk requests to index into Elasticsearch? What does disk I/O and iowait look like on the Elasticsearch host?

Continuing the discussion from this parallel thread: How to tune elasticsearch performance

12

sorry, How could I know whether I use bulk requests to index in ES.
disk I/O and IO wait do you mean this :

image
image.png1217×550 53.9 KB

13

image
image.png1218×162 14 KB

14

yes I use /_bulk

15

What is the size of your bulk requests? How large are your documents?

16

sorry, where can I find the information.

17

That will be decided by the application ingesting data into Elasticsearch, so you will need to look there. You may also be able to find out by looking at the network traffic.

18

Does the ES can tune the performance or the only way is to improve my device (such as CPU or RAM)

19

Indexing individual documents generally results in dramatically lower indexing throughput compared to using bulk requests, so I would recommend ensuring that you are using bulk requests of an appropriate size before starting to try and tune Elasticsearch.

20

yes, I thought I am using bulk requests
this is the Application command to pass the flow to ES:
ntopng /c -F "es;ntopng;ntopng-%Y.%m.%d;http://192.168.0.157:9200/_bulk;"