# Tuning the ES performance **URL:** \ **Category:** Elasticsearch\ **Created:** [October 31, 2017, 3:41am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832 "2017-10-31T03:41:56Z")\ **Posts on this page:** 17\ **Page:** 2
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 6, 2017, 9:30am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/21 "2017-11-06T09:30:15Z")
I do not know ntopng, so do not know how this indexes data nor what its performance characteristics are. ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 6, 2017, 9:56am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/22 "2017-11-06T09:56:31Z")
I ask the ntopng engineer, and they told me I should tune my ES java VM to increase ES ingestion speed. But I don't know how to tune it. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 6, 2017, 10:14am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/23 "2017-11-06T10:14:09Z")
Did he tell you what bulk size they use? What kind of storage do you have in your cluster? What does disk I/O and iowait look like during indexing? From what I have seen so far it does not look like Elasticsearch is struggling or being the bottleneck. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 6, 2017, 10:33am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/24 "2017-11-06T10:33:25Z")
I had a quick look at [the ntopng code](https://github.com/ntop/ntopng/blob/dev/src/ElasticSearch.cpp#L141), and it looks to me (it has been a while since I used C++, so I could be reading it wrong...) like they send requests that are no larger than 16 kB in size. If that is the case, it could mean quite small bulk requests (especially if events are not very small), which could be inefficient, especially if indexing is not multi-threaded. ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 6, 2017, 10:42am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/25 "2017-11-06T10:42:07Z")
sorry, may I ask you how to index using multi-threaded? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 6, 2017, 10:49am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/26 "2017-11-06T10:49:26Z")
That depends on the design of the application indexing into Elasticsearch, in this case ntopng. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 6, 2017, 10:51am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/27 "2017-11-06T10:51:03Z")
You might be able to get better performance by updating your indexing template to only have 1 primary shard instead of the default 5 (if you still are using this) and possibly also increasing the refresh interval. ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 7, 2017, 8:20am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/28 "2017-11-07T08:20:44Z")
sorry, How could I update my indexing template \>\< ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 7, 2017, 8:31am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/29 "2017-11-07T08:31:08Z")
this is my setting, and my indies such like ntopng-2017.11.06 Is my setting correct? thank you in advance. PUT /ntopng-\*/\_settings { "index" : { "refresh\_interval" : "5s" } } PUT \_template/template\_1 { "template": "ntopng-\*", "settings": { "number\_of\_shards": 1 } } ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 7, 2017, 8:33am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/30 "2017-11-07T08:33:26Z")
It still drop flow. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2017, 9:01am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/31 "2017-11-07T09:01:20Z")
What does a typical document look like? ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 7, 2017, 9:09am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/32 "2017-11-07T09:09:13Z")
a document. { "\_index": "ntopng-2017.11.06", "\_type": "ntopng", "\_id": "AV-Q0VuKEFis5A4jOVHT", "\_score": 1, "\_source": { "@timestamp": "2017-11-06T10:10:43.0Z", "type": "ntopng", "IN\_SRC\_MAC": "00:00:00:00:00:00", "OUT\_DST\_MAC": "00:00:00:00:00:00", "IPV4\_SRC\_ADDR": "120.127.163.167", "IPV4\_DST\_ADDR": "13.228.12.182", "L4\_SRC\_PORT": 43176, "L4\_DST\_PORT": 443, "PROTOCOL": 6, "L7\_PROTO": 178, "L7\_PROTO\_NAME": "SSL.Amazon", "TCP\_FLAGS": 0, "IN\_PKTS": 18, "IN\_BYTES": 5107, "OUT\_PKTS": 0, "OUT\_BYTES": 0, "FIRST\_SWITCHED": 1509962742, "LAST\_SWITCHED": 1509963043, "json": { "5": "0", "10": "130", "14": "0", "15": "120.127.163.1", "16": "0", "17": "0", "130": "120.127.163.4" }, "CLIENT\_NW\_LATENCY\_MS": 0, "SERVER\_NW\_LATENCY\_MS": 0, "SRC\_IP\_COUNTRY": "TW", "SRC\_IP\_LOCATION": [ 121.496597, 25.0418 ], "DST\_IP\_COUNTRY": "US", "DST\_IP\_LOCATION": [ -122.342201, 47.634399 ], "NTOPNG\_INSTANCE\_NAME": "DESKTOP-L1VSVD3", "INTERFACE": "tcp://127.0.0.1:2055" }, "fields": { "@timestamp": [ 1509963043000 ] } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2017, 9:10am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/33 "2017-11-07T09:10:10Z")
The fact that flows are still dropped does not mean that Elasticsearch is saturated and the bottleneck. It could also be that `ntopng` is indexing into Elasticsearch in an inefficient manner and therefore is not able to keep up with the traffic. In order to index efficiently into Elasticsearch, the application should ideally: - Use persistent connections for the HTTP interface - Use bulk requests of an appropriate size, generally no more than a couple of MB in size - Send multiple bulk indexing requests in parallel Logstash, Beats and our benchmarking tool [Rally](https://github.com/elastic/rally) all do this, so you could use one of these to run a test to see how much your node is able to handle. That would tell you whether Elasticsearch is the bottleneck or not. ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 8, 2017, 7:07am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/34 "2017-11-08T07:07:12Z")
ok, I got it! I think if I can't tune the ntopng indexing manner, the only way I could do is improve the ES performance. this is the reference I found: [https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html) I run the command below on Dev tool in Kibana PUT /ntopng-\*/\_settings { "index" : { "refresh\_interval" : "30s" } } PUT \_template/template\_1 { "template": "elasticsearch", "settings": { "number\_of\_shards": 5 } } and run the rest API GET \_nodes/process?pretty output: { "\_nodes": { "total": 1, "successful": 1, "failed": 0 }, "cluster\_name": "elasticsearch", "nodes": { "Ku2xA7CoTzec-5hieGRJgg": { "name": "Ku2xA7C", "transport\_address": "127.0.0.1:9300", "host": "127.0.0.1", "ip": "127.0.0.1", "version": "5.6.2", "build\_hash": "57e20f3", "roles": [ "master", "data", "ingest" ], "attributes": { "ml.max\_open\_jobs": "10", "ml.enabled": "true" }, "process": { "refresh\_interval\_in\_millis": 1000, "id": 8528, "mlockall": false } } } } the refresh\_interval\_in\_millis didn't change. Is the command I type correct? Or there's ES config I can tune it . thank you in advance! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 8, 2017, 7:28am UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/35 "2017-11-08T07:28:24Z")
That is the right guide, but I would expect any gains from this to be small compared to improving the indexing process. I looked at the ntopng repository, and it seems like you have added feedback to [this issue](https://github.com/ntop/ntopng/issues/1441). Instead of writing directly to Elasticsearch, you could try sending the data through Logstash using their [Logstash integration](https://www.ntop.org/ntopng/filling-the-pipe-exporting-ntopng-flows-to-logstash/), as this may have a more efficient implementation (I have not tested it or looked at the source code). Logstash is very flexible and can be tuned extensively, although default settings are quite efficient and sufficient for many use cases. ---
**Author:** ![f26227279](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/f26227279/32/21296_2.png) [@f26227279](https://discuss.elastic.co/u/f26227279)\ **Post date:** [November 8, 2017, 4:16pm UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/36 "2017-11-08T16:16:07Z")
That sounds great! I would try it in these days. Thank you : ) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 6, 2017, 4:16pm UTC](https://discuss.elastic.co/t/tuning-the-es-performance/105832/37 "2017-12-06T16:16:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/tuning-the-es-performance/105832.md?page=1)