# How to Improve ELK Performance **URL:** \ **Category:** Elasticsearch\ **Created:** [June 14, 2022, 6:10am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092 "2022-06-14T06:10:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [June 14, 2022, 6:10am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/1 "2022-06-14T06:10:09Z")
Hi Team, My flow is, JSON Logs files will be processed from FileBeat \> Logstash \> Elasticsearch. But it skips some data. In thread\_pool It shows below many rejected write requests. ```auto node-1 write 0 0 0 node-2 write 0 0 151424 node-3 write 0 0 573 ``` I can see below 2 errors many times in recent Logstash logs. **Error 1:** ```auto [INFO][logstash.outputs.elasticsearch][inventory] retrying failed action with response code: 429 ({"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of processing of [213657339][indices:data/write/bulk[s][p]]: request: BulkShardRequest [inventory][0]] containing [600] requests, target allocation id: AGnVTdHZSo-QDyap7mq4qg, primary term: 7 on EsThreadPoolExecutor[name = node-2/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@2f18ea8e[Running, pool size = 8, active threads = 8, queued tasks = 2539, completed tasks = 9050422]]"}) ``` **Error 2:** ```auto [INFO][logstash.outputs.elasticsearch][rtv] retrying failed action with response code: 429 ({"type"=>"circuit_breaking_exception", "reason"=>"[parent] Data too large, data for [] would be [8267602870/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8266697320/7.6gb], new bytes reserved: [905550/884.3kb], usages [request=0/0b, fielddata=13447/13.1kb, in_flight_requests=3286973706/3gb, accounting=53142083/50.6mb]", "bytes_wanted"=>8267602870, "bytes_limit"=>8094194073, "durability"=>"TRANSIENT"}) ``` Filebat YML: ```auto scan_frequency: 30m ignore_older: 73h close_inactive: 72h clean_inactive: 74h ``` Logstash pipeline.yml ```auto - pipeline.id: order path.config: "order.conf" queue.type: persisted pipeline.workers: 10 pipeline.batch.size: 1000 pipeline.batch.delay: 5 ``` Elastic configuration: ```auto Version: 7.6.0 (we will upgrade in near future) Nodes: 3 Disk Available: 32.79% || 96.2 GB / 293.4 GB JVM Heap: 51.83% || 12.3 GB / 23.8 GB Indices: 70 Documents: 186,098,201 Disk Usage: 141.4 GB Primary Shards: 78 Replica Shards: 78 ``` Logstash/filebeat server: 3 As per the CPU usage chart, it spikes to 90%-95% for a few seconds, every time filebeat read the data and then back to 1%-10%. We have 30 pipelines running and each configured with different log paths. For each pipeline, a new log file will be pushed every 2 hours. Approx content each file will have is 10000 logs. Please suggest how can I improve ELK performance. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 14, 2022, 6:50am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/2 "2022-06-14T06:50:41Z")
Based on the error messages it looks like you are overloading the cluster and that the Elasticsearch heap is too small for your load. How many indices and shards are you actively indexing into? How many different Elasticsearch output blocks do you have in your config? Are you using time-based indices? Are you indexing immutable documents and allowing Elasticsearch to assign the document ID? What is your average document size? Have you followed [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/tune-for-indexing-speed.html)? What is the specification of the cluster in terms of CPU, memory, heap and type of storage used? ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [June 14, 2022, 8:15am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/3 "2022-06-14T08:15:14Z")
Hi @Christian_Dahlqvist , > [@Christian\_Dahlqvist](#): > > How many indices and shards are you actively indexing into? 15-20 indices. index config is 1 shard 1 replica. > [@Christian\_Dahlqvist](#): > > How many different Elasticsearch output blocks do you have in your config? In total, I have 20 logstash configs. each has 1-2 elasticsearch output blocks. > [@Christian\_Dahlqvist](#): > > Are you using time-based indices? No > [@Christian\_Dahlqvist](#): > > Are you indexing immutable documents and allowing Elasticsearch to assign the document ID? Documents are mutable. DocumentId is an Explicit Id generated using fingerprint pugin in logsatsh. > [@Christian\_Dahlqvist](#): > > What is your average document size? Each document will approx have 20 fields. - 1 KB > [@Christian\_Dahlqvist](#): > > What is the specification of the cluster in terms of CPU, memory, heap and type of storage used? ```auto Nodes: 3 Disk Available: 32.16% - 94.4 GB / 293.4 GB JVM Heap: 53.48% - 12.7 GB / 23.8 GB indices: 70 Documents: 188,730,418 Disk Usage: 146.1 GB Primary Shards: 78 Replica Shards: 78 "thread_pool" : { "write" : { "queue_size" : "200", "size" : "8" } } jvm.options -Xms16g -Xmx16g ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 14, 2022, 8:20am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/4 "2022-06-14T08:20:53Z")
> [@truptir](#): > > In total, I have 20 logstash configs. each has 1-2 Elasticsearch output blocks. Are these defined as independent pipelines similar to the order pipeline shown earlier? If so, do they all have such high number of pipeline workers and batch size? Does each pipeline index into a single index? What type of storage does the cluster have? Local SSDs? If the nodes have 16GB heap, how do you end up with a total of 23.8 GB? Are all the nodes the same size and specification? ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [June 14, 2022, 8:58am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/5 "2022-06-14T08:58:25Z")
> [@Christian\_Dahlqvist](#): > > Are these defined as independent pipelines similar to the order pipeline shown earlier? If so, do they all have such a high number of pipeline workers and batch sizes? Does each pipeline index into a single index? Yes, all pipelines are defined as independent pipelines with the same config. most of all pipeline index into single index but 3-4 pipeline indexes into 2 different indexes based on value. Also, the data volume of each log file is different, for most of it is 1MB-10MB(1000 -10,000 documents) but for 2 log files will be of 1-1.5 GB(10,00,000-20,00,000 documents) . Please suggest how what value I should set to handle this. > [@Christian\_Dahlqvist](#): > > What type of storage does the cluster have? Local SSDs? Yes Local SSD. > [@Christian\_Dahlqvist](#): > > If the nodes have 16GB heap, how do you end up with a total of 23.8 GB? Are all the nodes the same size and specification? my Cluster has 3 modes each having 8GB so total 24GB in the cluster. But in jvm.option config we specified xmx and xms 16GB. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 14, 2022, 9:48am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/6 "2022-06-14T09:48:15Z")
Can you provide the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/cluster-stats.html). Based on error message 2 it seems like your heap may be 8GB and not 16GB. ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [June 14, 2022, 9:55am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/7 "2022-06-14T09:55:00Z")
> [@truptir](#): > > Yes, all pipelines are defined as independent pipelines with the same config. most of all pipeline index into single index but 3-4 pipeline indexes into 2 different indexes based on value. Also, the data volume of each log file is different, for most of it is 1MB-10MB(1000 -10,000 documents) but for 2 log files will be of 1-1.5 GB(10,00,000-20,00,000 documents) . Please suggest what value I should set to in the logstash pipeline to handle this. I have just edited this can u plz suggest if this causing issue. Output of cluster stats ```auto { "_nodes" : { "total" : 3, "successful" : 3, "failed" : 0 }, "cluster_name" : "elasticsearch_vms_prod", "cluster_uuid" : "EKWAEuLWSvqoGGw8kfOoWA", "timestamp" : 1655200466249, "status" : "green", "indices" : { "count" : 70, "shards" : { "total" : 156, "primaries" : 78, "replication" : 1.0, "index" : { "shards" : { "min" : 2, "max" : 10, "avg" : 2.2285714285714286 }, "primaries" : { "min" : 1, "max" : 5, "avg" : 1.1142857142857143 }, "replication" : { "min" : 1.0, "max" : 1.0, "avg" : 1.0 } } }, "docs" : { "count" : 188842771, "deleted" : 24028367 }, "store" : { "size_in_bytes" : 157050946893 }, "fielddata" : { "memory_size_in_bytes" : 87016, "evictions" : 0 }, "query_cache" : { "memory_size_in_bytes" : 1275774824, "total_count" : 168611050, "hit_count" : 5326317, "miss_count" : 163284733, "cache_size" : 62321, "cache_count" : 168149, "evictions" : 105828 }, "completion" : { "size_in_bytes" : 0 }, "segments" : { "count" : 1589, "memory_in_bytes" : 144731958, "terms_memory_in_bytes" : 91180630, "stored_fields_memory_in_bytes" : 47609552, "term_vectors_memory_in_bytes" : 0, "norms_memory_in_bytes" : 1181760, "points_memory_in_bytes" : 0, "doc_values_memory_in_bytes" : 4760016, "index_writer_memory_in_bytes" : 4203544, "version_map_memory_in_bytes" : 0, "fixed_bit_set_memory_in_bytes" : 1592000, "max_unsafe_auto_id_timestamp" : 1655164806472, "file_sizes" : { } } }, "nodes" : { "count" : { "total" : 3, "coordinating_only" : 0, "data" : 3, "ingest" : 3, "master" : 3, "ml" : 3, "voting_only" : 0 }, "versions" : [ "7.6.0" ], "os" : { "available_processors" : 24, "allocated_processors" : 24, "names" : [ { "name" : "Linux", "count" : 3 } ], "pretty_names" : [ { "pretty_name" : "Red Hat Enterprise Linux Server 7.6 (Maipo)", "count" : 3 } ], "mem" : { "total_in_bytes" : 48003846144, "free_in_bytes" : 3718291456, "used_in_bytes" : 44285554688, "free_percent" : 8, "used_percent" : 92 } }, "process" : { "cpu" : { "percent" : 0 }, "open_file_descriptors" : { "min" : 780, "max" : 1059, "avg" : 964 } }, "jvm" : { "max_uptime_in_millis" : 12757830494, "versions" : [ { "version" : "13.0.2", "vm_name" : "OpenJDK 64-Bit Server VM", "vm_version" : "13.0.2+8", "vm_vendor" : "AdoptOpenJDK", "bundled_jdk" : true, "using_bundled_jdk" : true, "count" : 3 } ], "mem" : { "heap_used_in_bytes" : 10073213000, "heap_max_in_bytes" : 25560612864 }, "threads" : 341 }, "fs" : { "total_in_bytes" : 315079139328, "free_in_bytes" : 115267661824, "available_in_bytes" : 100721975296 }, "plugins" : [], "network_types" : { "transport_types" : { "security4" : 3 }, "http_types" : { "security4" : 3 } }, "discovery_types" : { "zen" : 3 }, "packaging_types" : [ { "flavor" : "default", "type" : "tar", "count" : 3 } ], "ingest" : { "number_of_pipelines" : 2, "processor_stats" : { "gsub" : { "count" : 0, "failed" : 0, "current" : 0, "time_in_millis" : 0 }, "script" : { "count" : 0, "failed" : 0, "current" : 0, "time_in_millis" : 0 } } } } } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 14, 2022, 10:10am UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/8 "2022-06-14T10:10:10Z")
Based on that it looks like your nodes each have 16GB RAM and the heap set to 8GB. It seems like you are overloading the cluster, so I would recommend reducing the number of pipeline workers in order to not overload the cluster. Maybe set it to 2 for all of them and see if that helps. If that does not help you may also try reducing the batch size. The default I believe is 125 so 1000 is a significant increase and will lead to more data being in flight and requiring more memory on the nodes. ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [June 23, 2022, 3:07pm UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/9 "2022-06-23T15:07:44Z")
Hi @Christian_Dahlqvist, sure we will reduce the pipelines but I doubt that our elastic and logstash are running on different servers still reducing pipeline workers for logstash will it improove elastic cluster performance? Also for each pipeline, we have set the pipeline.batch.delay: to 5 does it cause any issue on performance? Also when it gives error [logstash.outputs.elasticsearch] "reason"=\>"[parent] Data too large, It indicates logstash heap or elastic heap? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 21, 2022, 3:07pm UTC](https://discuss.elastic.co/t/how-to-improve-elk-performance/307092/10 "2022-07-21T15:07:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.