Logstash performance issues

Elastic Stack Logstash
1

Hi ,

I Have a Logstash 7.17 version , i have two logstash servers in my Setup that gets connected to a 3 node ELK Cluster. One Kibana server

We are experiencing less data getting populated in the kibana dashboards. when i look at 15 minutes interval, i see there is very less data coming up or no data coming up for the beats servers.

What might be the issue. i have increased the jvm for my logstash to 16g and the CPU and Memory is in control.

any advise will be helpful

2

It is impossible to know unless you provide more context.

What is the source of the data? What does your logstash configuration looks like? Do you have anything in the logs of Logstash? Do you have anything in the logs of Filebeat?

Also, Logstash is more CPU bound than memory bound, increase the JVM for Logstash may not change anything. What is the CPU configuration of the logstash server? How many cores it have?

3

We have only metric beats and winlog beats sending data to logstash servers. we have around 90 servers that are sending the data from these beats.

input {

beats {

port => 5044

}

}

output {
if [agent][type] == "metricbeat" {
stdout { codec => rubydebug }
elasticsearch {
hosts =>
user => logstash
password => logstash
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
ilm_enabled => "true"
ilm_rollover_alias => "metricbeat"
ilm_pattern => "000001"
}
}
else if [agent][type] == "winlogbeat" {
stdout { codec => rubydebug }
elasticsearch {
hosts =>
user => logstash
password => logstash
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
ilm_enabled => "true"
ilm_rollover_alias => "winlogbeat-7.17.3"
ilm_pattern => "000001"
}
}
}

4

The Logstash server has a Intel Xeon gold 6248 @2.50Ghz(4 core) processors and ram is 32 gb . there is enough space in the disk also.

i have increased the worker nodes to 4 and piepline batch size to 1000 and pipeline batch delay to 300ms

5

your index => is commented out (#). Or are you trying to use datastreams?

6

yes we have created ILM Policies and index templates .

7

Your pipeline is pretty simple, there is no filters that could increase the processing time.

Are you using persistent queues?

Also, most of the time when you have some indexing issues, like lags, the issue is in Elasticsearch, not Logstash.

What are your Elasticsearch configurations? CPU, Memory, Heap and Disk type for example.

8

Hi,
There is no persistent queues. we have a 3 node Elastic Search cluster . every node has Intel Xenon Gold 6248 cpu @2.50 gHZ(4 Core), Memory is 32 gb and heap is 16gb. We have a 1 TB Disk for every server.

9

What is the kind of disk, it is HDD or SSD?

Also, do you have something in both Logstash and Elasticsearch logs?

10

The Disk is HDD . i Checked the logstash plain logs-- there is no errors. while in the cluster logs of ELK also i didnt find any errors.

Do i need to check any other logs in Elasticsearch, which could throw some light on the slowness or indexing issues

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.