Data loss happening somewhere

[Shimon_k]

Hi guys,

I am totally lost with this issue, I spent countless hours trying to figure it out but I came up empty. It seems like somewhere along the process from input into Logstash to the ingestion into Elastic data is being lost. I looked through the Logstash logs but found no errors. I have been running almost an identical config file in a second cluster and the second cluster is working fine and ingesting way more.

I know that I should be seeing at least four times the amount of data then what I see now. I am attaching a screenshot below of the ingestion rate into Elastic. You can see the sharp rises and falls in the graph, and that its not coming in steady. I cant seem to figure out whether its even an Elastic or Logstash issue. Any help would be awesome!

[Shimon_k]

Hi I have found that this is helping me solve a issue with logstash getting clogged up. I am sending a huge amount a data through it, the machine has 128GB of RAM and a 16 core CPU. I found that by raising the LS_Heap size, the number of workers, and the batch size is helping it process this large amount of data.

My question now is, how many workers can I afford to give it? I have plenty of free RAM, but I know its not recommended to exceed the number of CPU cores. Can I go above 16?

Its getting better but still clogged up...