I am totally lost with this issue, I spent countless hours trying to figure it out but I came up empty. It seems like somewhere along the process from input into Logstash to the ingestion into Elastic data is being lost. I looked through the Logstash logs but found no errors. I have been running almost an identical config file in a second cluster and the second cluster is working fine and ingesting way more.
I know that I should be seeing at least four times the amount of data then what I see now. I am attaching a screenshot below of the ingestion rate into Elastic. You can see the sharp rises and falls in the graph, and that its not coming in steady. I cant seem to figure out whether its even an Elastic or Logstash issue. Any help would be awesome!