Events are lost when elasticsearch output is temporary unavailable

Elastic Stack Logstash
4

Hi,

Good to know that the suggestions work.

I think that the storage overhead in LogStash is to be expected:

  1. Elasticsearch compresses data by default (see details here) while LogStash does not store the queue compressed (I think)
  2. LogStash stores metadata from each input/output/filter in each event. This contains for example the source ip for beats input. You can view the metadata for example by writting all events including metadata to a file (see details here: How to access the value in the logstash metadata - #2 by Christian_Dahlqvist).

Best regards
Wolfram