How to clear counter without loss?


(Kelvin Fann) #1

Hi!
I am relatively new to log stash, so forgive my ignorance.

I am trying to count the throughput of logs in log stash for a fixed amount sets of time ('.count'). Naturally, therefore, I have been trying the filter 'metric', with:

  • meter => "message"
  • add_tag => "metric"
  • periodic_flush => true
  • flush_interval => 10
  • clear_interval => 10

The problem with this is that log stash clears before it flushes, which results in no data output. I could have a difference in time between flush and clear, but that opens the room for messages coming in-between the flush and clear time, which would result in a miscount, which I cannot afford.

What I need to be able to do is assign the metric to be flushed and then immediately cleared sequentially, without other messages going through between the flush and clear so I can get an accurate count of the messages sent through in the past X seconds (every X seconds) and pass that through as a separate log. Is that possible?


(system) #2