Handle logs comming only when Logstash is turned on

Elastic Stack Logstash
1

Hi Everyone,
I am new one here. I have got a question regarding to 'file input' plugin and sincedb file. Let me try to describe situation:
Traces are coming all the time to the specific log file. Generally Logstash is turned on,
but sometimes I would like to turn it off. In this case I would like to skip traces, I do not need to take care of them. When I turn Logstash on again I wish to process only new incoming traces.

So my question is: how to handle logs comming only when Logstash is turned on?
Should I manipulate/update sincedb file? Is it the only proper solution?
Maybe there is certain flag to deal with it in this way?
Please correct me if I am wrong, but it seems that default value of start_position ('end') does not fit in this case.

I really appreciate. Any suggestions are welcomed.

2

If the program writing to the log file picks up when the file has been rotated you could rotate the file immediately before Logstash starts. Another option is to throw away events older than Logstash's start time. You can probably find the start time using a ruby filter but a simpler option is probably to run Logstash with an autogenerated configuration file that contains a filter that drops too old events.

3

Thank you. I will try approach as you described: " to throw away events older than Logstash start (...)".

4

I have got another proposition:
In 'file' input plugin I set: start_position => "end".
Each time when I start Logstash I remove ".sincedb_****" file.
At first sight it works perfectly fine.

Maybe it would be helpful for other people.
Btw: Are you aware of any downsides related with this solution?

5

Ah, right. Yes, that's a very clean solution.

In that case you should use the sincedb_path option to pick an exact path for the sincedb file so that you a) don't hardcode the name of the autogenerated file and b) don't have to delete $HOME/.sincedb_*.

1 Like