File input: ignore_older quick question

Elastic Stack Logstash
1

I just want to check the behaviour of the ignore_older parameter. Logstash 2.3.3.

Logstash has a file input and is set start_position: beginning.
ignore_older set to default of 24h.

  • Logstash has been down for 48h.
  • More than 24h ago a new log was created but it has not been modified in the last 24h.
  • Logstash comes back up. It ignores that log file and does not read from it.
  • The log file has new lines appended.
  • Logstash only starts reading those new log lines and still ignores the older log lines.

Is this all correct? Even though Logstash has start_position: beginning it will only read the new log lines when the file is modified again?

2

Hello,

Logstash will read the new log lines and the older ones.

You have 2 cases:
Case 1:

  1. You created the file while Logstas was down but it is older than 24 hours
  2. Logstash comes back up. It ignores that log file and does not read from it
  3. The log file is modified by adding new lines
  4. Logstash will starts reading the file from the beginning, new lines and older ones will be read

Case 2:

  1. You created the file while Logstas was up so it is older than 48 hours. Meaning that Logstash already read that file.
  2. The file appended to be modified when Logstash was down but the modification appended at least 24 hours before restarting Logstash . Meaning that the last position known for this file in the sincedb file is not the last line of the file.
  3. Logstash comes back up. It ignores that log file and does not read from it
  4. The log file is modified by adding new lines
  5. Logstash will starts reading the file from the last position known (old lines not read yet and new ones)

I don't know if I'm understandable...
I hope this help though.

3

Hi Noémie,

I understood perfectly :slight_smile:

So in summary, all log lines will be processed as long as the file has been modified in the last 24h. All previously ignored lines will be picked up.

Thanks!

4

Exactly!

The only lines that won't be read again are those already read before the crash :slight_smile:
If not read, when modifiying the file, Logstash will just resume from the last know position in the sincedb file for the file that was modified :slight_smile:

1 Like