Filebeat holding deleted files... please help with 'correct' values to use in close_inactive, etc

Elastic Stack Beats
1

Hoping the community here can help me as google hasn't. I have a case where filebeat is holding deleted files open until the process is recycled. I know there on configs to help address this, but I'm struggling on what the optimum values might be and which too add.

The log I am shipping can grow to 10 MB in size before being rotated, and 3 backups are kept before compressing and moving to an archive folder. For our busiest app, this 10 MB file rotates 2 to 3 times per minute. With this fast of rotation, please help answer (please see input config and ls listing below for reference):

  • What configs should be added (close_timeout, close_inactive, etc) and with what values.
  • Is it wrong to have filebeat just looking at the Trace.txt file vs. all the files in the trace directory?

Note: I do realize these values are not one size fits all, but I can't reproduce in non-prod and I don't have any filebeat background to leverage.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  - /opt/servers/app1/logs/trace/Trace.txt

  tags:
    - trace

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  multiline.pattern: ^(.{3} .{3} \d{1,2} \d{2}:\d{2}:\d{2}:\d{3} .{3} \d{4} \| )

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  multiline.negate: true

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  multiline.match: after

total 44540
-rw-r--r-- 1 oracle dba 10485943 Aug 19 07:27 TraceBkup2.txt
-rw-r--r-- 1 oracle dba 10485890 Aug 19 07:28 TraceBkup0.txt
-rw-r--r-- 1 oracle dba 10485872 Aug 19 07:28 TraceBkup1.txt
drwxr-xr-x 2 oracle dba    32768 Aug 19 07:28 archive
-rw-r--r-- 1 oracle dba  7212771 Aug 19 07:28 Trace.txt
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.