Filebeat not closing opened files,consuming around 60GB on Production systems

Elastic Stack Beats
1

Hello,

We have filebeat installed on the production system. I see disk getting full after every 3-4 days. For the temporary purpose, we are restarting filebeat and everything comes back to normal.

My understanding is filebeat is not closing the opened files. How to get rid of this permanently?

Please, any help appreciated.

Thanks
Nitin

2

can you share your filebeat config file? Sounds like you've enabled publish_async + load balance to logstash?

3

Thanks @steffens Here is our configuration file

################### Filebeat Configuration Example #########################

############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
      paths:
        - /var/log/messages
        - /var/log/secure
      document_type: colour-syslog
    -
      paths:
        - /var/log/cron
      document_type: colour-cron
    -
      paths:
        - /var/log/yum.log
      document_type: colour-yum
    -
      paths:
        - /var/log/rabbitmq/rabbit@blue2.log
      document_type: blue2-rabbitmq
      multiline:
          pattern: '^\='
          negate: true
          match: after
    -
      paths:
        - /var/log/colour/green.log
        - /var/log/colour/pink.log
      document_type: colour-dbus
      multiline:
          pattern: '^\['
          negate: true
          match: after

      #encoding: plain
      input_type: log
      # exclude_lines: ["^DBG"]
      # include_lines: ["^ERR", "^WARN"]
      # exclude_files: [".gz$"]
      #fields:
      #  level: debug
      #  review: 1
      #fields_under_root: false
      #ignore_older: 0
      #close_older: 1h
      #document_type: log
      #scan_frequency: 10s
      #harvester_buffer_size: 16384
      #max_bytes: 10485760
      #multiline:
        #pattern: ^\[
        #negate: false
        #match: after
        #max_lines: 500
        #timeout: 5s
      #tail_files: false
      #backoff: 1s
      #max_backoff: 10s
      #backoff_factor: 2
      #force_close_files: false

    # Additional prospector
    #-
      # Configuration to use stdin input
      #input_type: stdin

  #spool_size: 2048


  #publish_async: false


  #idle_timeout: 5s

  registry_file: /var/lib/filebeat/registry

  
  #config_dir:

###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

output:

  ### Elasticsearch as output
   #elasticsearch:
   
  #  hosts: ["localhost:9200"]
    #username: "admin"
    #password: "s3cr3t"
    #worker: 1
    #index: "filebeat"

    #template:
      #name: "filebeat"
      #path: "filebeat.template.json"
      #overwrite: false
    #path: "/elasticsearch"
    #proxy_url: http://proxy:3128
    #max_retries: 3
    #bulk_max_size: 50
    #timeout: 90
    #flush_interval: 1
    #save_topology: false
    #topology_expire: 15
    #tls:
      #certificate_authorities: ["/etc/pki/root/ca.pem"]
      #certificate: "/etc/pki/client/cert.pem"
      #certificate_key: "/etc/pki/client/cert.key"
      #insecure: true
      #cipher_suites: []
      #curve_types: []
      #min_version: 1.0
      #max_version: 1.2

  logstash:

    hosts: ["10.204.218.14:5044","10.204.218.15:5044","10.204.218.16:5044"]
    worker: 2
    #bulk_max_size: 2048
    #compression_level: 3
    loadbalance: true
    #index: filebeat
    #tls:
      #certificate_authorities: ["/etc/pki/root/ca.pem"]
      #certificate: "/etc/pki/client/cert.pem"
      #certificate_key: "/etc/pki/client/cert.key"
      #insecure: true
      #cipher_suites: []
      #curve_types: []


  #file:
    #path: "/tmp/filebeat"
    #filename: filebeat
    #rotate_every_kb: 10000
    #number_of_files: 7


  # console:

    #pretty: false


############################# Shipper #########################################

shipper:
  #name:
  #tags: ["service-X", "web-tier"]
  #ignore_outgoing: true
  #refresh_topology_freq: 10
  #topology_expire: 15
  #queue_size: 1000

  #geoip:
    #paths:
    #  - "/usr/share/GeoIP/GeoLiteCity.dat"
    #  - "/usr/local/var/GeoIP/GeoLiteCity.dat"


############################# Logging #########################################

logging:

  #to_syslog: true
   #to_files: false

   files:
    #path: /var/log/mybeat
    #name: mybeat
    rotateeverybytes: 10485760 # = 10MB
    #keepfiles: 7
  #selectors: [ ]
  #level: error
4

Which filebeat version are you using? Is your output always available or do you see some output errors in the filebeat logs?

5

@ruflin i am using filebeat 1.3.1

Is your output always available or do you see some output errors in the filebeat logs?

I don't get any output errors.

6

There are some known issues in the 1.x release which can lead to race conditions and keeping some files open. Can you update to the most recent 5.x release? This should fix the problem. The release is backward compatible.

7

Thanks @ruflin ...we did that....Also we are doing this in production..can you please tell what should be the proper configuration for Closing files after reading...

In the newer version (5.x) i see a lot of options...Our application is rotating files using log4j and also files will be renamed from filename.log to filename.log.1. Could you please tell what will be the proper configuration to avoid this kind of issue ??

Thanks
Nitin

8

You should be just fine with the standard config options except in case you have very heavy file rotation. How often are your files rotated and how long does it take until a rotated file is removed?

9

File gets rotated/renamed when it reaches 1.1 GB..thats it..and there are 5 files like this
filename.log --> filename.log.1 --> filename.log.2 --> filename.log.3 --> filename.log.4 --> filename.log.5
The 5th file gets automatically deleted after we get a new file.
Also, we have 3 different types of files like "filename.log".

It has some really heavy logs, the file gets to its limit within a day or two.

10

The above sounds like a very standard environment with not too many log entries. The default config options should work well for you. Let me know if you hit any issues.

11

I tried using close_renamed field ...but the file is still open in lsof output. But when i restart filebeat it vanishes. So do i have to restart filebeat at regular intervals ??

#------------------------------ Log prospector --------------------------------
- input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  # To fetch all ".log" files from a specific level of subdirectories
  # /var/log/*/*.log can be used.
  # For each file found under this path, a harvester is started.
  # Make sure not file is defined twice as this can lead to unexpected behaviour.
  paths:
    - /var/log/elk/abc.log
  document_type: digit
    #- c:\programdata\elasticsearch\logs\*

  # Configure the file encoding for reading files with international characters
  # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
  # Some sample encodings:
  #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
  #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
  #encoding: plain


  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list. The include_lines is called before
  # exclude_lines. By default, no lines are dropped.
  #exclude_lines: ["^DBG"]

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list. The include_lines is called before
  # exclude_lines. By default, all the lines are exported.
  #include_lines: ["^ERR", "^WARN"]

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: [".gz$"]

  # Optional additional fields. These field can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

  # Set to true to store the additional fields as top level fields instead
  # of under the "fields" sub-dictionary. In case of name conflicts with the
  # fields added by Filebeat itself, the custom fields overwrite the default
  # fields.
  #fields_under_root: false

  # Ignore files which were modified more then the defined timespan in the past.
  # ignore_older is disabled by default, so no files are ignored by setting it to 0.
  # Time strings like 2h (2 hours), 5m (5 minutes) can be used.
  #ignore_older: 0

  # Type to be published in the 'type' field. For Elasticsearch output,
  # the type defines the document type these entries should be stored
  # in. Default: log
  #document_type: log

  # How often the prospector checks for new files in the paths that are specified
  # for harvesting. Specify 1s to scan the directory as frequently as possible
  # without causing Filebeat to scan too frequently. Default: 10s.
  #scan_frequency: 10s

  # Defines the buffer size every harvester uses when fetching the file
  #harvester_buffer_size: 16384

  # Maximum number of bytes a single log event can have
  # All bytes after max_bytes are discarded and not sent. The default is 10MB.
  # This is especially useful for multiline log messages which can get large.
  #max_bytes: 10485760

  ### JSON configuration

  # Decode JSON options. Enable this if your logs are structured in JSON.
  # JSON key on which to apply the line filtering and multiline settings. This key
  # must be top level and its value must be string, otherwise it is ignored. If
  # no text key is defined, the line filtering and multiline features cannot be used.
  #json.message_key:

  # By default, the decoded JSON is placed under a "json" key in the output document.
  # If you enable this setting, the keys are copied top level in the output document.
  #json.keys_under_root: false

  # If keys_under_root and this setting are enabled, then the values from the decoded
  # JSON object overwrite the fields that Filebeat normally adds (type, source, offset, etc.)
  # in case of conflicts.
  #json.overwrite_keys: false

  # If this setting is enabled, Filebeat adds a "json_error" key in case of JSON
  # unmarshaling errors or when a text key is defined in the configuration but cannot
  # be used.
  #json.add_error_key: false

  multiline.pattern: ^(201[6-9]{1})
  multiline.negate: true
  multiline.match: after
  #multiline.max_lines: 500
  #multiline.timeout: 5s
  #tail_files: false
  #symlinks: false
  #backoff: 1s
  #max_backoff: 10s
  #backoff_factor: 2
  #harvester_limit: 0
  #close_inactive: 5m
  #close_renamed: true
  #close_removed: true
  #close_eof: false
  ### State options
  #clean_inactive: 0
  #clean_removed: true
  #close_timeout: 0

Thanks

12

It could be that in your case it closes the file on rename and reopens it afterward again because it didn't finish. I would recommend you to keep close_inactive set to 5m. This will close the file 5 minutes after it finished reading. In your case, that should work pretty well. If a file is not closed, best check your logs to see why. Normally if a file cannot be closed, a message should be logged.

No need to restart filebeat.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.