Filebeat keeps files open forever

ronny.lerch · October 12, 2016, 11:45am

I would like to join the discussion. I'm following and applying changes in "Filebeat keeps open files forever" topic the past week.
Sadly I still experience the problem:
filebeat 31 root 1r REG 253,16 10486094 4751403 /opt/xxx/logs/supervisor.out.log.10 (deleted)
.
.
.

Currently I'm running filebeat version 6.0.0-alpha1 with following configuration:

################### Filebeat Configuration Example #########################

############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations

      # Type of the files. Based on this the way the file is read is decided.
      # The different types cannot be mixed in one prospector
      #
      # Possible options are:
      # * log: Reads every line of the log file (default)
      # * stdin: Reads the standard in
    - input_type: log

      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log/*/*.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      paths:
        - /opt/decoder/logs/*.log
        #- c:\programdata\elasticsearch\logs\*

      close_older: 1m
      ignore_older: 1m
      scan_frequency: 10s
      force_close_files: true
      # Configure the file encoding for reading files with international characters
      # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
      # Some sample encodings:
      #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
      #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
      #encoding: plain


###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.
output:

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["localhost:5044"]

############################# Shipper #########################################

shipper:
  # The name of the shipper that publishes the network data. It can be used to group
  # all the transactions sent by a single shipper in the web interface.
  # If this options is not defined, the hostname is used.
  #name:


############################# Logging #########################################

# There are three options for the log output: syslog, file, stderr.
# Under Windows systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
logging:

  # Send all logging output to syslog. On Windows default is false, otherwise
  # default is true.
  #to_syslog: true

  # Write all logging output to files. Beats automatically rotate files if rotateeverybytes
  # limit is reached.
  #to_files: false

  # To enable logging to files, to_files option has to be set to true
  files:
    # The directory where the log files will written to.
    #path: /var/log/mybeat

    # The name of the files where the logs are written to.
    #name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7

ruflin · October 12, 2016, 11:52am

Any chance to share the log files from filebeat for further analysis?

Note: I edited your post and added ticks around the config file to make it readable.

ronny.lerch · October 12, 2016, 12:16pm

Thank you for your fast answer.

I would like to share additional information which help solve the problem.
The /var/log directory does not include any filebeat log.

Can you direct me to a directory where I should find the filebeat log?

ruflin · October 13, 2016, 6:49am

How did you install and run filebeat? You can define the log directory by uncommenting path under logging.files.

I'm a little bit surprised that you have mybeat in the logging part of the config file. This was the case in an older snapshot build but was fixed quite some time ago. Did you use an older config file? From when is your snapshot build?

warkolm · October 13, 2016, 7:55am

Maybe you mean 5.0.0 alpha 1?

ronny.lerch · October 13, 2016, 2:04pm

/usr/bin/filebeat --version
filebeat version 6.0.0-alpha1 (amd64), libbeat 6.0.0-alpha1

ronny.lerch · October 13, 2016, 2:05pm

I compiled filebeat from the github source and my previous configuration file from filebeat 1.3.1.

ronny.lerch · October 14, 2016, 1:05pm

Please let me know how if there is a solution to fix the filebeat issue.

ruflin · October 16, 2016, 1:32pm

Did you manage to get some log files by defining the path? Do you know from which commit you built it? Alternative you can use our snapshot builds: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/ What is your OS?

ronny.lerch · October 18, 2016, 12:10pm

Good morning,

I enabled log file generation.
Current OS is: Linux version 3.10.0-327.22.2.el7.x86_64
Nightly filebeat version: filebeat-6.0.0-alpha1-SNAPSHOT-amd64.deb
All executes run in a docker container.

Please find log file below:

2016-10-17T18:08:07Z INFO Metrics logging every 30s
2016-10-17T18:08:07Z INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2016-10-17T18:08:07Z INFO Setup Beat: filebeat; Version: 6.0.0-alpha1
2016-10-17T18:08:07Z INFO Max Retries set to: 3
2016-10-17T18:08:07Z INFO Activated logstash as output plugin.
2016-10-17T18:08:07Z INFO Publisher name: 6b47993cb070
2016-10-17T18:08:07Z INFO Flush Interval set to: 1s
2016-10-17T18:08:07Z INFO Max Bulk Size set to: 2048
2016-10-17T18:08:07Z INFO filebeat start running.
2016-10-17T18:08:07Z INFO No registry file found under: /var/lib/filebeat/registry. Creating a new registry file.
2016-10-17T18:08:07Z INFO Loading registrar data from /var/lib/filebeat/registry
2016-10-17T18:08:07Z INFO States Loaded from registrar: 0
2016-10-17T18:08:07Z INFO Loading Prospectors: 1
2016-10-17T18:08:07Z INFO Load previous states from registry into memory
2016-10-17T18:08:07Z INFO Previous states loaded: 0
2016-10-17T18:08:07Z WARN DEPRECATED: force_close_files was set to true. Use close_removed + close_rename
2016-10-17T18:08:07Z WARN DEPRECATED: close_older is deprecated. Use close_inactive
2016-10-17T18:08:07Z INFO Loading Prospectors completed. Number of prospectors: 1
2016-10-17T18:08:07Z INFO All prospectors are initialised and running with 0 states to persist
2016-10-17T18:08:07Z INFO Starting Registrar
2016-10-17T18:08:07Z INFO Start sending events to output
2016-10-17T18:08:07Z INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016-10-17T18:08:07Z INFO Starting prospector of type: log
2016-10-17T18:08:07Z WARN DEPRECATED: force_close_files was set to true. Use close_removed + close_rename
2016-10-17T18:08:07Z WARN DEPRECATED: close_older is deprecated. Use close_inactive
2016-10-17T18:08:07Z INFO Harvester started for file: /opt/decoder/logs/supervisor.out.log
2016-10-17T18:08:07Z ERR Connecting error publishing events (retrying): dial tcp [::1]:5044: getsockopt: connection refused

ronny.lerch · October 18, 2016, 12:11pm

Following ( your file upload only allows images) :

2016-10-17T18:08:15Z ERR Failed to publish events caused by: EOF
2016-10-17T18:08:15Z INFO Error publishing events (retrying): EOF
2016-10-17T18:08:21Z ERR Failed to publish events caused by: EOF
2016-10-17T18:08:21Z INFO Error publishing events (retrying): EOF
2016-10-17T18:08:28Z ERR Failed to publish events caused by: EOF
2016-10-17T18:08:28Z INFO Error publishing events (retrying): EOF
2016-10-17T18:08:37Z ERR Failed to publish events caused by: EOF
2016-10-17T18:08:37Z INFO Error publishing events (retrying): EOF
2016-10-17T18:08:37Z INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=2044 filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=4 libbeat.logstash.publish.read_errors=4 libbeat.logstash.publish.write_bytes=85412 libbeat.logstash.published_but_not_acked_events=5032 registrar.writes=1 filebeat.harvester.open_files=1 libbeat.logstash.publish.read_bytes=294 libbeat.logstash.published_and_acked_events=786 filebeat.harvester.running=1
2016-10-17T18:08:50Z ERR Failed to publish events caused by: EOF
2016-10-17T18:08:50Z INFO Error publishing events (retrying): EOF
2016-10-17T18:09:07Z INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=3656
2016-10-17T18:09:36Z ERR Failed to publish events caused by: read tcp 127.0.0.1:40950->127.0.0.1:5044: i/o timeout
2016-10-17T18:09:36Z INFO Error publishing events (retrying): read tcp 127.0.0.1:40950->127.0.0.1:5044: i/o timeout
2016-10-17T18:09:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.publish.read_errors=1
2016-10-17T18:10:07Z INFO No non-zero metrics in the last 30s
2016-10-17T18:10:13Z ERR Failed to publish events caused by: EOF
2016-10-17T18:10:13Z INFO Error publishing events (retrying): EOF
2016-10-17T18:10:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=1128 libbeat.logstash.published_but_not_acked_events=1258
2016-10-17T18:10:57Z WARN DEPRECATED: force_close_files was set to true. Use close_removed + close_rename
2016-10-17T18:10:57Z WARN DEPRECATED: close_older is deprecated. Use close_inactive
2016-10-17T18:11:07Z INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=1
2016-10-17T18:11:18Z ERR Failed to publish events caused by: EOF
2016-10-17T18:11:18Z INFO Error publishing events (retrying): EOF
2016-10-17T18:11:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=1024 libbeat.logstash.publish.read_errors=1 libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.call_count.PublishEvents=1
2016-10-17T18:12:07Z INFO No non-zero metrics in the last 30s
2016-10-17T18:12:23Z ERR Failed to publish events caused by: EOF
2016-10-17T18:12:23Z INFO Error publishing events (retrying): EOF
2016-10-17T18:12:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.publish.write_bytes=285 libbeat.logstash.publish.read_errors=1 libbeat.logstash.call_count.PublishEvents=1
2016-10-17T18:13:07Z INFO No non-zero metrics in the last 30s
2016-10-17T18:13:28Z ERR Failed to publish events caused by: EOF
2016-10-17T18:13:28Z INFO Error publishing events (retrying): EOF
2016-10-17T18:13:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=286 libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1
2016-10-17T18:14:07Z INFO No non-zero metrics in the last 30s
2016-10-17T18:14:33Z ERR Failed to publish events caused by: EOF
2016-10-17T18:14:33Z INFO Error publishing events (retrying): EOF
2016-10-17T18:14:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.publish.write_bytes=286 libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1
2016-10-17T18:15:07Z INFO No non-zero metrics in the last 30s
2016-10-17T18:15:37Z INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.write_bytes=286
2016-10-17T18:15:38Z ERR Failed to publish events caused by: EOF
2016-10-17T18:15:38Z INFO Error publishing events (retrying): EOF
2016-10-17T18:16:07Z INFO Non-zero metrics in the last 30s: libbeat.logstash.published_but_not_acked_events=1258 libbeat.logstash.publish.read_errors=1

ruflin · October 24, 2016, 6:50am

Your log files show that the there are issues sending your events. In case events cannot be sent, the file will be kept open.

For larger log files instead of posting them here, you can post them into a gist a link it here.

ronny.lerch · October 27, 2016, 1:16pm

Hi,

I will look into your suggestion.
Thank you for your feedback.

system · November 2, 2016, 11:45am

This topic was automatically closed after 21 days. New replies are no longer allowed.