I have updated my machine to filebeat 5 and now when trying to ship my logs I see that the shipping sometimes fails and I don't understand the reason.
The error is the following when enabling debug
2017-06-14T16:30:21+02:00 DBG Run prospector 2017-06-14T16:30:21+02:00 DBG Start next scan 2017-06-14T16:30:21+02:00 DBG Check file for harvesting: /var/log/user.log 2017-06-14T16:30:21+02:00 DBG Update existing file for harvesting: /var/log/user.log, offset: 7595 2017-06-14T16:30:21+02:00 DBG Harvester for file is still running: /var/log/user.log 2017-06-14T16:30:21+02:00 DBG Prospector states cleaned up. Before: 1, After: 1 2017-06-14T16:30:28+02:00 DBG handle error: read tcp IPA->IPB:5044: i/o timeout 2017-06-14T16:30:28+02:00 DBG 0 events out of 11 events sent to logstash. Continue sending 2017-06-14T16:30:28+02:00 DBG close connection 2017-06-14T16:30:28+02:00 DBG closing 2017-06-14T16:30:28+02:00 ERR Failed to publish events caused by: read tcp IPA->IPB:5044: i/o timeout 2017-06-14T16:30:28+02:00 INFO Error publishing events (retrying): read tcp IPA->IPB:5044: i/o timeout 2017-06-14T16:30:28+02:00 DBG close connection 2017-06-14T16:30:28+02:00 DBG send fail 2017-06-14T16:30:30+02:00 DBG connect 2017-06-14T16:30:30+02:00 DBG Try to publish 11 events to logstash with window size 2 2017-06-14T16:30:31+02:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_bytes=1390 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=703 libbeat.logstash.publish.write_errors=1 libbeat.logstash.published_but_not_acked_events=11 2017-06-14T16:30:31+02:00 DBG Run prospector
I have noticed that if I stop syslog daemon and filebeat, start first filebeat and then the syslog, filbeats sends out some logs properly but then happens as the above error.
My filebeat config is
#=========================== Filebeat prospectors ============================= filebeat.prospectors: # Each - is a prospector. Most options can be set at the prospector level, so # you can use different prospectors for various configurations. # Below are the prospector specific configurations. - input_type: log # Paths that should be crawled and fetched. Glob based paths. paths: - /var/log/syslog.log - /var/log/user.log ignore_older: 24h scan_frequency: 10s # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ["^DBG"] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. include_lines: ['TTN-.*$'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #exclude_files: [".gz$"] # Optional additional fields. These field can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 ### Multiline options # Mutiline can be used for log messages spanning multiple lines. This is common # for Java Stack Traces or C-Line Continuation # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ #multiline.pattern: ^\[ # Defines if the pattern set under pattern should be negated or not. Default is false. #multiline.negate: false # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern # that was (not) matched before or after or as long as a pattern is not matched based on negate. # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash #multiline.match: after #================================ General ===================================== # The name of the shipper that publishes the network data. It can be used to group # all the transactions sent by a single shipper in the web interface. #name: # The tags of the shipper are included in their own field with each # transaction published. #tags: ["service-X", "web-tier"] # Optional fields that you can specify to add additional information to the # output. #fields: # env: staging filebeat.registry_file: /var/lib/filebeat/registry #================================ Outputs ===================================== # Configure what outputs to use when sending the data collected by the beat. # Multiple outputs may be used. #-------------------------- Elasticsearch output ------------------------------ #output.elasticsearch: # Array of hosts to connect to. # hosts: ["localhost:9200"] # Optional protocol and basic auth credentials. #protocol: "https" #username: "elastic" #password: "changeme" #----------------------------- Logstash output -------------------------------- output.logstash: # The Logstash hosts hosts: ["IPB:5044"] bulk_max_size: 2048 # Optional SSL. By default is off. # List of root certificates for HTTPS server verifications ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] # Certificate for SSL client authentication #ssl.certificate: "/etc/pki/client/cert.pem" # Client Certificate Key #ssl.key: "/etc/pki/client/cert.key" #================================ Logging ===================================== # Sets log level. The default log level is info. # Available log levels are: critical, error, warning, info, debug logging.level: debug # At debug level, you can selectively enable logging only for some components. # To enable all selectors use ["*"]. Examples of other selectors are "beat", # "publish", "service". #logging.selectors: ["*"] logging.to_files: true logging.to_syslog: false logging.files: path: /var/log/mybeat name: mybeat.log keepfiles: 7 rotateeverybytes: 10485760 # = 10MB