Filebeat not working with 2 different log files


(Mitesh Jaiswal) #1

Hi,
I am using file-beat with different different log files, but only last one log files is seding data. why it is happening, also please find my configuration over filebeat.yml
#################################################
paths:
- /FirstKannel/opt/kannel/var/log/kannel_access.log
input_type: log
document_type: kannel2-access

paths:
- /var/log/tomcat8/localhost_access_log.*.txt
input_type: log
document_type: Report-Access
#############################################


(Steffen Siering) #2

Can you post your complete config file? paths is an array and should be given only once per prospector. Your sample config does not include the prospectors.


(Mitesh Jaiswal) #3

Thanks for reply,
Please find the below my config file:-

############################# Filebeat ######################################
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    # Each - is a prospector. Below are the prospector specific configurations
    -
      # Paths that should be crawled and fetched. Glob based paths.
      # To fetch all ".log" files from a specific level of subdirectories
      # /var/log/*/*.log can be used.
      # For each file found under this path, a harvester is started.
      # Make sure not file is defined twice as this can lead to unexpected behaviour.
      #paths:
       # - /var/log/*.log
        #- c:\programdata\elasticsearch\logs\*
      paths:
         - /FirstKannel/opt/kannel/var/log/kannel_access.log
      document_type: kannel2-access


      paths:
        - "/var/log/tomcat8/localhost_access_log.*.txt"
      document_type: Report-Access

      # Configure the file encoding for reading files with international characters
      # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
      # Some sample encodings:
      #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
      #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
      #encoding: plain

      # Type of the files. Based on this the way the file is read is decided.
      # The different types cannot be mixed in one prospector
      #
      # Possible options are:
      # * log: Reads every line of the log file (default)
      # * stdin: Reads the standard in
      input_type: log

registry_file: /var/lib/filebeat/registry

output:

### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["10.0.0.21:5088"]
    bulk_max_size: 1024
    # Number of workers per Logstash host.
    #worker: 1

    # Set gzip compression level.
    #compression_level: 3

    # Optional load balance the events between the Logstash hosts
    #loadbalance: true

    # Optional index name. The default index name depends on the each beat.
    # For Packetbeat, the default is set to packetbeat, for Topbeat
    # top topbeat and for Filebeat to filebeat.
    #index: filebeat

    # Optional TLS. By default is off.
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder-local.crt"]

(ruflin) #4

It looks like you try to define two prospectors but only defined one. There is a - missing in front of the second paths:.


(Mitesh Jaiswal) #5

@ruflin, thanks a lot, you are right after adding - , now I am getting both logs in kibana... again thanks...


(Andrew Kroh) #6