Filebeat fail to pick log event from the respective path

(Ganesh) #1

HI Team,
I'm using below configuration to process my log from my path to elasticsearch,

filebeat.inputs:

- type: log

  enabled: false

  paths:
    #- /var/log/*.log
    - C:\Go\unirest\go\unirest\output.json

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 5
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
  #index: "qoe_nrt_qoe_attributes"
#setup.template.name: "qoe_nrt_qoe_attributes"
#setup.template.pattern: "qoe_nrt_qoe_attributes"

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

And ouput which i'm getting in the screen ,

C:\Users\ganeshe>D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\filebeat.exe -c D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\filebeat.yml -e -v -d \"*\"
2019-02-18T17:43:35.963+0530    INFO    instance/beat.go:544    Home path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64] Config path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64] Data path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\data] Logs path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\logs]
2019-02-18T17:43:35.966+0530    INFO    instance/beat.go:551    Beat UUID: bdecc354-9e1b-4c38-b605-09a605798171
2019-02-18T17:43:35.966+0530    INFO    [beat]  instance/beat.go:768    Beat info       {"system_info": {"beat": {"path": {"config": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64", "data": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\data", "home": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64", "logs": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "bdecc354-9e1b-4c38-b605-09a605798171"}}}
2019-02-18T17:43:35.966+0530    INFO    [beat]  instance/beat.go:777    Build info      {"system_info": {"build": {"commit": "34b4e2cc75fbbee5e7149f3916de72fb8892d070", "libbeat": "6.4.0", "time": "2018-08-17T22:19:27.000Z", "version": "6.4.0"}}}
2019-02-18T17:43:35.966+0530    INFO    [beat]  instance/beat.go:780    Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-02-18T17:43:36.006+0530    INFO    [beat]  instance/beat.go:784    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-02-14T11:47:19.48+05:30","hostname":"ST21IND1332","ips":["fe80::4472:647a:68f9:9cec/64","5.250.139.202/21","fe80::9035:4ebf:6e1b:b810/64","172.18.251.225/28","fe80::9191:8eea:cea5:d68c/64","10.0.75.1/24","fe80::3d37:49e6:22be:402b/64","192.168.4.147/24","fe80::74e7:3a3c:d5ae:25a6/64","169.254.37.166/16","fe80::9535:bdc6:a3cc:265/64","169.254.2.101/16","fe80::a54d:5e45:4a0b:7b89/64","10.0.200.243/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.590 (WinBuild.160101.0800)","mac_addresses":["00:05:9a:3c:7a:00","02:15:68:25:eb:82","00:15:5d:c8:e8:00","54:e1:ad:0e:96:5c","6a:00:e3:d4:35:8f","5a:00:e3:d4:35:8f","58:00:e3:d4:35:8f"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.590"},"timezone":"IST","timezone_offset_sec":19800,"id":"1b0dcdb1-e75b-453d-8efa-31d567fff8e3"}}}
2019-02-18T17:43:36.006+0530    INFO    instance/beat.go:273    Setup Beat: filebeat; Version: 6.4.0
2019-02-18T17:43:36.007+0530    INFO    pipeline/module.go:98   Beat name: ST21IND1332
2019-02-18T17:43:36.009+0530    INFO    [monitoring]    log/log.go:114  Starting metrics logging every 30s
2019-02-18T17:43:36.009+0530    INFO    instance/beat.go:367    filebeat start running.
2019-02-18T17:43:36.009+0530    INFO    registrar/registrar.go:97       No registry file found under: D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\data\registry. Creating a new registry file.
2019-02-18T17:43:36.072+0530    INFO    registrar/registrar.go:134      Loading registrar data from D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\data\registry
2019-02-18T17:43:36.072+0530    INFO    registrar/registrar.go:141      States Loaded from registrar: 0
2019-02-18T17:43:36.073+0530    WARN    beater/filebeat.go:371  Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-02-18T17:43:36.073+0530    INFO    crawler/crawler.go:72   Loading Inputs: 1
2019-02-18T17:43:36.074+0530    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 0
2019-02-18T17:43:36.074+0530    INFO    cfgfile/reload.go:140   Config reloader started
2019-02-18T17:44:06.153+0530    INFO    [monitoring]    log/log.go:141  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":46,"time":{"ms":46}}},"info":{"ephemeral_id":"58408f99-60d6-498c-9c4f-3413518dfe82","uptime":{"ms":30519}},"memstats":{"gc_next":4194304,"memory_alloc":2334200,"memory_total":3856752,"rss":18534400}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":2},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}
(Steffen Siering) #2

Can you try to set enabled: true in your input configuration?

1 Like
(Ganesh) #3

Sorry for late reply and thanks for your response. It works :slight_smile:

(system) closed #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.