HI Team,
I'm using below configuration to process my log from my path to elasticsearch,
filebeat.inputs:
- type: log
enabled: false
paths:
#- /var/log/*.log
- C:\Go\unirest\go\unirest\output.json
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 5
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
#index: "qoe_nrt_qoe_attributes"
#setup.template.name: "qoe_nrt_qoe_attributes"
#setup.template.pattern: "qoe_nrt_qoe_attributes"
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
And ouput which i'm getting in the screen ,
C:\Users\ganeshe>D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\filebeat.exe -c D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\filebeat.yml -e -v -d \"*\"
2019-02-18T17:43:35.963+0530 INFO instance/beat.go:544 Home path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64] Config path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64] Data path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\data] Logs path: [D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\logs]
2019-02-18T17:43:35.966+0530 INFO instance/beat.go:551 Beat UUID: bdecc354-9e1b-4c38-b605-09a605798171
2019-02-18T17:43:35.966+0530 INFO [beat] instance/beat.go:768 Beat info {"system_info": {"beat": {"path": {"config": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64", "data": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\data", "home": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64", "logs": "D:\\Ganesh\\ELK\\filebeat-6.4.0-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "bdecc354-9e1b-4c38-b605-09a605798171"}}}
2019-02-18T17:43:35.966+0530 INFO [beat] instance/beat.go:777 Build info {"system_info": {"build": {"commit": "34b4e2cc75fbbee5e7149f3916de72fb8892d070", "libbeat": "6.4.0", "time": "2018-08-17T22:19:27.000Z", "version": "6.4.0"}}}
2019-02-18T17:43:35.966+0530 INFO [beat] instance/beat.go:780 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-02-18T17:43:36.006+0530 INFO [beat] instance/beat.go:784 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-02-14T11:47:19.48+05:30","hostname":"ST21IND1332","ips":["fe80::4472:647a:68f9:9cec/64","5.250.139.202/21","fe80::9035:4ebf:6e1b:b810/64","172.18.251.225/28","fe80::9191:8eea:cea5:d68c/64","10.0.75.1/24","fe80::3d37:49e6:22be:402b/64","192.168.4.147/24","fe80::74e7:3a3c:d5ae:25a6/64","169.254.37.166/16","fe80::9535:bdc6:a3cc:265/64","169.254.2.101/16","fe80::a54d:5e45:4a0b:7b89/64","10.0.200.243/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.590 (WinBuild.160101.0800)","mac_addresses":["00:05:9a:3c:7a:00","02:15:68:25:eb:82","00:15:5d:c8:e8:00","54:e1:ad:0e:96:5c","6a:00:e3:d4:35:8f","5a:00:e3:d4:35:8f","58:00:e3:d4:35:8f"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.590"},"timezone":"IST","timezone_offset_sec":19800,"id":"1b0dcdb1-e75b-453d-8efa-31d567fff8e3"}}}
2019-02-18T17:43:36.006+0530 INFO instance/beat.go:273 Setup Beat: filebeat; Version: 6.4.0
2019-02-18T17:43:36.007+0530 INFO pipeline/module.go:98 Beat name: ST21IND1332
2019-02-18T17:43:36.009+0530 INFO [monitoring] log/log.go:114 Starting metrics logging every 30s
2019-02-18T17:43:36.009+0530 INFO instance/beat.go:367 filebeat start running.
2019-02-18T17:43:36.009+0530 INFO registrar/registrar.go:97 No registry file found under: D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\data\registry. Creating a new registry file.
2019-02-18T17:43:36.072+0530 INFO registrar/registrar.go:134 Loading registrar data from D:\Ganesh\ELK\filebeat-6.4.0-windows-x86_64\data\registry
2019-02-18T17:43:36.072+0530 INFO registrar/registrar.go:141 States Loaded from registrar: 0
2019-02-18T17:43:36.073+0530 WARN beater/filebeat.go:371 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-02-18T17:43:36.073+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-02-18T17:43:36.074+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 0
2019-02-18T17:43:36.074+0530 INFO cfgfile/reload.go:140 Config reloader started
2019-02-18T17:44:06.153+0530 INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":46,"time":{"ms":46}}},"info":{"ephemeral_id":"58408f99-60d6-498c-9c4f-3413518dfe82","uptime":{"ms":30519}},"memstats":{"gc_next":4194304,"memory_alloc":2334200,"memory_total":3856752,"rss":18534400}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":2},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}