Filebeat 6.0 not loading the data file

[mussa572 (Musa)]

Hi

I have recently downloaded a filbeat 6.0 and trying to load the json file . However filebeat Starts Successfully but do not load the json file to send data .

Is there any extra setting I need to enable in order to access the file .

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
  - /opt/uploaddata/*.json

2017/11/23 15:47:18.031971 crawler.go:78: INFO Loading and starting Prospectors completed. Enabled prospectors: 0
2017/11/23 15:47:18.032040 reload.go:128: INFO Config reloader started
2017/11/23 15:47:18.032069 registrar.go:150: INFO Starting Registrar
2017/11/23 15:47:18.032086 reload.go:128: INFO Config reloader started
2017/11/23 15:47:18.032111 reload.go:152: DBG Scan for new config files
2017/11/23 15:47:18.032131 reload.go:171: DBG Number of module configs found: 0
2017/11/23 15:47:18.032133 reload.go:152: DBG Scan for new config files
2017/11/23 15:47:18.032140 reload.go:220: INFO Loading of config files completed.
2017/11/23 15:47:18.032194 reload.go:171: DBG Number of module configs found: 0
2017/11/23 15:47:18.032205 reload.go:220: INFO Loading of config files completed.
2017/11/23 15:47:48.031236 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=2957560 beat.memstats.memory_total=2957560 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.config.reloads=2 libbeat.output.type=logstash libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=0
2017/11/23 15:48:18.031451 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=2972216 beat.memstats.memory_total=2972216 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=0
2017/11/23 15:48:48.031181 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=2979992 beat.memstats.memory_total=2979992 filebeat.harvester.open_files=0 filebeat.harvester.running=0 libbeat.config.module.running=0 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=0 registrar.states.current=0

[pierhugues (Pier-Hugues Pellerin)]

@mussa572 I've looked at your configuration I think you are missing a space before the path, it should look like the following:

 paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - /opt/uploaddata/*.json

If this change doesn't work for you can you include the output of the following command, this will give an idea of what configuration filebeat is currently reading from the yaml.

/filebeat -v -e -d "config"

Thanks

[mussa572 (Musa)]

Hi Pierhugues,

Thanks for the reply I have updated the missing space and still not working . Seeing error below message . Just a bit if history behind the installation . I was using 5.4 and installed using the RPM . I have downloaded a copy of Tar file and unzip in one of my home folder and trying to ingest .

Could there be a issue with permission ?

filebeat2017/11/24 10:11:06.970508 config.go:276: DBG load config file 'filebeat.yml' =>
missing field accessing 'path' accessing 'filebeat' (source:'filebeat.yml')
filebeat2017/11/24 10:11:06.970869 config.go:276: DBG Complete configuration loaded:
{
"filebeat": {
"config": {
"modules": {
"path": "/home/ELK/filebeat/modules.d/.yml",
"reload": {
"enabled": false
}
},
"prospectors": [
{
"enabled": true,
"paths": [
"/opt/uploaddata/.json"
],
"type": "log"
}
]
}
},
"output": {
"logstash": {
"hosts": [
"xxxxx"
]
}
},
"path": {
"config": "/home/ELK/filebeat",
"data": "/home/ELK/filebeat/data",
"home": "/home/ELK/filebeat",
"logs": "/home/ELK/filebeat/logs"
},
"setup": {
"kibana": null,
"template": {
"settings": {
"index": {
"number_of_shards": 3
}
}
}
}
}
2017/11/24 10:11:06.971100 beat.go:426: INFO Home path: [/home/ELK/filebeat] Config path: [/home/ELK/filebeat] Data path: [/home/ELK/filebeat/data] Logs path: [/home/ELK/filebeat/logs]
2017/11/24 10:11:06.971194 beat.go:433: INFO Beat UUID: 1106ba9e-2afe-40c8-bd83-64a10486d24d
2017/11/24 10:11:06.971215 beat.go:192: INFO Setup Beat: filebeat; Version: 6.0.0
2017/11/24 10:11:06.971213 metrics.go:23: INFO Metrics logging every 30s
2017/11/24 10:11:06.971635 module.go:80: INFO Beat name: beat
2017/11/24 10:11:06.971995 beat.go:260: INFO filebeat start running.
2017/11/24 10:11:06.972057 registrar.go:88: INFO Registry file set to: /home/ELK/filebeat/data/registry
2017/11/24 10:11:06.972092 registrar.go:108: INFO Loading registrar data from /home/ELK/filebeat/data/registry
2017/11/24 10:11:06.972118 registrar.go:119: INFO States Loaded from registrar: 0
2017/11/24 10:11:06.972135 filebeat.go:260: WARN Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2017/11/24 10:11:06.972151 crawler.go:44: INFO Loading Prospectors: 0
2017/11/24 10:11:06.972308 crawler.go:78: INFO Loading and starting Prospectors completed. Enabled prospectors: 0
2017/11/24 10:11:06.972384 reload.go:128: INFO Config reloader started
2017/11/24 10:11:06.972417 reload.go:128: INFO Config reloader started
2017/11/24 10:11:06.972417 registrar.go:150: INFO Starting Registrar
2017/11/24 10:11:06.972497 reload.go:220: INFO Loading of config files completed.
2017/11/24 10:11:06.972506 reload.go:220: INFO Loading of config files completed.