Missing field accessing 'path' accessing 'filebeat' (source:'filebeat.yml')

[teejayuu]

Trying to work out why filebeat 6.1.0 is not shipping logs and ran the following from powershell .\filebeat -v -e -d "config" and this was returned:

PS D:\Program Files\filebeat-6.1.0> .\filebeat -v -e -d "config"
filebeat2017/12/21 15:07:23.997253 config.go:214: DBG [config] load config file 'filebeat.yml' =>
<config error> missing field accessing 'path' accessing 'filebeat' (source:'filebeat.yml')
filebeat2017/12/21 15:07:24.003355 config.go:214: DBG [config] Complete configuration loaded:
{
  "filebeat": {
    "config": {
      "modules": {
        "path": "D:\\Program Files\\filebeat-6.1.0/modules.d/*.yml",
        "reload": {
          "enabled": false
        }
      }
    },
    "prospectors": [
      {
        "enabled": true,
        "exclude_lines": [
          "#"
        ],
        "paths": [
          "d:\\Inetpub\\logs\\logfiles\\*\\*"
        ],
        "type": "log"
      }
    ]
  },
  "logging": {
    "files": {
      "keepfiles": 7,
      "name": "mybeat.log",
      "path": "D:\\Program Files\\filebeat-6.1.0\\log"
    },
    "level": "warning",
    "to_files": true,
    "to_syslog": false
  },
  "output": {
    "logstash": {
      "hosts": [
        "xxxxx"
      ],
      "index": "filebeat_iis"
    }
  },
  "path": {
    "config": "D:\\Program Files\\filebeat-6.1.0",
    "data": "D:\\Program Files\\filebeat-6.1.0\\data",
    "home": "D:\\Program Files\\filebeat-6.1.0",
    "logs": "D:\\Program Files\\filebeat-6.1.0\\logs"
  },
  "setup": {
    "kibana": null,
    "template": {
      "settings": {
        "index": {
          "number_of_shards": 3
        }
      }
    }
  },
  "tags": [
    "iis-logs"
  ]
}

What does this mean missing field accessing 'path' accessing 'filebeat' (source:'filebeat.yml')?

[teejayuu]

Additional:

2017/12/21 15:07:24.179870 metrics.go:23: INFO Metrics logging every 30s
2017/12/21 15:07:24.179870 beat.go:436: INFO Home path: [D:\Program Files\filebeat-6.1.0] Config path: [D:\Program Files
\filebeat-6.1.0] Data path: [D:\Program Files\filebeat-6.1.0\data] Logs path: [D:\Program Files\filebeat-6.1.0\logs]
2017/12/21 15:07:24.182867 beat.go:443: INFO Beat UUID: 7f14eae6-4592-4dc4-be2e-7cccadd4cc5f
2017/12/21 15:07:24.183869 beat.go:203: INFO Setup Beat: filebeat; Version: 6.1.0
2017/12/21 15:07:24.189145 module.go:76: INFO Beat name: hemera
2017/12/21 15:07:24.194116 beat.go:276: INFO filebeat start running.
2017/12/21 15:07:24.267673 registrar.go:88: INFO Registry file set to: D:\Program Files\filebeat-6.1.0\data\registry
2017/12/21 15:07:24.280840 registrar.go:108: INFO Loading registrar data from D:\Program Files\filebeat-6.1.0\data\regis
try
2017/12/21 15:07:24.287736 registrar.go:119: INFO States Loaded from registrar: 52
2017/12/21 15:07:24.288763 registrar.go:150: INFO Starting Registrar
2017/12/21 15:07:24.288763 filebeat.go:261: WARN Filebeat is unable to load the Ingest Node pipelines for the configured
 modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipeline
s or are using Logstash pipelines, you can ignore this warning.
2017/12/21 15:07:24.312996 crawler.go:48: INFO Loading Prospectors: 1
2017/12/21 15:07:24.410150 prospector.go:87: INFO Starting prospector of type: log; ID: 9709682197378411112
2017/12/21 15:07:24.419968 crawler.go:82: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017/12/21 15:07:24.419968 reload.go:127: INFO Config reloader started
2017/12/21 15:07:24.424963 reload.go:219: INFO Loading of config files completed.

[ruflin]

Can you share your filebeat.yml file?

[teejayuu]

Thanks ruflin, sorry for the delay...holidays etc. Here is my filebeat.yml:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - d:\Inetpub\logs\logfiles\*\*
  exclude_lines: ['#']
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
tags: ["iis-logs"]
setup.kibana:
output.logstash:
  hosts: ["scamander.hstv.local:5044"]
  index: "filebeat_iis"
logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: D:\Program Files\filebeat-6.1.0\log
  name: mybeat.log
  keepfiles: 7

Thanks

[ruflin]

I wonder if one of the following paths cannot be accessed:

    "config": "D:\\Program Files\\filebeat-6.1.0",
    "data": "D:\\Program Files\\filebeat-6.1.0\\data",
    "home": "D:\\Program Files\\filebeat-6.1.0",
    "logs": "D:\\Program Files\\filebeat-6.1.0\\logs"

Do these directories all exist. @steffens might know more about the log message.

But TBH I don't think this is related to why Filebeat does not ship logs. Based on the log output you posted above all looks normal.

How do you update your log file?
Can share a bit more of your log file on how it looks after 2-3 minutes? There should be an output every 30s with some stats.

[steffens (Steffen Siering)]

The message seems to be related to debug output only. Might be we have a debug statement trying to print a yet incomplete filebeat configuration (right after reading filebeat.yml, the paths namespace is not yet available). But after merging all settings, the debug creates a valid config (printed as JSON), with all path configs being properly expanded.

I wonder if a) there is actually an harvester running b) we have errors in the output. The sample startup logs don't draw a complete picture yet.