Problem with filebeat yml file on Windows

dhiraj_khanna · February 5, 2019, 3:47pm

I am quite new to the Elastic stack and trying to experiment with visualization of apache log files in Kibana. I am using filebeat to ingest the apache logs. However when I run .\filebeat.exe setup -e, I get the following error:

2019-02-05T20:53:10.515+0530    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-02-05T20:53:10.520+0530    INFO    elasticsearch/client.go:721     Connected to Elasticsearch version 6.6.0
2019-02-05T20:53:10.520+0530    INFO    kibana/client.go:118    Kibana url: http://localhost:5601
2019-02-05T20:53:10.567+0530    WARN    fileset/modules.go:388  X-Pack Machine Learning is not enabled
2019-02-05T20:53:10.572+0530    ERROR   instance/beat.go:911    Exiting: 1 error: error loading config file: invalid con
fig: yaml: line 4: did not find expected hexdecimal number

My filebeat.yml file looks like this:

filebeat.inputs:

- type: log
  enabled: true
  paths: C:\Users\bigdataadmin\Downloads\ApacheLogs\*


#============================= Filebeat modules ===============================

filebeat.config.modules:
  
  path: C:\Program Files\Filebeat\modules.d\*.yml
  reload.enabled: true
  reload.period: 60s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
 
  hosts: ["localhost:9200"]


# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

I also checked the yml on http://www.yamllint.com/ but didn't find any problems. I can't seem to figure out what's wrong with line 4 of this file.

I am using filebeat 6.6

adrisr · February 5, 2019, 4:12pm

Have a look at our YAML tips and gotchas.

The problem is with the backslash character in the different paths:
path: C:\Users\.... <--- Wrong

Here, the YAML parser undestands \U as the start of a UNICODE escape character.

As the documentation suggests, you should wrap paths in 'single quotes':

path: 'C:\Users....' <---- Correct

dhiraj_khanna · February 5, 2019, 4:52pm

@adrisr thanks for your comment, as suggested I have added the single quotes. However I still get the same error message. This is what the yml looks like now:

filebeat.inputs:

- type: log
  enabled: true
  paths: 'C:\Users\bigdataadmin\Downloads\ApacheLogs\*'


#============================= Filebeat modules ===============================

filebeat.config.modules:
  
  path: 'C:\Program Files\Filebeat\modules.d\*.yml'
  reload.enabled: true
  reload.period: 60s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3

setup.kibana:
  host: "localhost:5601"

output.elasticsearch:
 
  hosts: ["localhost:9200"]


# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

dhiraj_khanna · February 5, 2019, 5:08pm

@adrisr I also tried filebeat test config -c filebeat.yml and it says Config OK.

dhiraj_khanna · February 5, 2019, 5:13pm

@adrisr If it helps, this is the entire out put of .\filebeat.exe setup -e:

2019-02-05T22:40:28.844+0530    INFO    instance/beat.go:616    Home path: [C:\Program Files\Filebeat] Config path:
Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2019-02-05T22:40:28.845+0530    INFO    instance/beat.go:623    Beat UUID: 2b19a527-f5cb-4444-af6f-6350d1f02a93
2019-02-05T22:40:28.846+0530    INFO    [beat]  instance/beat.go:936    Beat info       {"system_info": {"beat": {"
: {"config": "C:\\Program Files\\Filebeat", "data": "C:\\Program Files\\Filebeat\\data", "home": "C:\\Program Files
ebeat", "logs": "C:\\Program Files\\Filebeat\\logs"}, "type": "filebeat", "uuid": "2b19a527-f5cb-4444-af6f-6350d1f0
}}}
2019-02-05T22:40:28.846+0530    INFO    [beat]  instance/beat.go:945    Build info      {"system_info": {"build": {
it": "2c385a0764bdc537b6dc078a1d9bf11bb6d7bd95", "libbeat": "6.6.0", "time": "2019-01-24T10:28:39.000Z", "version":
.0"}}}
2019-02-05T22:40:28.846+0530    INFO    [beat]  instance/beat.go:948    Go runtime info {"system_info": {"go": {"os
ndows","arch":"amd64","max_procs":64,"version":"go1.10.8"}}}
2019-02-05T22:40:28.859+0530    INFO    [beat]  instance/beat.go:952    Host info       {"system_info": {"host": {"
tecture":"x86_64","boot_time":"2019-01-29T19:24:45.34+05:30","name":"JTDCRACK2SRV001","ip":["10.20.11.61/24","10.20
1/24","fe80::184b:c902:6a34:f43a/64","169.254.244.58/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.14393.2758
_release_1.190104-1904)","mac":["a8:b4:56:03:dd:b2","a8:b4:56:03:dd:b8","02:97:04:51:e6:9f","00:00:00:00:00:00:00:e
0:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2016 Datacenter","ver
:"10.0","major":10,"minor":0,"patch":0,"build":"14393.2759"},"timezone":"IST","timezone_offset_sec":19800,"id":"7f3
-1b74-41ad-8a17-9045ace64353"}}}
2019-02-05T22:40:28.863+0530    INFO    [beat]  instance/beat.go:981    Process info    {"system_info": {"process":
d": "C:\\Program Files\\Filebeat", "exe": "C:\\Program Files\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid
704, "ppid": 14256, "start_time": "2019-02-05T22:40:28.692+0530"}}}
2019-02-05T22:40:28.863+0530    INFO    instance/beat.go:281    Setup Beat: filebeat; Version: 6.6.0
2019-02-05T22:40:31.887+0530    INFO    add_cloud_metadata/add_cloud_metadata.go:319    add_cloud_metadata: hosting
ider type not detected.
2019-02-05T22:40:31.887+0530    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-02-05T22:40:31.888+0530    INFO    [publisher]     pipeline/module.go:110  Beat name: JTDCRACK2SRV001
2019-02-05T22:40:31.893+0530    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-02-05T22:40:31.902+0530    INFO    elasticsearch/client.go:721     Connected to Elasticsearch version 6.6.0
2019-02-05T22:40:31.907+0530    INFO    template/load.go:130    Template already exists and will not be overwritten
2019-02-05T22:40:31.907+0530    INFO    instance/beat.go:894    Template successfully loaded.
Loaded index template
Loading dashboards (Kibana must be running and reachable)
2019-02-05T22:40:31.908+0530    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-02-05T22:40:31.912+0530    INFO    elasticsearch/client.go:721     Connected to Elasticsearch version 6.6.0
2019-02-05T22:40:31.912+0530    INFO    kibana/client.go:118    Kibana url: http://localhost:5601
2019-02-05T22:41:22.084+0530    INFO    instance/beat.go:741    Kibana dashboards successfully loaded.
Loaded dashboards
2019-02-05T22:41:22.085+0530    INFO    elasticsearch/client.go:165     Elasticsearch url: http://localhost:9200
2019-02-05T22:41:22.089+0530    INFO    elasticsearch/client.go:721     Connected to Elasticsearch version 6.6.0
2019-02-05T22:41:22.090+0530    INFO    kibana/client.go:118    Kibana url: http://localhost:5601
2019-02-05T22:41:23.150+0530    WARN    fileset/modules.go:388  X-Pack Machine Learning is not enabled
2019-02-05T22:41:23.155+0530    ERROR   instance/beat.go:911    Exiting: 1 error: error loading config file: invali
fig: yaml: line 4: did not find expected hexdecimal number
Exiting: 1 error: error loading config file: invalid config: yaml: line 4: did not find expected hexdecimal number

adrisr · February 5, 2019, 6:50pm

Which modules are enabled?

filebeat.exe modules list

It might be a module.yml file inside modules.d directory.

dhiraj_khanna · February 6, 2019, 1:17am

@adrisr you are absolutely right. The same path issue existed in the apache2 module which I have now corrected and can now see the logs in Kibana. Thanks for your help, much appreciated!

system · March 6, 2019, 1:17am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.