How to troubleshoot Windows Filebeat to logstash

I'm very new to all this and having problems figuring some basics.
I got elastic and kibana running, and then got logstash running and taking stdin and giving me output to stdout. Now I set logstash to (hopefully) send to elastic and also pump to stdout so I can see it.

But it looks like filebeat isn't doing anything and I'm not sure how to troubleshoot.

Everything is running on the same host.

I ran the ps scripts to do the install of filebeat.
I'm using the example filebeat.yml - I've just deleted a bunch of the unused lines and it looks like this:

#=========================== Filebeat inputs =============================


- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
    #- /var/log/*.log
    - E:\T24Logs\*.txt

#============================= Filebeat modules ===============================

  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.

  host: "localhost:5601"

#================================ Outputs =====================================

#----------------------------- Logstash output --------------------------------
  # The Logstash hosts
  hosts: ["localhost:5044"]

#================================ Procesors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

  - add_host_metadata: ~
  - add_cloud_metadata: ~

I have no idea what the processors bit is about so left it alone.
I have a cut down example log file (sample.txt) in E:\T24Logs

If I execute 'start-service filebeat' in powershell, I don't get any errors or anything, but I also don't see the filebeat service started in the services applet.

The C:\Program Files\Filebeat\logs folder has a newest file dated 17th Jan which is when I started installing the stack.
I deleted a file out of C:\Program Files\Filebeat\data earlier as I read somewhere about resetting the registry

One thought I had - do I need to re-install if I change the filebeat.yml config?

I found a command on stack overflow which lets the filebeat display onscreen:

PS C:\Program Files\Filebeat> .\filebeat.exe -c filebeat.yml -e -d "*"

That seems have walked through my sample file, which is good, but nothing seems to gotten to logstash from what I can tell. After listing the contents of my sample file, the output is:

2019-01-25T14:35:13.270+1100    DEBUG   [harvester]     log/harvester.go:510    Stopping harvester for file: E:\T24Logs\
2019-01-25T14:35:13.271+1100    DEBUG   [harvester]     log/harvester.go:520    Closing file: E:\T24Logs\Sample.txt
2019-01-25T14:35:13.272+1100    DEBUG   [harvester]     log/harvester.go:390    Update state: E:\T24Logs\Sample.txt, off
set: 2342
2019-01-25T14:35:13.273+1100    DEBUG   [harvester]     log/harvester.go:531    harvester cleanup finished for file: E:\
2019-01-25T14:35:13.274+1100    DEBUG   [publish]       pipeline/client.go:148  client: closing acker
2019-01-25T14:35:13.278+1100    DEBUG   [publish]       pipeline/client.go:150  client: done closing acker
2019-01-25T14:35:13.279+1100    DEBUG   [publish]       pipeline/client.go:154  client: cancelled 0 events
2019-01-25T14:35:13.279+1100    INFO    crawler/crawler.go:165  Crawler stopped
2019-01-25T14:35:13.280+1100    INFO    registrar/registrar.go:356      Stopping Registrar
2019-01-25T14:35:13.280+1100    DEBUG   [publish]       pipeline/client.go:148  client: closing acker
2019-01-25T14:35:13.280+1100    DEBUG   [publish]       pipeline/client.go:148  client: closing acker
2019-01-25T14:35:13.281+1100    INFO    registrar/registrar.go:282      Ending Registrar
2019-01-25T14:35:13.282+1100    DEBUG   [publish]       pipeline/client.go:150  client: done closing acker
2019-01-25T14:35:13.282+1100    DEBUG   [publish]       pipeline/client.go:150  client: done closing acker
2019-01-25T14:35:13.283+1100    DEBUG   [registrar]     registrar/registrar.go:400      Write registry file: C:\Program
2019-01-25T14:35:13.284+1100    DEBUG   [publish]       pipeline/client.go:154  client: cancelled 0 events
2019-01-25T14:35:13.285+1100    DEBUG   [publish]       pipeline/client.go:154  client: cancelled 0 events
2019-01-25T14:35:13.293+1100    DEBUG   [registrar]     registrar/registrar.go:393      Registry file updated. 1 states
2019-01-25T14:35:13.451+1100    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics
": {"beat":{"cpu":{"system":{"ticks":218,"time":{"ms":218}},"total":{"ticks":530,"time":{"ms":530},"value":530},"user":{
2019-01-25T14:35:13.451+1100    INFO    [monitoring]    log/log.go:153  Uptime: 3.5922996s
2019-01-25T14:35:13.454+1100    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
2019-01-25T14:35:13.455+1100    INFO    instance/beat.go:410    filebeat stopped.
2019-01-25T14:35:13.456+1100    ERROR   instance/beat.go:800    Exiting: No paths were defined for input accessing confi
Exiting: No paths were defined for input accessing config

I now have 3 files in the C:\Program Files\Filebeat\data folder (meta.json, registry and registry.old). No update to file in C:\Program Files\Filebeat\logs

So the issue seemed to be that within C:\Program Files\Filebeat\modules.d the elasticsearch, logstash and system yaml's were not disabled. I think I had enabled them via powershell when I was trying to figure out how to get the stack to 'do' something.

Renaming the files with .disabled and restarting the service resulted in a flow into logstash and through to elastic :slight_smile:

The command

.\filebeat export config

May be helpful in troubleshooting next time :slight_smile:

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.