How to troubleshoot Windows Filebeat to logstash

[Freebird]

Hi,
I'm very new to all this and having problems figuring some basics.
I got elastic and kibana running, and then got logstash running and taking stdin and giving me output to stdout. Now I set logstash to (hopefully) send to elastic and also pump to stdout so I can see it.

But it looks like filebeat isn't doing anything and I'm not sure how to troubleshoot.

Everything is running on the same host.

I ran the ps scripts to do the install of filebeat.
I'm using the example filebeat.yml - I've just deleted a bunch of the unused lines and it looks like this:

#
#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - E:\T24Logs\*.txt

 
#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false


#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "localhost:5601"


#================================ Outputs =====================================

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]


#================================ Procesors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

I have no idea what the processors bit is about so left it alone.
I have a cut down example log file (sample.txt) in E:\T24Logs

If I execute 'start-service filebeat' in powershell, I don't get any errors or anything, but I also don't see the filebeat service started in the services applet.

The C:\Program Files\Filebeat\logs folder has a newest file dated 17th Jan which is when I started installing the stack.
I deleted a file out of C:\Program Files\Filebeat\data earlier as I read somewhere about resetting the registry

One thought I had - do I need to re-install if I change the filebeat.yml config?

[Freebird]

I found a command on stack overflow which lets the filebeat display onscreen:

PS C:\Program Files\Filebeat> .\filebeat.exe -c filebeat.yml -e -d "*"

That seems have walked through my sample file, which is good, but nothing seems to gotten to logstash from what I can tell. After listing the contents of my sample file, the output is:

2019-01-25T14:35:13.270+1100    DEBUG   [harvester]     log/harvester.go:510    Stopping harvester for file: E:\T24Logs\
Sample.txt
2019-01-25T14:35:13.271+1100    DEBUG   [harvester]     log/harvester.go:520    Closing file: E:\T24Logs\Sample.txt
2019-01-25T14:35:13.272+1100    DEBUG   [harvester]     log/harvester.go:390    Update state: E:\T24Logs\Sample.txt, off
set: 2342
2019-01-25T14:35:13.273+1100    DEBUG   [harvester]     log/harvester.go:531    harvester cleanup finished for file: E:\
T24Logs\Sample.txt
2019-01-25T14:35:13.274+1100    DEBUG   [publish]       pipeline/client.go:148  client: closing acker
2019-01-25T14:35:13.278+1100    DEBUG   [publish]       pipeline/client.go:150  client: done closing acker
2019-01-25T14:35:13.279+1100    DEBUG   [publish]       pipeline/client.go:154  client: cancelled 0 events
2019-01-25T14:35:13.279+1100    INFO    crawler/crawler.go:165  Crawler stopped
2019-01-25T14:35:13.280+1100    INFO    registrar/registrar.go:356      Stopping Registrar
2019-01-25T14:35:13.280+1100    DEBUG   [publish]       pipeline/client.go:148  client: closing acker
2019-01-25T14:35:13.280+1100    DEBUG   [publish]       pipeline/client.go:148  client: closing acker
2019-01-25T14:35:13.281+1100    INFO    registrar/registrar.go:282      Ending Registrar
2019-01-25T14:35:13.282+1100    DEBUG   [publish]       pipeline/client.go:150  client: done closing acker
2019-01-25T14:35:13.282+1100    DEBUG   [publish]       pipeline/client.go:150  client: done closing acker
2019-01-25T14:35:13.283+1100    DEBUG   [registrar]     registrar/registrar.go:400      Write registry file: C:\Program
Files\Filebeat\data\registry
2019-01-25T14:35:13.284+1100    DEBUG   [publish]       pipeline/client.go:154  client: cancelled 0 events
2019-01-25T14:35:13.285+1100    DEBUG   [publish]       pipeline/client.go:154  client: cancelled 0 events
2019-01-25T14:35:13.293+1100    DEBUG   [registrar]     registrar/registrar.go:393      Registry file updated. 1 states
written.
2019-01-25T14:35:13.451+1100    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics
": {"beat":{"cpu":{"system":{"ticks":218,"time":{"ms":218}},"total":{"ticks":530,"time":{"ms":530},"value":530},"user":{
"ticks":312,"time":{"ms":312}}},"handles":{"open":218},"info":{"ephemeral_id":"f9cbf3de-63fc-4691-90d4-02f786a1b636","up
time":{"ms":3434}},"memstats":{"gc_next":4194304,"memory_alloc":2450272,"memory_total":5041976,"rss":22736896}},"filebea
t":{"events":{"active":13,"added":14,"done":1},"harvester":{"closed":1,"open_files":0,"running":0,"started":1}},"libbeat
":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":12,"filtere
d":2,"published":12,"total":14}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":3,"total":3}},"syst
em":{"cpu":{"cores":4}}}}}
2019-01-25T14:35:13.451+1100    INFO    [monitoring]    log/log.go:153  Uptime: 3.5922996s
2019-01-25T14:35:13.454+1100    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
2019-01-25T14:35:13.455+1100    INFO    instance/beat.go:410    filebeat stopped.
2019-01-25T14:35:13.456+1100    ERROR   instance/beat.go:800    Exiting: No paths were defined for input accessing confi
g
Exiting: No paths were defined for input accessing config

I now have 3 files in the C:\Program Files\Filebeat\data folder (meta.json, registry and registry.old). No update to file in C:\Program Files\Filebeat\logs

[Freebird]

So the issue seemed to be that within C:\Program Files\Filebeat\modules.d the elasticsearch, logstash and system yaml's were not disabled. I think I had enabled them via powershell when I was trying to figure out how to get the stack to 'do' something.

Renaming the files with .disabled and restarting the service resulted in a flow into logstash and through to elastic

[Brett_Larson]

The command

.\filebeat export config

May be helpful in troubleshooting next time