Filebeat does not send data to logstash

Kavita_Sidar · June 3, 2018, 8:08pm

I am new to filebeat and logstash and was trying to send log data from filebeat to logstash as per the documentation. But logstash does not disply any output.
This is the filebeat.yml file:

    filebeat.prospectors:
    - type: log
      paths:
        - C:\Users\Sidar\Downloads\Compressed\logstash-tutorial.log
    output.logstash:
         hosts: ["127.0.0.1:5044"]

and this is the logstash config file

input {
    beats {
        port => "5044"
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
    stdout { codec => rubydebug }
}

This should display the logs on console but there is not output after

[2018-06-04T01:27:21,147][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-06-04T01:27:21,677][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-06-04T01:27:21,782][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5e86e3df run>"}
[2018-06-04T01:27:21,858][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-06-04T01:27:21,954][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}

Please help as I'm not able to diagnose this.

harshbajaj16 · June 4, 2018, 3:16am

Hi @Kavita_Sidar,

i've checked you config files it seems ok and your logstash logs are showing that its working fine and ready to collect the logs from filebeat.

but for more understanding the issue i need your filebeat logs so that i can provide the solution to you.

Request you to paste the filebeat logs.

Path: /etc/filebeat/filebeat

Regards,

Kavita_Sidar · June 4, 2018, 5:26am

Here is the filebeat log. Thank you.
2018-06-01T17:06:37.488-0700 INFO instance/beat.go:468 Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2018-06-01T17:06:37.503-0700 INFO instance/beat.go:475 Beat UUID: 4cb6864c-ceef-459d-a24d-13e037c99d5b
2018-06-01T17:06:37.503-0700 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2018-06-01T17:06:37.504-0700 INFO pipeline/module.go:76 Beat name: DESKTOP-F4FFD42
2018-06-01T17:06:37.507-0700 INFO instance/beat.go:301 filebeat start running.
2018-06-01T17:06:37.507-0700 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-06-01T17:06:37.507-0700 INFO registrar/registrar.go:110 Loading registrar data from C:\Program Files\Filebeat\data\registry
2018-06-01T17:06:37.507-0700 INFO registrar/registrar.go:121 States Loaded from registrar: 0
2018-06-01T17:06:37.507-0700 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-06-01T17:06:37.507-0700 INFO crawler/crawler.go:48 Loading Prospectors: 1
2018-06-01T17:06:37.508-0700 INFO log/prospector.go:111 Configured paths: [C:\Program Files\Filebeat\logstash-tutorial.log]
2018-06-01T17:06:37.508-0700 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 1
2018-06-01T17:07:07.640-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":77,"time":77,"value":77},"user":{"ticks":46,"time":46}},"info":{"ephemeral_id":"b399c579-03de-4604-84f7-46a1bc0ad41c","uptime":{"ms":30073}},"memstats":{"gc_next":4473924,"memory_alloc":2854736,"memory_total":2854736,"rss":16138240}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4}}}}}
2018-06-01T17:07:37.509-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":77,"time":77,"value":77},"user":{"ticks":46,"time":46}},"info":{"ephemeral_id":"b399c579-03de-4604-84f7-46a1bc0ad41c","uptime":{"ms":60074}},"memstats":{"gc_next":4473924,"memory_alloc":2906016,"memory_total":2906016,"rss":2428928}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2018-06-01T17:08:07.509-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":77,"time":77,"value":77},"user":{"ticks":46,"time":46}},"info":{"ephemeral_id":"b399c579-03de-4604-84f7-46a1bc0ad41c","uptime":{"ms":90074}},"memstats":{"gc_next":4473924,"memory_alloc":2946368,"memory_total":2946368,"rss":-12288}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2018-06-01T17:08:37.509-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":77,"time":77,"value":77},"user":{"ticks":46,"time":46}},"info":{"ephemeral_id":"b399c579-03de-4604-84f7-46a1bc0ad41c","uptime":{"ms":120075}},"memstats":{"gc_next":4473924,"memory_alloc":2984352,"memory_total":2984352,"rss":-12288}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2018-06-01T17:09:07.509-0700 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":77,"time":77,"value":77},"user":{"ticks":46,"time":46}},"info":{"ephemeral_id":"b399c579-03de-4604-84f7-46a1bc0ad41c","uptime":{"ms":150075}},"memstats":{"gc_next":4473924,"memory_alloc":3025136,"memory_total":3025136,"rss":40960}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}}}}}
2018-06-01T17:09:28.109-0700 INFO beater/filebeat.go:323 Stopping filebeat
2018-06-01T17:09:28.109-0700 INFO crawler/crawler.go:109 Stopping Crawler
2018-06-01T17:09:28.109-0700 INFO crawler/crawler.go:119 Stopping 1 prospectors
2018-06-01T17:09:28.109-0700 INFO prospector/prospector.go:121 Prospector ticker stopped
2018-06-01T17:09:28.109-0700 INFO prospector/prospector.go:138 Stopping Prospector: 9758062311585772904
2018-06-01T17:09:28.109-0700 INFO crawler/crawler.go:135 Crawler stopped
2018-06-01T17:09:28.109-0700 INFO registrar/registrar.go:239 Stopping Registrar
2018-06-01T17:09:28.109-0700 INFO registrar/registrar.go:167 Ending Registrar
2018-06-01T17:09:28.167-0700 INFO instance/beat.go:308 filebeat stopped.
2018-06-01T17:09:28.169-0700 INFO [monitoring] log/log.go:132 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":31,"time":31},"total":{"ticks":93,"time":93,"value":93},"user":{"ticks":62,"time":62}},"info":{"ephemeral_id":"b399c579-03de-4604-84f7-46a1bc0ad41c","uptime":{"ms":170734}},"memstats":{"gc_next":4473924,"memory_alloc":3078504,"memory_total":3078504,"rss":18706432}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":4}}}}}
2018-06-01T17:09:28.170-0700 INFO [monitoring] log/log.go:133 Uptime: 2m50.737057s
2018-06-01T17:09:28.170-0700 INFO [monitoring] log/log.go:110 Stopping metrics logging.

harshbajaj16 · June 4, 2018, 6:11am

it seems ok no error found. Are you able to telnet or ping from filebeat node?

Please check 5044 port is open or not.

Regards,

Kavita_Sidar · June 4, 2018, 6:22am

Yes it is able to telnet and ping. The port is open.

Kavita_Sidar · June 4, 2018, 7:13am

It worked when i specified the address directly to the log file:
-C:\Users\Sidar\Downloads\Compressed\logstash-tutorial.log\logstash-tutorial-dataset
in the filebeat.yml file. It's showing output in logstash.

But i dont understand why it was not able to pick file when only the folder was specified.

harshbajaj16 · June 4, 2018, 7:41am

Hi,

Sorry but i think there is something wrong in your path you have specified is .log/logstash-tutorial-dataset means after log file you mentioned directory or what is this i'm not able to understand.

please explain below configuration a bit more.

Kavita_Sidar · June 4, 2018, 8:59am

Hi,
logstash-tutorial-dataset is the apache log file and logstash-tutorial.log is the folder which contains logstash-tutorial-dataset file.

harshbajaj16 · June 4, 2018, 9:16am

Hi,

Ohk i understood your point now. Please use below format for same and restart the services and try it

Please do let me know if you are still facing issue.

Regards,

Kavita_Sidar · June 4, 2018, 10:19am

It worked.
Thank you.

system · July 2, 2018, 10:19am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.