Filebeat is running but not sending logs to logstash

Elastic Stack Beats
1

Hi Team,

I am trying to setup filebeat on my centos 7 machine. filebeat is working fine but still not able to see any logs to logz.io.

filebeat.yml file configuration

filebeat.prospectors:
- type: log
  enabled: true
  paths:  ['/tm/logs/dataio/default/logstash.json']
  fields:
    logzio_codec: json
   
  fields_under_root: true
  encoding: utf-8

# Exclude fields
processors:
 - drop_fields:
    fields: ["beat.name", "beat.hostname", "beat.version"]

#-------------------------- Elasticsearch output ------------------------------
output:
  logstash:
    hosts: ["listener.logz.io:5015"]

#  The below configuration is used for Filebeat 1.3 or lower
    # tls:
    #   certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']

#  The below configuration is used for Filebeat 5.0 or higher
    ssl:
      certificate_authorities: ['/etc/pki/tls/certs/COMODORSADomainValidationSecureServerCA.crt']/>

any one can help me?

Thanks
Jitendra Bansal

2

The config looks normal to me. Were there any errors coming out of Filebeat? Can you check the Filebeat log?

3

yes i checked filebeat log file but did not see any error.

<2018-02-23T18:09:17.357Z INFO instance/beat.go:468 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2018-02-23T18:09:17.358Z INFO instance/beat.go:475 Beat UUID: 51e77e87-8056-4c11-a66d-1b1b7753c7e6
2018-02-23T18:09:17.358Z INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.1
2018-02-23T18:09:17.358Z INFO pipeline/module.go:76 Beat name: stage1-servicenode1.aca1.tidemark.net
2018-02-23T18:09:17.359Z INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-02-23T18:09:17.359Z INFO instance/beat.go:301 filebeat start running.
2018-02-23T18:09:17.359Z INFO registrar/registrar.go:108 Loading registrar data from /var/lib/filebeat/registry
2018-02-23T18:09:17.359Z INFO registrar/registrar.go:119 States Loaded from registrar: 22
2018-02-23T18:09:17.359Z WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-02-23T18:09:17.359Z INFO crawler/crawler.go:48 Loading Prospectors: 1
2018-02-23T18:09:17.360Z INFO log/prospector.go:111 Configured paths: [/tm/logs/dataio/default/logstash.json]
2018-02-23T18:09:17.360Z INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 1
2018-02-23T18:09:47.360Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":16},"total":{"ticks":20,"time":27,"value":20},"user":{"ticks":10,"time":11}},"info":{"ephemeral_id":"248a9c2f-5a4c-44a1-ac01-76253741f487","uptime":{"ms":30009}},"memstats":{"gc_next":4473924,"memory_alloc":2939104,"memory_total":2939104,"rss":14442496}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":16},"load":{"1":0,"15":0.09,"5":0.05,"norm":{"1":0,"15":0.0056,"5":0.0031}}}}}}
2018-02-23T18:10:17.360Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":18},"total":{"ticks":20,"time":29,"value":20},"user":{"ticks":10,"time":11}},"info":{"ephemeral_id":"248a9c2f-5a4c-44a1-ac01-76253741f487","uptime":{"ms":60009}},"memstats":{"gc_next":4473924,"memory_alloc":3140368,"memory_total":3140368}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0,"15":0.09,"5":0.04,"norm":{"1":0,"15":0.0056,"5":0.0025}}}}}}/>

4

Based on the metrics output. Filebeat is not harvesting any files.

      "filebeat": {
        "harvester": {
          "open_files": 0,
          "running": 0
        }
      },

The file you have configured probably does not exists.

5

Is that supposed to be /tmp?

6

no, file path is correct, this file /tm/logs/dataio/default/logstash.json exist and it have data inside it.

7

Could you please post the log output again, but with debug enabled?

# Debug logging config.
logging.level: debug
logging.selectors: ['*']
8

Hi Andrew,

Thanks for looking into the issue, but it is working now.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.