Filebeat is running but not sending logs/data to logstash

Elastic Stack Beats
1

Hi Team,

I am trying to setup filebeat on my centos 7 machine. filebeat is working fine but still not able to send logstash.
Filebeat.yml file
###################### Filebeat Configuration Example #########################

This file is an example configuration file highlighting only the most common

options. The filebeat.reference.yml file from the same directory contains all the

supported options with more comments. You can use it as a reference.

You can find the full configuration reference here:

https://www.elastic.co/guide/en/beats/filebeat/index.html

For more available modules and options, please see the filebeat.reference.yml sample

configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

Each - is an input. Most options can be set at the input level, so

you can use different inputs for various configurations.

Below are the input specific configurations.

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /home/rahul-1/logstash-tutorial.log
      #- c:\programdata\elasticsearch\logs*

    Exclude lines. A list of regular expressions to match. It drops the lines that are

    matching any regular expression from the list.

    #exclude_lines: ['^DBG']

    Include lines. A list of regular expressions to match. It exports the lines that are

    matching any regular expression from the list.

    #include_lines: ['^ERR', '^WARN']

AND
LOGS are
2019-08-05T18:48:52.726+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-08-05T18:48:52.726+0530 DEBUG [input] log/config.go:204 recursive glob enabled
2019-08-05T18:48:52.726+0530 DEBUG [input] log/input.go:160 exclude_files: . Number of stats: 2
2019-08-05T18:48:52.726+0530 DEBUG [input] file/states.go:68 New state added for /home/rahul-1/logstash-tutorial.log
2019-08-05T18:48:52.726+0530 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2019-08-05T18:48:52.726+0530 DEBUG [publisher] pipeline/client.go:200 Pipeline client receives callback 'onFilteredOut' for event: %+v{0001-01-01 00:00:00 +0000 UTC null null {8947485-2049 true /home/rahul-1/logstash-tutorial.log 24464 2019-08-05 18:36:56.138135938 +0530 IST -1ns log map 8947485-2049} false}
2019-08-05T18:48:52.726+0530 DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
2019-08-05T18:48:52.726+0530 DEBUG [input] log/input.go:181 input with previous states loaded: 1
2019-08-05T18:48:52.726+0530 INFO log/input.go:148 Configured paths: [/home/rahul-1/logstash-tutorial.log]
2019-08-05T18:48:52.726+0530 INFO input/input.go:114 Starting input of type: log; ID: 6744791836451698202
2019-08-05T18:48:52.726+0530 DEBUG [cfgfile] cfgfile/reload.go:133 Checking module configs from: /usr/share/filebeat/bin/modules.d/*.yml
2019-08-05T18:48:52.727+0530 DEBUG [cfgfile] cfgfile/reload.go:147 Number of module configs found: 0
2019-08-05T18:48:52.727+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-08-05T18:48:52.726+0530 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2019-08-05T18:48:52.727+0530 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2019-08-05T18:48:52.727+0530 DEBUG [registrar] registrar/registrar.go:346 Registrar states cleaned up. Before: 2, After: 2, Pending: 0
2019-08-05T18:48:52.727+0530 DEBUG [input] log/input.go:187 Start next scan
2019-08-05T18:48:52.727+0530 INFO cfgfile/reload.go:171 Config reloader started
2019-08-05T18:48:52.727+0530 DEBUG [input] log/input.go:417 Check file for harvesting: /home/rahul-1/logstash-tutorial.log
2019-08-05T18:48:52.727+0530 DEBUG [input] log/input.go:507 Update existing file for harvesting: /home/rahul-1/logstash-tutorial.log, offset: 24464
2019-08-05T18:48:52.727+0530 DEBUG [input] log/input.go:561 File didn't change: /home/rahul-1/logstash-tutorial.log
2019-08-05T18:48:52.727+0530 DEBUG [input] log/input.go:208 input states cleaned up. Before: 1, After: 1, Pending: 0
2019-08-05T18:48:52.727+0530 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /usr/share/filebeat/bin/data/registry/filebeat/data.json (2)
2019-08-05T18:48:52.727+0530 DEBUG [cfgfile] cfgfile/reload.go:197 Scan for new config files
2019-08-05T18:48:52.727+0530 DEBUG [cfgfile] cfgfile/reload.go:216 Number of module configs found: 0
2019-08-05T18:48:52.727+0530 DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0
2019-08-05T18:48:52.728+0530 DEBUG [reload] cfgfile/list.go:80 Start list: 0, Stop list: 0
2019-08-05T18:48:52.728+0530 INFO cfgfile/reload.go:226 Loading of config files completed.
2019-08-05T18:48:52.795+0530 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 2 states written.
2019-08-05T18:49:02.727+0530 DEBUG [input] input/input.go:152 Run input
2019-08-05T18:49:02.727+0530 DEBUG [input] log/input.go:187 Start next scan
2019-08-05T18:49:02.727+0530 DEBUG [input] log/input.go:417 Check file for harvesting: /home/

2

Are you sure the file hasn't been processed before? In the debug logs, it says that the file hasn't changed.
Could you share the registry file and the example logs you are trying to send?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.