Filebeat not reading log files

Elastic Stack Beats
1

Hi,

Filebeat is not processing any files from the input folders
Setup : Filebeat -> Logstash -> Elasticsearch -> Kibana (All are version 7.6.1)
Filebeat docker running on mac, only one instance running.
ELK running in Openshift 3.1l.
Logstash service is port-forwarded (5044) to localhost

logstash pipeline

      input {
        beats {
          port => 5044
        }
      }

logstash startup logs

[INFO ] 2020-03-26 15:19:34.417 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2020-03-26 15:19:34.475 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2020-03-26 15:19:34.487 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2020-03-26 15:19:34.585 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-03-26 15:19:34.717 [[main]<beats] Server - Starting server on port: 5044

filebeat.yaml

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

output.logstash:
  hosts: ["docker.for.mac.localhost:5044"]

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /Users/u740885/git/ocp-elk/development/logs/*.log

filebeat startup logs

2020-03-26T19:47:33.916Z	INFO	instance/beat.go:298	Setup Beat: filebeat; Version: 7.6.1
2020-03-26T19:47:33.917Z	INFO	[publisher]	pipeline/module.go:110	Beat name: ac9600497852
2020-03-26T19:47:33.918Z	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-03-26T19:47:33.918Z	INFO	instance/beat.go:439	filebeat start running.
2020-03-26T19:47:33.918Z	INFO	registrar/migrate.go:104	No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2020-03-26T19:47:33.919Z	INFO	registrar/migrate.go:112	Initialize registry meta file
2020-03-26T19:47:33.920Z	INFO	registrar/registrar.go:108	No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2020-03-26T19:47:33.923Z	INFO	registrar/registrar.go:145	Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2020-03-26T19:47:33.923Z	INFO	registrar/registrar.go:152	States Loaded from registrar: 0
2020-03-26T19:47:33.923Z	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-03-26T19:47:33.923Z	INFO	crawler/crawler.go:72	Loading Inputs: 1
2020-03-26T19:47:33.924Z	INFO	log/input.go:152	Configured paths: [/Users/u740885/git/ocp-elk/development/logs/*.log]
2020-03-26T19:47:33.924Z	INFO	input/input.go:114	Starting input of type: log; ID: 16971255910140926814
2020-03-26T19:47:33.924Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2020-03-26T19:47:33.924Z	INFO	cfgfile/reload.go:171	Config reloader started
2020-03-26T19:47:33.924Z	INFO	cfgfile/reload.go:226	Loading of config files completed.

docker exec -it filebeat ./filebeat test output

logstash: docker.for.mac.localhost:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.65.2
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

docker exec -it filebeat ./filebeat -c filebeat.yml -e -d "*"

2020-03-26T20:22:11.626Z	INFO	instance/beat.go:622	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2020-03-26T20:22:11.626Z	DEBUG	[beat]	instance/beat.go:674	Beat metadata path: /usr/share/filebeat/data/meta.json
2020-03-26T20:22:11.626Z	INFO	instance/beat.go:630	Beat ID: 0df76fb1-c1e5-44b2-99e7-751f0ce616ed
2020-03-26T20:22:11.626Z	INFO	instance/beat.go:380	filebeat stopped.
2020-03-26T20:22:11.626Z	ERROR	instance/beat.go:933	Exiting: data path already locked by another beat
Exiting: data path already locked by another beat
2

Instead of running filebeat as docker, ran it in macOS changing hosts as below it worked. Any thoughts ?

["localhost:5044"]

3

The docker exec commands runs a new command in a running container. In your case the filebeat container is already running, which means the filebeat process inside that container is already running. So when you try to run it again with docker exec -it filebeat ./filebeat -c filebeat.yml -e -d "*", it doesn't allow you to do that prevent corruption of state Filebeat maintains in the data path.

In other words, if you are already running the Filebeat container, you are already running Filebeat for log processing. There's no need to try and run it again.

Now, back to your main question, which is why Filebeat is not reading log files. Thanks for posting your Filebeat logs. From that we can tell that the Filebeat process running inside the container starts watching for files that match this pattern/Users/u740885/git/ocp-elk/development/logs/*.log. So the next thing to check would be whether such a path indeed exists inside the Filebeat container. To do that I'd suggest running this command:

docker exec -it filebeat ls -al /Users/u740885/git/ocp-elk/development/logs/*.log`

What do you get back?

Thanks,

Shaunak

4

Thanks @shaunak, had a typo in the volume name.

Raj

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.