Filebeat does not send data to logstash

Elastic Stack Beats
1

Hi!

here is my filebeat config

filebeat.inputs:
- type: syslog
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/nginx/access*.log
    - /var/log/messages
    - /bar/log/secure
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
output.logstash:
  hosts: ["192.168.0.61:5044"]
  tls:
     certificate_authorities: ["/etc/ssl/logstash_frwrd.crt"]
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

this is my logstash config

input {
	beats {
		port => 5044
		ssl => true
		ssl_certificate => "/etc/ssl/logstash_frwrd.crt"
		ssl_key => "/etc/ssl/logstash-forwarder.key"
		}
}
filter {
	if [type] == "syslog" {
				grok {
					match=>{ "message" => "%{SYSLOGLINE}" }
                                     }
				date {
					match => [ "timestamp", "MMM d HH:mm:ss",  "MMM dd HH:mm:ss" ]
	     			     }
			      }
	}
output {
	elasticsearch {
			hosts=> ["localhost"]
			index=>"%{[@metadata][beat]}-%{+YYYY.MM.dd}"
		      }
	stdout {
		codec => rubydebug
       	       }
       }

I'm checking with tcpdump if packets are sent to remote logstash with
tcpdump dst 192.168.0.61 - no packages are sent
and I'm listening on port 5044 on host 192.168.0.61 with tcpdump -Xni ens33 port 5044 - no packages are coming

Can you tell me what is wrong with my filebeat config?

2

Do you have anything in filebeat logs?

You need to share the logs from filebeat.

3

that's what I can see

tail -f filebeat-20221201.ndjson  
{"log.level":"info","@timestamp":"2022-12-01T12:56:32.006-0700","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-12-01T12:56:32.006-0700","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: ffb5ad29-8928-4222-83fb-716e7f30846a","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-12-01T12:56:35.011-0700","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
4

from jornalctl

Dec 01 13:09:30 fbeat filebeat[7796]: {"log.level":"info","@timestamp":"2022-12-01T13:09:30.051-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":40},"total":{"ticks":240,"value":240},"user":{"ticks":200}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"0d6df58d-f1b8-4882-8e8b-d11be7a4b027","uptime":{"ms":63078},"version":"8.5.2"},"memstats":{"gc_next":20875240,"memory_alloc":10354608,"memory_total":74655464,"rss":85393408},"runtime":{"goroutines":13}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.2,"15":0.36,"5":0.38,"norm":{"1":0.2,"15":0.36,"5":0.38}}}},"ecs.version":"1.6.0"}}
Dec 01 13:10:00 fbeat filebeat[7796]: {"log.level":"info","@timestamp":"2022-12-01T13:10:00.054-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":40},"total":{"ticks":250,"time":{"ms":10},"value":250},"user":{"ticks":210,"time":{"ms":10}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"0d6df58d-f1b8-4882-8e8b-d11be7a4b027","uptime":{"ms":93082},"version":"8.5.2"},"memstats":{"gc_next":20875240,"memory_alloc":10481696,"memory_total":74782552,"rss":85393408},"runtime":{"goroutines":13}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.12,"15":0.35,"5":0.34,"norm":{"1":0.12,"15":0.35,"5":0.34}}}},"ecs.version":"1.6.0"}}
Dec 01 13:10:30 fbeat filebeat[7796]: {"log.level":"info","@timestamp":"2022-12-01T13:10:30.058-0700","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":186},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":50,"time":{"ms":10}},"total":{"ticks":260,"time":{"ms":10},"value":260},"user":{"ticks":210}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"0d6df58d-f1b8-4882-8e8b-d11be7a4b027","uptime":{"ms":123078},"version":"8.5.2"},"memstats":{"gc_next":20875240,"memory_alloc":10690000,"memory_total":74990856,"rss":85393408},"runtime":{"goroutines":13}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.38,"15":0.36,"5":0.38,"norm":{"1":0.38,"15":0.36,"5":0.38}}}},"ecs.version":"1.6.0"}}
5

I can see now the packages are sent to logstash and received on the other side

How can I verify that logstash receves logs data? any logs, data storage on logstash side? or any storage in elasticsearch I should verify?

6

I believe we can close the thread as I can see now that logstash receives data from filebeat but not able to send data to elasticsearch. New topic will be opened

7

what did you find was the issue where beats wasn't sending data to logstash?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.