Filebeat do not send data to logstash

Elastic Stack Beats
1 
2019-11-03T00:54:09.479+0300	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2019-11-03T00:54:09.480+0300	INFO	log/harvester.go:251	Harvester started for file: /home/cowrie/cowrie/var/log/cowrie/cowrie.json
2019-11-03T00:54:10.480+0300	INFO	pipeline/output.go:95	Connecting to backoff(async(tcp://10.1.0.19:5044))
2019-11-03T00:54:10.482+0300	INFO	pipeline/output.go:105	Connection to backoff(async(tcp://10.1.0.19:5044)) established
2019-11-03T00:54:39.481+0300	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":160,"time":{"ms":164}},"total":{"ticks":610,"time":{"ms":620},"value":610},"user":{"ticks":450,"time":{"ms":456}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"f1b10eb4-9fb7-47b9-b973-70ac5132de6d","uptime":{"ms":30097}},"memstats":{"gc_next":8262640,"memory_alloc":5880576,"memory_total":66507072,"rss":56578048},"runtime":{"goroutines":26}},"filebeat":{"events":{"active":39,"added":2349,"done":2310},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":2308,"batches":29,"total":2308},"read":{"bytes":174},"type":"logstash","write":{"bytes":158031}},"pipeline":{"clients":1,"events":{"active":39,"filtered":2,"published":2347,"retry":1157,"total":2349},"queue":{"acked":2308}}},"registrar":{"states":{"current":13,"update":2310},"writes":{"success":31,"total":31}},"system":{"cpu":{"cores":2},"load":{"1":0.62,"15":0.16,"5":0.37,"norm":{"1":0.31,"15":0.08,"5":0.185}}}}}}
2019-11-03T00:55:09.482+0300	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":220,"time":{"ms":64}},"total":{"ticks":990,"time":{"ms":380},"value":990},"user":{"ticks":770,"time":{"ms":316}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"f1b10eb4-9fb7-47b9-b973-70ac5132de6d","uptime":{"ms":60097}},"memstats":{"gc_next":10673904,"memory_alloc":7281336,"memory_total":118809800},"runtime":{"goroutines":26}},"filebeat":{"events":{"active":5,"added":1225,"done":1220},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1220,"batches":30,"total":1220},"read":{"bytes":180},"write":{"bytes":95896}},"pipeline":{"clients":1,"events":{"active":44,"published":1225,"total":1225},"queue":{"acked":1220}}},"registrar":{"states":{"current":13,"update":1220},"writes":{"success":30,"total":30}},"system":{"load":{"1":0.49,"15":0.16,"5":0.36,"norm":{"1":0.245,"15":0.08,"5":0.18}}}}}}
2019-11-03T00:55:39.482+0300	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":290,"time":{"ms":72}},"total":{"ticks":1350,"time":{"ms":368},"value":1350},"user":{"ticks":1060,"time":{"ms":296}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"f1b10eb4-9fb7-47b9-b973-70ac5132de6d","uptime":{"ms":90100}},"memstats":{"gc_next":10978672,"memory_alloc":5671952,"memory_total":171286296,"rss":262144},"runtime":{"goroutines":26}},"filebeat":{"events":{"active":-7,"added":1228,"done":1235},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1235,"batches":30,"total":1235},"read":{"bytes":180},"write":{"bytes":96690}},"pipeline":{"clients":1,"events":{"active":37,"published":1228,"total":1228},"queue":{"acked":1235}}},"registrar":{"states":{"current":13,"update":1235},"writes":{"success":30,"total":30}},"system":{"load":{"1":0.47,"15":0.18,"5":0.38,"norm":{"1":0.235,"15":0.09,"5":0.19}}}}}}

filebeat.yml
filebeat:
inputs:
-
paths:
- /home/cowrie/cowrie/var/log/cowrie/cowrie.json*
encoding: plain
input_type: log
registry.path: /var/lib/filebeat/registry
output:
logstash:
hosts: ["10.1.0.19:5044"]
shipper:
logging:
to_syslog: false
to_files: true
files:
path: /var/log/filebeat/
name: mybeat
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
level: info

but filebeat do not send cowire.json log to logstash. Connection is established

2

Who can help me? I can send any other information from these system

3 
- type: log

  # Change to true to enable this input configuration.
  enabled: true // I think you need to change enabled option to true.
4

Thanks for the answer. But it did not help, the same problem. Any suggestions?

5

Hi @Vladimir_Martyshin

Any error logs in Logstash? Everything I can see here looks good.

Do you have other logs going through the same Logstash input that work as expected?

6

Hi @Vladimir_Martyshin
is it the first time that you are trying to send the data??if you have already sent the data you have to remove some registry file in filebeat server

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.