Apparently logs are transferred from Filebeat to logstash, however the filebeat logs continiously show this message:
2020-07-15T01:18:20.789Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":164920,"time":{"ms":37}},"total":{"ticks":1260130,"time":{"ms":282},"value":1260130},"user":{"ticks":1095210,"time":{"ms":245}}},"info":{"ephemeral_id":"8f755a57-1431-4a63-9a10-985fdbd764a2","uptime":{"ms":107460017}},"memstats":{"gc_next":4985264,"memory_alloc":3396320,"memory_total":376637838888}},"filebeat":{"events":{"active":-16,"added":186,"done":202},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":202,"active":-8,"batches":30,"total":194}},"outputs":{"kafka":{"bytes_read":3038,"bytes_write":81397}},"pipeline":{"clients":1,"events":{"active":3,"published":186,"total":186},"queue":{"acked":202}}},"registrar":{"states":{"current":1,"update":202},"writes":{"success":30,"total":30}},"system":{"load":{"1":0,"15":0.23,"5":0.1,"norm":{"1":0,"15":0.0575,"5":0.025}}}}}}
using:
- logstash-6.3.1
- kafka_2.12-2.2.0
- zookeeper-3.4.14
- elasticsearch-6.3.1
- kibana-6.3.1
- filebeat-6.3.1
#============================= Filebeat modules ===============================
#=========================== Filebeat inputs =============================
filebeat.inputs:
-
type: log
Change to true to enable this input configuration.
enabled: true
Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log- /var/log/nginx/demo.ssc-live.com/ssl-access.log
#- c:\programdata\elasticsearch\logs*
fields:
log_topics: sscplblogs
- /var/log/nginx/demo.ssc-live.com/ssl-access.log
#-------------------------- Kafka Output ----------------------------------
output.kafka:
initial brokers for reading cluster metadata
enabled: true
hosts: ["kafka Ip address:9092"]
message topic selection + partitioning
topic: '%{[fields][log_topics]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
We have check the logs in the kafka there is no log on it. Can you please help to fix this issue.