Filebeat - Non-zero metrics in the last 30s

[saravananveera]

Apparently logs are transferred from Filebeat to logstash, however the filebeat logs continiously show this message:

2020-07-15T01:18:20.789Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":164920,"time":{"ms":37}},"total":{"ticks":1260130,"time":{"ms":282},"value":1260130},"user":{"ticks":1095210,"time":{"ms":245}}},"info":{"ephemeral_id":"8f755a57-1431-4a63-9a10-985fdbd764a2","uptime":{"ms":107460017}},"memstats":{"gc_next":4985264,"memory_alloc":3396320,"memory_total":376637838888}},"filebeat":{"events":{"active":-16,"added":186,"done":202},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":202,"active":-8,"batches":30,"total":194}},"outputs":{"kafka":{"bytes_read":3038,"bytes_write":81397}},"pipeline":{"clients":1,"events":{"active":3,"published":186,"total":186},"queue":{"acked":202}}},"registrar":{"states":{"current":1,"update":202},"writes":{"success":30,"total":30}},"system":{"load":{"1":0,"15":0.23,"5":0.1,"norm":{"1":0,"15":0.0575,"5":0.025}}}}}}

using:

1. logstash-6.3.1
2. kafka_2.12-2.2.0
3. zookeeper-3.4.14
4. elasticsearch-6.3.1
5. kibana-6.3.1
6. filebeat-6.3.1

#============================= Filebeat modules ===============================

#=========================== Filebeat inputs =============================

filebeat.inputs:

• type: log

  Change to true to enable this input configuration.

  enabled: true

  Paths that should be crawled and fetched. Glob based paths.

  paths:
  #- /var/log/*.log

  • /var/log/nginx/demo.ssc-live.com/ssl-access.log
    #- c:\programdata\elasticsearch\logs*
    fields:
    log_topics: sscplblogs

#-------------------------- Kafka Output ----------------------------------
output.kafka:

initial brokers for reading cluster metadata

enabled: true
hosts: ["kafka Ip address:9092"]

message topic selection + partitioning

topic: '%{[fields][log_topics]}'
partition.round_robin:
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000

We have check the logs in the kafka there is no log on it. Can you please help to fix this issue.

[ChrsMark (Chris Mark)]

Hi!

The message you see is not an error message and it logs Filebeat's internal statistics. Since logs are being shipped to Logstash then Filebeat is properly configured.

C.

[saravananveera]

Hi @ChrsMark

Still i'm not able to receive the logs from filebeat to logstash, I have received this error log also from filebeat. kindly check and help me.

2020-07-15T11:07:21.306Z INFO kafka/log.go:36 client/metadata fetching metadata for all topics from broker ip203.ip-139-99-112.net:9092

Logstash conf:

input {
kafka {
bootstrap_servers =>["localhost:9092"]
topics => ["sscplblogs"]
codec => "json"
add_field => { "logz" => "plblogz" }
}
}

output {

if [logz] == "plblogz" {
elasticsearch {
hosts => ["Elastic IP:9200"]
index => "plblogs-%{+YYYY.MM.dd}"
}
file {
path => "/var/log/logstash/plblogs.log"
}
}
}

Kafka:

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic sscplblogs
Topic:sscplblogs PartitionCount:3 ReplicationFactor:1 Configs:
Topic: sscplblogs Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: sscplblogs Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: sscplblogs Partition: 2 Leader: 0 Replicas: 0 Isr: 0