Filebeat - Non-zero metrics in the last 30s

Apparently logs are transferred from Filebeat to logstash, however the filebeat logs continiously show this message:

2020-07-15T01:18:20.789Z INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":164920,"time":{"ms":37}},"total":{"ticks":1260130,"time":{"ms":282},"value":1260130},"user":{"ticks":1095210,"time":{"ms":245}}},"info":{"ephemeral_id":"8f755a57-1431-4a63-9a10-985fdbd764a2","uptime":{"ms":107460017}},"memstats":{"gc_next":4985264,"memory_alloc":3396320,"memory_total":376637838888}},"filebeat":{"events":{"active":-16,"added":186,"done":202},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":202,"active":-8,"batches":30,"total":194}},"outputs":{"kafka":{"bytes_read":3038,"bytes_write":81397}},"pipeline":{"clients":1,"events":{"active":3,"published":186,"total":186},"queue":{"acked":202}}},"registrar":{"states":{"current":1,"update":202},"writes":{"success":30,"total":30}},"system":{"load":{"1":0,"15":0.23,"5":0.1,"norm":{"1":0,"15":0.0575,"5":0.025}}}}}}

using:

  1. logstash-6.3.1
  2. kafka_2.12-2.2.0
  3. zookeeper-3.4.14
  4. elasticsearch-6.3.1
  5. kibana-6.3.1
  6. filebeat-6.3.1

#============================= Filebeat modules ===============================

#=========================== Filebeat inputs =============================

filebeat.inputs:

  • type: log

    Change to true to enable this input configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:
    #- /var/log/*.log

    • /var/log/nginx/demo.ssc-live.com/ssl-access.log
      #- c:\programdata\elasticsearch\logs*
      fields:
      log_topics: sscplblogs

#-------------------------- Kafka Output ----------------------------------
output.kafka:

initial brokers for reading cluster metadata

enabled: true
hosts: ["kafka Ip address:9092"]

message topic selection + partitioning

topic: '%{[fields][log_topics]}'
partition.round_robin:
reachable_only: false

required_acks: 1
compression: gzip
max_message_bytes: 1000000

We have check the logs in the kafka there is no log on it. Can you please help to fix this issue.

Hi!

The message you see is not an error message and it logs Filebeat's internal statistics. Since logs are being shipped to Logstash then Filebeat is properly configured.

C.

Hi @ChrsMark

Still i'm not able to receive the logs from filebeat to logstash, I have received this error log also from filebeat. kindly check and help me.

2020-07-15T11:07:21.306Z INFO kafka/log.go:36 client/metadata fetching metadata for all topics from broker ip203.ip-139-99-112.net:9092

Logstash conf:

input {
kafka {
bootstrap_servers =>["localhost:9092"]
topics => ["sscplblogs"]
codec => "json"
add_field => { "logz" => "plblogz" }
}
}

output {

if [logz] == "plblogz" {
elasticsearch {
hosts => ["Elastic IP:9200"]
index => "plblogs-%{+YYYY.MM.dd}"
}
file {
path => "/var/log/logstash/plblogs.log"
}
}
}

Kafka:

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic sscplblogs
Topic:sscplblogs PartitionCount:3 ReplicationFactor:1 Configs:
Topic: sscplblogs Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: sscplblogs Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: sscplblogs Partition: 2 Leader: 0 Replicas: 0 Isr: 0

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.