Don't send log to Elastic search

Elastic Stack Beats
1

Now, we use logging system on ELK (filebeat > kafka > logstash> elasticsearch/kibana)
But we restart server on elasticsearch(=ES) recently.

I don't know why I can search only log file after when elasticsearch restart, but I don't find old log in kibana before restarting ES server.

For that I send to old log to ES, I had changed logfile name. But, I still don't find old log in kibana.

I don't know what component has problem (filebeat, kafka, logstash, elasticsearch/kibana)

But, I think that filebeat don't send new file(just change name)

I restart filebeat after change logfile name, but still don't send

2

if you think the problem might be with filebeat, it's a good idea to start with sharing filebeat configuration + logs. Maybe add kafka logs too.

3

filebeat log don't have something special.

I just wonder that filebeat recognizes the file which be re-named by new one although it is already scanned.

it is filebeat config

  input_type: log

  # Paths that should be crawled and fetched. Glob based paths.
  # To fetch all ".log" files from a specific level of subdirectories
  # /var/log/*/*.log can be used.
  # For each file found under this path, a harvester is started.
  # Make sure not file is defined twice as this can lead to unexpected behaviour.
  paths:
    - /opt/tomcat/logs/localhost_access_log.*.txt

  encoding: utf-8

  fields:
    module: "ACCESS"

  fields_under_root: true

  scan_frequency: 1s

############################# Output ##########################################

Configure what outputs to use when sending the data collected by the beat.

Multiple outputs may be used.

output:

kafka:
hosts: ["xxx.xxx.xxx.xxx:9092"]
topic: "applog"
use_type: false
client_id: "beats"
worker: 1

And it is kafka log (controller log)

[2016-11-14 01:46:26,639] DEBUG [Controller 0]: topics not in preferred replica Map() (kafka.controller.KafkaController)
[2016-11-14 01:46:26,640] TRACE [Controller 0]: leader imbalance ratio for broker 0 is 0.000000 (kafka.controller.KafkaController
[2016-11-14 01:51:26,639] TRACE [Controller 0]: checking need to trigger partition rebalance (kafka.controller.KafkaController)
[2016-11-14 01:51:26,639] DEBUG [Controller 0]: preferred replicas by broker Map(0 -> Map([applog,0] -> List(0))) (kafka.controllr.KafkaController)
[2016-11-14 01:51:26,639] DEBUG [Controller 0]: topics not in preferred replica Map() (kafka.controller.KafkaController)
[2016-11-14 01:51:26,640] TRACE [Controller 0]: leader imbalance ratio for broker 0 is 0.000000 (kafka.controller.KafkaController
[2016-11-14 01:56:26,639] TRACE [Controller 0]: checking need to trigger partition rebalance (kafka.controller.KafkaController)
[2016-11-14 01:56:26,639] DEBUG [Controller 0]: preferred replicas by broker Map(0 -> Map([applog,0] -> List(0))) (kafka.controllr.KafkaController)
[2016-11-14 01:56:26,639] DEBUG [Controller 0]: topics not in preferred replica Map() (kafka.controller.KafkaController)
[2016-11-14 01:56:26,640] TRACE [Controller 0]: leader imbalance ratio for broker 0 is 0.000000 (kafka.controller.KafkaController

4

Could you share the filebeat log file?

5

filebeat log is that just recursive below

16-11-15T01:47:26Z INFO Registry file updated. 204 states written.
2016-11-15T01:47:27Z INFO Run prospector
2016-11-15T01:47:27Z INFO Run prospector
2016-11-15T01:47:27Z INFO Run prospector

6

hm, interresting. The log basically states, the events have been successfully published to kafka.

kafka by default has a script to follow/display a topic:

bin/kafka-console-consumer.sh --zookeeper <zookeeper_ip>:2181 --topic applog --from-beginning

Anything in logstash logs?

7

I already try to do that, I don't find the log which i want to send.

Also, filebeat maybe send today's log but not previous log about filebeat log which share before.

Give a detailed,

I restart elastic serach for scaleup server in November. After restarting, elasticsearch don't show the log before restaring. Maybe delete all index of elasticsearch for some reason.

So I changed the name of log which made in October and restart filebeat in order to it is recognized as new files .
But I just see the log on kibana after restart elasticsearch. I don't see the log in October.

8

Renaming a file does not update the modification time. Also if the file is in the filebeat registry, it will not be fetched again. If you want to resend all data you must remove the registry file.

9

How can you tell filebeat did not send anything in the past? Did you check number of events and timestamps in kafka? Any old filebeat logs?

Before deleting the registry, can you share a copy? The registry contains collected filenames and offsets?

Every component in your chain is quite complex in itself, let's look at each one after another before applying changes to all services at the same time.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.