# Don't send log to Elastic search **URL:** \ **Category:** Beats\ **Created:** [November 11, 2016, 10:08am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787 "2016-11-11T10:08:59Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jyp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jyp/32/13114_2.png) [@jyp](https://discuss.elastic.co/u/jyp)\ **Post date:** [November 11, 2016, 10:08am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/1 "2016-11-11T10:08:59Z")
Now, we use logging system on ELK (filebeat \> kafka \> logstash\> elasticsearch/kibana) But we restart server on elasticsearch(=ES) recently. I don't know why I can search only log file after when elasticsearch restart, but I don't find old log in kibana before restarting ES server. For that I send to old log to ES, I had changed logfile name. But, I still don't find old log in kibana. I don't know what component has problem (filebeat, kafka, logstash, elasticsearch/kibana) But, I think that filebeat don't send new file(just change name) I restart filebeat after change logfile name, but still don't send ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 11, 2016, 1:59pm UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/2 "2016-11-11T13:59:02Z")
if you think the problem might be with filebeat, it's a good idea to start with sharing filebeat configuration + logs. Maybe add kafka logs too. ---
**Author:** ![jyp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jyp/32/13114_2.png) [@jyp](https://discuss.elastic.co/u/jyp)\ **Post date:** [November 14, 2016, 9:55am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/3 "2016-11-14T09:55:38Z")
filebeat log don't have something special. I just wonder that filebeat recognizes the file which be re-named by new one although it is already scanned. ## it is filebeat config ``` input_type: log # Paths that should be crawled and fetched. Glob based paths. # To fetch all ".log" files from a specific level of subdirectories # /var/log/*/*.log can be used. # For each file found under this path, a harvester is started. # Make sure not file is defined twice as this can lead to unexpected behaviour. paths: - /opt/tomcat/logs/localhost_access_log.*.txt encoding: utf-8 fields: module: "ACCESS" fields_under_root: true scan_frequency: 1s ``` ############################# Output ########################################## # Configure what outputs to use when sending the data collected by the beat. # Multiple outputs may be used. output: ## kafka: hosts: ["[xxx.xxx.xxx.xxx:9092](http://xxx.xxx.xxx.xxx:9092)"] topic: "applog" use\_type: false client\_id: "beats" worker: 1 And it is kafka log (controller log) [2016-11-14 01:46:26,639] DEBUG [Controller 0]: topics not in preferred replica Map() (kafka.controller.KafkaController) [2016-11-14 01:46:26,640] TRACE [Controller 0]: leader imbalance ratio for broker 0 is 0.000000 (kafka.controller.KafkaController [2016-11-14 01:51:26,639] TRACE [Controller 0]: checking need to trigger partition rebalance (kafka.controller.KafkaController) [2016-11-14 01:51:26,639] DEBUG [Controller 0]: preferred replicas by broker Map(0 -\> Map([applog,0] -\> List(0))) (kafka.controllr.KafkaController) [2016-11-14 01:51:26,639] DEBUG [Controller 0]: topics not in preferred replica Map() (kafka.controller.KafkaController) [2016-11-14 01:51:26,640] TRACE [Controller 0]: leader imbalance ratio for broker 0 is 0.000000 (kafka.controller.KafkaController [2016-11-14 01:56:26,639] TRACE [Controller 0]: checking need to trigger partition rebalance (kafka.controller.KafkaController) [2016-11-14 01:56:26,639] DEBUG [Controller 0]: preferred replicas by broker Map(0 -\> Map([applog,0] -\> List(0))) (kafka.controllr.KafkaController) [2016-11-14 01:56:26,639] DEBUG [Controller 0]: topics not in preferred replica Map() (kafka.controller.KafkaController) [2016-11-14 01:56:26,640] TRACE [Controller 0]: leader imbalance ratio for broker 0 is 0.000000 (kafka.controller.KafkaController ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 14, 2016, 3:36pm UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/4 "2016-11-14T15:36:17Z")
Could you share the filebeat log file? ---
**Author:** ![jyp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jyp/32/13114_2.png) [@jyp](https://discuss.elastic.co/u/jyp)\ **Post date:** [November 15, 2016, 1:48am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/5 "2016-11-15T01:48:42Z")
filebeat log is that just recursive below * * * 16-11-15T01:47:26Z INFO Registry file updated. 204 states written. 2016-11-15T01:47:27Z INFO Run prospector 2016-11-15T01:47:27Z INFO Run prospector 2016-11-15T01:47:27Z INFO Run prospector * * * ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 15, 2016, 10:48am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/6 "2016-11-15T10:48:38Z")
hm, interresting. The log basically states, the events have been successfully published to kafka. kafka by default has a script to follow/display a topic: ```auto bin/kafka-console-consumer.sh --zookeeper :2181 --topic applog --from-beginning ``` Anything in logstash logs? ---
**Author:** ![jyp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jyp/32/13114_2.png) [@jyp](https://discuss.elastic.co/u/jyp)\ **Post date:** [November 16, 2016, 12:10am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/7 "2016-11-16T00:10:41Z")
I already try to do that, I don't find the log which i want to send. Also, filebeat maybe send today's log but not previous log about filebeat log which share before. Give a detailed, I restart elastic serach for scaleup server in November. After restarting, elasticsearch don't show the log before restaring. Maybe delete all index of elasticsearch for some reason. So I changed the name of log which made in October and restart filebeat in order to it is recognized as new files . But I just see the log on kibana after restart elasticsearch. I don't see the log in October. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 16, 2016, 7:37am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/8 "2016-11-16T07:37:25Z")
Renaming a file does not update the modification time. Also if the file is in the filebeat registry, it will not be fetched again. If you want to resend all data you must remove the registry file. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 16, 2016, 9:12am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/9 "2016-11-16T09:12:53Z")
How can you tell filebeat did not send anything in the past? Did you check number of events and timestamps in kafka? Any old filebeat logs? Before deleting the registry, can you share a copy? The registry contains collected filenames and offsets? Every component in your chain is quite complex in itself, let's look at each one after another before applying changes to all services at the same time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 2, 2016, 10:09am UTC](https://discuss.elastic.co/t/dont-send-log-to-elastic-search/65787/10 "2016-12-02T10:09:01Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.