Howto retry from begining the today log file into es

Elastic Stack Beats
1

howto retry from begining the today log file into es

when i run follow command

#rm -rf filebeat/data/registry
#restart filebeat service

the result:
ES display curent time not today log file begining line

2

Then you maybe filter per @timestamp. @timestamp is always the current UTC time the log was read and not the time the log was created. What you can do is to send your logs to logstash and then filter by a field which contains the timestamp the log was created. See here

3

the file of registry that record the log file offset num every time

so i delete the registry file

4

Ok, sorry. I thought you mean the timestamp is not correct. Can please post your filebeat config and maybe the filebeat log.

5

i mean curent time logfile : nginx_perf.log

i want though logstash import ES yesterday log file and two days ago log file and so on
eg: nginx_perf.2017-04-15.log nginx_perf.2017-04-17.log

between 20170415 and 20170417 our cluster crash
so log file hasn't import ES cluster

now we want import old log into ES cluster

6

What you can do is to set ignore_older in your filebeat.yml. For example if you want all logs since 20170415 set this to 96h or a time range back from now where youre cluster has crash. See here for more info.

7

my way is that we create logstash and filebeat another port used history log file

but elasticserach don't display content

8

I think the best way is to delete all registry files and use the ignore_older option instead of using and old registry file. @ruflin, do you have any thoughts?

9

very sorry i have to ask your some silly question

i feel very insteresting

kibana portal discover item display Diagram

original log file format :

2017-04-20 15:00:30 INFO - 0.0.0.0 1349 domain xxxxxx 1 0
2017-04-20 15:00:30 INFO - 0.0.0.0 1463 domain xxxxxx 1 0
2017-04-20 15:00:30 INFO - 0.0.0.0 1459 domain xxxxxxxx 1 0
2017-04-20 15:00:30 INFO - 0.0.0.0 25607 domain xxxxxxx 0 0

my filebeat.yml :
filebeat.prospectors:

  • input_type: log
    paths:
    • /home/ops/logs/perf.log
      exclude_lines: ["DEBUG -"]
      fields:
      logtype: "webservice"

output.logstash:
hosts: ["1.1.1.2:5043"]

kibana display dataset

    time                                                       datetime 
    April 20th 2017, 15:26:59.143  2017-04-20 15:11:03 
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03
April 20th 2017, 15:26:59.143	2017-04-20 15:11:03

first Column is curent time second Column is from begining line read log file content time so kibana display icon from begging line until current line concentrated in curent time

10

Not 100% sure I fully understand the problem. But if you just want to reship the data from one log file, you have 2 options. Follow the approach suggested by @maddin2016 with removing the registry and use ignore_older, or you edit the registry file yourself when filebeat is not running and remove the file you want to reship. But make sure the content stays valid json.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.