Logstash Index problem after logrotate

Elastic Stack Logstash
1

Logstash is having problem to read log files after log rotation and send it to Elasticsearch.
This issue occures when a log file with only one line of log is getting rotated and the new log file also has only one line.

Is there anything i can do? any Idea ?

Thanks
Vafa

2

What does your configuration look like? How are the files rotated (truncated or renamed)?

3

Hi Magnus,
Files are truncatated (gziped)
After log rotation, only one line is written into the file.

My problem is , that that single line is not read by logstash.

can start_position be a problem ?

Below is logstash pipeline config

input {
  file {
    path => "/logs/wpdirect/wpdirect/server1/wpdirectabnahmetest/1/wpdirect_order.log"
    type => "wpdirect"
    start_position => "end"
    codec => multiline {
                   pattern => "^20"
                   negate => true
                   what => previous
    }
  }
}

filter {
  if [type] =~ /^wpdirect$/ {
        grok {
                match => { "path" => "/logs/wpdirect/wpdirect/dwp(wpd|tuv|tus|tuh|)app(?<stage>[^/?0?]+)" }
        }
        mutate {
                add_field => { "customer" => "dwpbank" }
                add_field => { "app" => "wpdir" }
                convert => { "logtype" => "string" }
                convert => { "loglevel" => "string" }
                convert => { "path" => "string" }
                convert => { "stage" => "string" }
                convert => { "user" => "string" }
                convert => { "path" => "string" }
    }
  }
  grok {
      match => { "path" => "/logs/wpdirect/wpdirect/(?<host>[^/]+)/" }
  }
  if ("wpdirect_order.log" in [path])  {
     mutate {
        convert => { "thread" => "string" }
        convert => { "integration" => "string" }
        convert => { "class" => "string" }
        convert => { "orderid" => "integer" }
    }
    grok {
        patterns_dir => [ "/usr/share/logstash/patterns" ]
        match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp};%{LOGLEVEL:loglevel}(.*);%{OPTIONALUSERNAME:user};%{TNR:tnr};%{INTEGRATION:integration};%{THREAD:thread};%{CLASS:class};%{CLASS:irgendwas};%{NUMBER:orderid:int}$" }
        add_field => [ "logtype", "wpdirect_order" ]
    }
  }
  date {
    match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-d HH:mm:ss,SSS"]
  }
  
  }
  
  output {
    redis {
      id => "output_redis"
      data_type => "list"
      host => ["REDIS-SERVER:6379" ]
      key => "logstash"
    }
    stdout { codec => json }
  }
4

Yes, try using start_position => "beginning" instead.

5

Thanks Magnus, It works now.

6

Hi Magnus

I found another problem after changing start_position to beginning.

It seems that Logstash loses position of last read offset when ever we have to restart the instance.

Is this a normal behavior or it is a bug ?

Thanks
Vafa

7

That's not normal behavior. Increasing the log level will make Logstash emit more messages related to sincedb and you can e.g. verify whether Logstash saves the correct offset upon shutdown.

8

I am not using sincedb in this configuration, because due to this help page , sincedb will be created in home dir of the user rungs logstash.

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files

I can't find any sincedb file.

What shall i do, shall i create a sincedb_path for each file inside inplut plugin ?

Thanks in advance for your support.

Regards
Vafa

9

Well, if you disable sincedb then Logstash will of course start from the beginning each time. If the default sincedb location isn't suitable (and overriding the HOME variable for Logstash isn't an option) you'll have to set sincedb_path for each file input.

10

Ok, this means i should use a sincedb for every file
path=> "file1"
sincedb_path => "sincedb_file1"

Shall i still use start_position => "beginning" ?

I ask beause now everytime make a config change and restart logstash, it start to read files from beginning

11

Ok, this means i should use a sincedb for every file
path=> "file1"
sincedb_path => "sincedb_file1"

Yes.

Shall i still use start_position => "beginning" ?

Yes.

12

Thanks Magnus,
I did this but I still have the problem after logrotate , when the file is only having one line of log.

This application most of the time writes only one sigle lineto the file after every logrotate.
Logs are looking like this:

2017-11-25 02:00:01,022;INFO ;NA;NA;;[Timer-0];AdminFacade;Logging started

After log rotate this single line of log is not read by Logstash and also not indexed.

But if we write a second line of log , than it will be read by Logstash and indexed by ES.

Any Idea ?

Thanks in advance

13

How are the files being rotated? What's the logrotate configuration?

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.