Logstash runs on the linux container and extremely slow

[ranjini (ranjiniganeshan)]

logstash runs on linux container. Below is my configuration. It is very slow. Sharing my configuration for reference.

This is my service configuration.

file {
        path => "/common/logs/**/*.log"
        start_position => "beginning"
        sincedb_path => "/common/logs/.sincedb"
        type => "default"
        codec => multiline {
          # Grok pattern names are valid! :)
          pattern => "^%{TIMESTAMP_ISO8601}\|"
          negate => true
          what => previous
          max_lines => 1000
        }
 filter {
      if [type] =~ "common" {
        # The bootstrap log does not follow any specific format
      } else if [type] =~ "default" {
        grok {
          match => {"message" => "%{TIMESTAMP_ISO8601}\|%{LOGLEVEL:level}\|%{DATA:service}\|%{DATA:env}\|%{DATA:dc}\|%{HOSTNAME:host}\|%{DATA:class}\|%{DATA:tenant}\|%{IP:ip}\|%{GREEDYDATA:msg}"}
        }
        mutate{
          lowercase=>["service"]
        }
      }
 }

the log file in this location is written every 1 minute.but logs in kibana is getting loaded once in 30 minutes or 40 minutes.

starting logstash using this command

 /usr/share/logstash/bin/logstash --path.settings=/usr/share/logstash/config -f /usr/share/logstash/conf.d/ -w 2

• There is no error in .sincedb error file.
  In logstash-plain.log as well no errors are reported.

[Badger]

Read this blog post. If that configuration is slow it is most likely because grok is slow. grok will be slow when it fails to match DATA or GREEDYDATA correctly and has to backtrack and start over. I would suggest replacing %{DATA:service} with (?<service>[^|]*) and likewise for env, dc, etc.

[ranjini (ranjiniganeshan)]

Thank you Badger. That help.
I could see below error in the container.

[2023-12-14T11:03:25,013][WARN ][filewatch.tailmode.processor][main][993ec5e5da348f46223f1ca5d2f39d980341bc3b6c17be730f7dc2ed5b7db313] >>> Rotation In Progress - inode change detected and original content is not fully read, file is closed and path points to new content {:watched_file=>"<FileWatch::WatchedFile: @filename='status-cog-1.1.1553-74766c4c6d-fnhqg', @state=:rotation_in_progress, @recent_states=[], @bytes_read=0, @bytes_unread=0, current_size=4515709, last_stat_size=4515709, file_open?=false, @initial=true, sincedb_key='138215548 0  66305'>"}

What does this mean?
Is logstash sending the purged file? How does this handle purged file?
I don't see sincedb file created for this service which I m not receiving in kibana.

I will improve the groks as suggested.

[Badger]

@ranjini

Rotation In Progress

If filewatch detects that a rotation is in progress (i.e. my.log.1 is being renamed to my.log.2, my.log is being renamed to my.log.1 etc.) then if the code currently has a file handle open to read my.log it will simply read the whole of the rest of the file before closing it. If it doesn't have an open file handle then it logs that warning to tell you that it will not finish reading my.log.1 after the rotation.

A file input looks for logs by name, but tracks them in the sincedb by inode, so it it calls stat and the inode for my.log has changed then it knows there has been a file rotation.