Logstash and delete of gz files

[Rhh (Rolf)]

Hi

I have the following my conf file ...

input {
file {
path => "C:/TDS.Extra/ConsoleApp13/ConsoleApp13/bin/Debug/test.gz"
sincedb_path => "nul"
mode => "read"
file_completed_action => "delete"
codec => "json"
}
}

Running logstash on a windows 10 machine. Parse the files fine, BUT the file is not deleted after it is parsed.

logstash run under administrator account and the administrator has write permissions to the folder where test.tz resides.

When I try the same with a txt file it works fine. The file is deleted.

input {
file {
path => "C:/TDS.Extra/ConsoleApp13/ConsoleApp13/bin/Debug/test.txt"
sincedb_path => "nul"
mode => "read"
file_completed_action => "delete"
codec => "json"
}
}

SO IT HAS TO DO WITH GZ FILES.

What am I missing here ?

Is this a known limitation / bug ?

Any help is apprishitated

Regards

[Badger]

@Rhh

Is this a known limitation / bug ?

Read through to the end of this issue.

[Rhh (Rolf)]

I have compared the logs using a txt file and a gz file.

Using a txt file, the file is deleted as expected.

using a gz file the file is not deleted as expected.

The only line different in the logs ( from the gz file log ) is this ...

[2023-02-06T10:40:14,945][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

I read something about auto_flush_interval, but do not understand how this should be used...

ANY HELP ???

This line does not come if I use a txt file.

Here is my conf file for txt files :

input {
  file {
    path => "C:/Debug/debug-*.txt"
    sincedb_path => "nul"
    mode => "read"
    file_completed_action => "delete"
    codec => "json"
  }       
}

filter {   

    grok { match => { "path" => "(?<filename>debug-[A-Z][A-Z]-[0-9]+-[0-9]+.txt)" } }

    date {
      match => ["timestamp", "ISO8601"]
      timezone => "Europe/Oslo"
      locale => "no"
      target => "@timestamp"
    }

    mutate { 
      remove_field => [ "timestamp", "path","port", "@version" ] 
      replace => { "source" => "DEBUG" }
      replace => { "logger" => "LOGSTASH.FTP" }	
      replace => { "host" => "%{host}" }
      uppercase => ["host"]		
    }
}

output {
   stdout {
    codec => rubydebug
   }
   
   elasticsearch {
      hosts => "127.0.0.1:9200"
      manage_template => false
      index => "tds-logs-debug-%{+YYYY.MM.dd}"
   }
  
   file {
     'path' => 'c:/logs/tds-logs-debug-%{+YYYY.MM.dd}.log'
   }
}

Here is my conf file for gz files :

input {
  file {
    path => "C:/Debug/debug-*.gz"
    sincedb_path => "nul"
    mode => "read"
    file_completed_action => "delete"
    codec => "json"
  }       
}

filter {   

    grok { match => { "path" => "(?<filename>debug-[A-Z][A-Z]-[0-9]+-[0-9]+.gz)" } }

    date {
      match => ["timestamp", "ISO8601"]
      timezone => "Europe/Oslo"
      locale => "no"
      target => "@timestamp"
    }

    mutate { 
      remove_field => [ "timestamp", "path","port", "@version" ] 
      replace => { "source" => "DEBUG" }
      replace => { "logger" => "LOGSTASH.FTP" }	
      replace => { "host" => "%{host}" }
      uppercase => ["host"]		
    }
}

output {
   stdout {
    codec => rubydebug
   }
   
   elasticsearch {
      hosts => "127.0.0.1:9200"
      manage_template => false
      index => "tds-logs-debug-%{+YYYY.MM.dd}"
   }
  
   file {
     'path' => 'c:/logs/tds-logs-debug-%{+YYYY.MM.dd}.log'
   }
}

Regards