Logstash - .gz files - Error: Unexpected end of ZLIB input stream

Elastic Stack Logstash
1

Hi,

I am using logstash to read .gz files. The gzip files are around 10 to 15GB for which logstash was unable to process and crashing. So I have split the .gz files using 7zip split based on size of 3GB .gz file chunks. i see logstash easily processing so far except the issue that it cannot determine the EOF which is significant in read mode to process .gz files. and logstash keep processing the same file and not reading the other files specified in path.

Error: Unexpected end of ZLIB input stream

  • Exception: Java::JavaIo::EOFException*
  • Stack: java.util.zip.InflaterInputStream.fill(java/util/zip/InflaterInputStream.java:245)*
    ava.util.zip.InflaterInputStream.read(java/util/zip/InflaterInputStream.java:159)
    ava.util.zip.GZIPInputStream.read(java/util/zip/GZIPInputStream.java:118)

Any help is greatly appreciated. Thanks.

2

Here is the logstash.conf file for your reference.
input {
file {
path => "F:/Export_Group/May-2020_export/2020-05-29/*.gz"
start_position => beginning
sincedb_path => "./logs/SinceDB.out"
mode => "read"
file_completed_action => "log"
file_completed_log_path => "./logs/FilesTracking.log"
}
}

filter {
  sleep {
    time => "1"   # Sleep 1 second
    every => 4000   # on every 10th event
  }
}

output {
syslog {
host => "xx.xx.xx.xx"
port => 514
protocol => "udp"
codec => line { format => "xxx-LTE:%{message}" }
}
stdout{codec => rubydebug}
}

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

4

Hope it helps for someone in the future.

This error means your .gz is corrupted or write stream didn't close it yet. We may skip such cases when error happens but you may miss the data to the outputstream. So, suggestion will be to fix the corrupted file or waif till processing file is closed.