Using gzip_line codec

[paresh_Panda]

Hi
I am trying to evaluate the gzip_line codec . Below is my use-case.

I have my application trace files in CSV format and as days go it is gziped to gz files. Now I want to feed the gz files to logstash where I can have CSV filter and ultimately feed to elastic index. When I do that I see logstash is throwing exception.

input {
file {
type => "gzip"
sincedb_path => "/dev/null"
start_position => beginning
codec => gzip_lines
path => "/home/ppanda/logs/nRData/rordata*gz"
}
}

I can see that files are getting parsed but after that it is throwing exception.

[2018-07-18T18:39:43,967][DEBUG][logstash.inputs.file ] _globbed_files: /home/ppanda/SiteLogs/ntrRomerData/*gz: glob is: ["/home/ppanda/SiteLogs/ntrRomerData/roamerdata.rts1.24001.20180527-040423.log.gz"]
[2018-07-18T18:39:43,969][DEBUG][logstash.inputs.file ] _discover_file: /home/ppanda/SiteLogs/ntrRomerData/*gz: new: /home/ppanda/SiteLogs/ntrRomerData/roamerdata.rts1.24001.20180527-040423.log.gz (exclude is [])
[2018-07-18T18:39:43,975][DEBUG][logstash.inputs.file ] _open_file: /home/ppanda/SiteLogs/ntrRomerData/roamerdata.rts1.24001.20180527-040423.log.gz: opening
[2018-07-18T18:39:43,976][DEBUG][logstash.inputs.file ] /home/ppanda/SiteLogs/ntrRomerData/roamerdata.rts1.24001.20180527-040423.log.gz: initial create, no sincedb, seeking to beginning of file
[2018-07-18T18:39:43,978][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/ppanda/SiteLogs/ntrRomerData/roamerdata.rts1.24001.20180527-040423.log.gz", :text=>"\x1F\x8B\b\b&\xA8N[\x00\x03roamerdata.rts1.24001.20180527-040423.log\x00\xEC=\xD9r\xDBH\x92\xEF\xFB\x15|\xDA\xD8\x8D\x059u\x1F~S\xFB\xE8q\x8C\xED\xEE\xB0\xD43\xB3\xFB\xD2A\x91\x90M[$\xB5$%\xB7\x1D\xFA\xF8\xC9\x04\t\xA0\x12(\x80\xA0L\xC8\xAD\t\x1A\x96\xAD"\x8A@"+\xEF\xCALL\x97\x93\xB3\xCDf5\xBB\xBC\xDD\xA4<\x99\x06#AF\x92\x8C\x14\x19i22dd\xC9\xC8\x91\x91'#\xCE\xE8\x90B\xC3)8\x9C\xC2\xC3)@\x9CB\xC4)H\x9C\xC2\xC4)P\x9CB%(T\xA2\x82#"}
[2018-07-18T18:39:43,979][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2018-07-18T18:39:43,986][DEBUG][logstash.inputs.file ] observe_read_file: gen9u~SûèqŒíî°Ô3³ûÒA‘ÂÃ)@œBÄ)HœÂÄ)PœB%(T¢‚# is not a legal argument to this wrapper, cause it doesn't respond to "read".IllegalArgu[2018-07-18T18:39:43,986][DEBUG][logstash.inputs.file ] writing sincedb (delta since last write = 1531919383)
[2018-07-18T18:39:43,988][DEBUG][logstash.inputs.file ] each: file grew: /home/ppanda/SiteLogs/nRData/rordata.rts1.24001.20180527-040423.log.gz: old size 0, new size 997763
[2018-07-18T18:39:43,988][DEBUG][logstash.agent ] Starting puma
[2018-07-18T18:39:43,988][DEBUG][logstash.inputs.file ] Received line {:path=>"/home/ppanda/SiteLogs/nRData/rordata.rts1.24001.20180527-040423.log.gz", :text=>"\x99(\xA8b\x96\xA0\x8A#\xA0\a\x05\xE4\x14\x93\xB1\x1D\x01\xDD\xD2\xABf\xE9iG\xCE\xA9\x96\xD9bY\x96\aX>\xB0\x9F\xE6,\x18\xB0"w\x8E\fQrD^Qr\xC431\xF1:qdU\xD7\xDCi\x95\xA7\xDCg\xCB\xA4\xB3\x8EP\xA2\x06\xE3\xC8&\xDF\xF2\x188\xB2 Z\x89\xB1\x92a\x03z>\x83\xF0\t\x80\xEA<\x1A9G{Z.J\xC0,5&\xF9\xF1\x81Y\x16@\xF2\x89\x01\xC9\xC3\xEFz \xF9\xB8\xA6\x9B!\x87\xB5\x18\xAE\x81\x92ix\xB2\xBE\x10:\x94\xD0\xCB\xCByl\x12\xE5\xFD\xE3R\x188\x8E]\e\xD7\x1E\xFF8\x17\xC6\x85\xA1[h\v\xE2\bF\xF5\fe\xB2\xFB4\x87\xE9\xAF\x8D=Raw\xA1ziDu\xC3\bD\x06\xF70Z\xF4\xB2{\x16\xAA\xAF\xCDF]\x80M1\x8FEmI\x80\x15b\x10\x82\xA9\x1C\x1E\xB3Q\xF5\x92]\xAA\x97\x02\xCA\xC0y\xC9\xB9\xD2u\x98\xE2{;D\x93\xC4\xEC\xC9\xC2\xB18\x9B\xF5\xF7\xF5w\x84\xCC\xD2\xEFh]rO\xAFU18\xF4P\x15\x03\xFA\x8DrBp\xE7\xC8&\xD8\x87\x02?X\xB0\x89\xA7Q\xCA,\xF5\xA7\x10\xECdf;\xAA\x10qn<Z\xF6\xF1\xDA2\xEF\x1D7V(\xEE\x10Sv\xEE3]\x84\eD\xA1\x8D\xFE\xD4\x96w\xDC\xCC\x0E\x0E0\x83k\xBC\xED[\xAC\xBE\xFA8\xD7\xF8\x1C\x8E\xEF\xDD\x12\xB4;\xB9\xE5#ly\x9Bm ^a\xF34\xDA^\xFF\x06\xBA\xA2a\x91\xFAR\xEC\x1C\x94L;\xD31\xCD\x03\xEE\xEC)2\xC6\xB9\xF1h\xD9\xDE!\x01\x99\xB0Bj^ND\xB3\\'\xA3\x85\x80k+\x02\xAB\x1Fv\xAD\xE8\xA8t\xB5Q\x03Q\xBB4\x10\xAD\b\xE8\xD1g\xB8\xB0\xCC\xB8\xD6R\\\x82\xB6\x0Fj7\xA2\x9F\xEE^\x9E\x1C0\xBB|\xE5\xD6\x01\xA3\xA3\x03&\xF2/\xEE~=\xF8 \x01\x9BS\xC1'U\xAB7\xBF\xBCDG\b\xE8\x85\xAF\xEE\xEE\x1F?\xBC\xDD\\7E]W\x0FgX\xD4\x05\xB7\xBA\xE2\xAA\x02L3\xE2~\x82/\xA6\xE2\xE3#3\x87\x19\x11\xBB\x1D\xB3\xEFcc[p7\xC197\x10\xEA^\x82\xB1\x87\xC8\x0Fx\xD0X\r;|\xF7F\xEC\x94a\xAE\b\xDE\x81\x83\x06\xB6?S\xF6s\xFC%q\x80d\b?t\xE6\xAC\xA1Eus?\xA4\xAFp\xEDi\x9B\x15\x1A\x87F\xC3-\xF5u\xB69w\x95\x00\xF3\x9C5\xD2\xE31Hi\x98\xA1\xE1\xA3\x93\x9D.M2z\";\xFDt\xCE\x9A\x17[g=\xE3iM\xE1\xE8q\xF1=\xF4q\x9F\xDD\xDD\x9F6\x15\xEE-K\xD0\xCAB\xF3\xCF\xB9,AL%\x80m\x0F\xFFC+\xF0\xE1qs_\xA7\xA3q\xBE\xFA~{\xFD\xF6\xEA\xC3M\xA3=\xDE\x7Fxx\xD8^?\xE0\xAD>+\x90\xF3S\xCE\x1D\xBF\x83u\a\x16\x9E\xB4\xECT\x8A5\xD6\t"} [2018-07-18T18:39:43,990][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600} [2018-07-18T18:39:43,990][DEBUG][logstash.inputs.file ] observe_read_file: general error reading /home/ppanda/SiteLogs/ntrRomerData/roamerdata.rts1.24001.20180527-040423.log.gz - error: java.lang.IllegalArgumentException: Object: ™(¨b Š# 䓱ÝÒ«féiGΩÙbYX>°Ÿæ,°"w QrD^QrÄ31ñ:qdU×Üi•§Ügˤ³P¢ãÈ&ßò8² Z‰±’az>ƒð €ê<9G{_Z.JÀ,5&ùñY@ò‰ÉÃïz *ù¸¦›!‡µ®’ix²¾:”ÐËËylåýãR8]e×ÿ8Æ ¡[h Fõ =Raw¡ziDuD÷0Zô²{ª¯ÍF]€M1EmI€b‚©³Qõ’]ª*ÊÀyɹÒu˜â{;D“Äìɱ8›õ÷õw rBpçÈ&؇?X°‰§QÊ,õ§ìdf;ªqn<ZöñÚ2ï7V(îSvî3] e²û4‡é¯ ÌÒïh]rO¯U18ôPú þÔwÜÌ0ƒk¼í[¬¾ú8×øïÝ´;¹å#ly ^aó4Ú^ÿº¢a‘úRì”L;Ó1Íîì)2ƹñhÙÞ!™°Bj^ND³'£ ¡
€k+«v­è¨tµQQ»4èÑg¸°Ì¸ÖR\‚¶j7¢Ÿî^ž0»|åÖ££&ò/î~=ø
Á'U«7¿¼Dè
;|÷Fì”aށƒ¶?Sösü%q€?t次Eus?¤¯píi›‡FÃ-õu¶9w•óœ5Òã1Hi˜¡á£“.M2z";ýtÎ[g=ãiMáèqñ=ôqŸÝݟ6î-KÐÊBóϹ,AL%€mÿC+ðáqs§£q¾ú~{ýöêÃM£=ÞxxØ^?à­>+óSοƒuž´ìTŠ5Ö is not a legal argument to this wrapper, cause it doesn't respond to "read".

Can somebody please help on this ?

Regards
Paresh