Read a gzip file with gzip_lines codec

Hi!

I need to use Logstash for parsing *.gz files. I've downloaded gzip_lines gem and installed it with

logstash-1.5.1\bin>plugin install logstash-codec-gzip_lines-0.1.5.gem

the output said "Installation succesful". After I checked plugin list and found logstash-codec-gzip_lines there. But when I run Logstash with such input declaration:

file {
type => "gzip"
path => "C:/run/log.gz"
start_position => "beginning"
sincedb_path => "gzip"
codec => "logstash-codec-gzip_lines"
}

I always have error message:

"Couldn't find any codec plugin named 'logstash-codec-gzip_lines'. Are you sure this is correct? Trying to lo
ad the logstash-codec-gzip_lines codec plugin resulted in this error: no such file to load -- logstash/codecs/
logstash-codec-gzip_lines"

What should I do to make it work?

Regards

The "logstash-codec" part of the gem name is added automatically. Try this:

codec => "gzip_lines"

Magnus,

thank you very much! LS is working now, but plugin still not. Next problem is

←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File type=>"gzip", path=>["C:/run/calypso.log.gz"], start_position=>"beginning", sincedb_path=>"gzip", delimiter=>"\n">
Error: Object: ▼ some crap ??# is not a legal argument to this wrapper, cause it doesn't respond to "read". {:level=>:error}←[0m

Is anyone got the text from gz file?

Regards

Also, I've got the debug:

←[36m_sincedb_open: reading from gzip {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatc
h-0.6.3/lib/filewatch/tail.rb", :line=>"218", :method=>"_sincedb_open"}←[0m
←[36m_sincedb_open: setting ["2719441811-5625-67567616", 0, 2] to 0 {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/
bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"222", :method=>"_sincedb_open"}←[0m
←[36m_discover_file_glob: C:/run/calypso.log.gz: glob is: ["C:/run/calypso.log.gz"] {:level=>:debug, :file=>"/RUN/logsta
sh-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}←
[0m
←[36m_discover_file: C:/run/calypso.log.gz: new: C:/run/calypso.log.gz (exclude is []) {:level=>:debug, :file=>"/RUN/log
stash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/watch.rb", :line=>"141", :method=>"_discover_file
"}←[0m
←[36m_open_file: C:/run/calypso.log.gz: opening {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gem
s/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"119", :method=>"_open_file"}←[0m
←[36mC:/run/calypso.log.gz: sincedb last value 0, cur size 193626 {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bu
ndle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"146", :method=>"_open_file"}←[0m
←[36mC:/run/calypso.log.gz: sincedb: seeking to 0 {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/g
ems/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"148", :method=>"open_file"}←[0m
←[36mReceived line {:path=>"C:/run/calypso.log.gz", :text=>"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x00\xEC\xBDmo\xE4\xC8\xB5
&\xF8y\v\x98\xFF\x10\xDBp\x032n)\xCC\b2\xF8\x92=\x9A\x81*%u\xAB\xBB\xAAK-\xA9\xBA\xC7c\x18\x05*\x93*\xE5\xADT\xA6\x9A\xC
C\xAC*\xB9\xD0\x80\xDD\xBB\x18/0\x17{w\xB1;
\x16\xF0\xCE\x9D\xFD\xB4\x1F\x16X\xDB\xEB\xC6m\xD8\xBE}\xFFB\xD6?\xDA\x13Af*
_"Hf\x16\x19\x91j\xD33\xB7Z\x99\xC9\x88xx\xC88\xE7\xC4sN\x9C\xF8\x05\xB5\b\xDB\xB5\xDC]\x1A \xCBj\x89\xFF\x8F-\xE6\xFF
x12\xFD\xE2\xC9\xD3\xB3N8\x18D1:\xFE\xFC\xE8)\x1A\xE0\xB0\e\xDE\x8C\xA28\xC1\xD7\xE1\x1C\xF6\xF1\xD3\xB8\e\xC5\xD95p\xF D\xDBNx|\xB0G\x98eY\xF6C\xD4\x19\xBD\xC1\a\xD1\xCD0\xE9\x8D\xF6\xFE\xC37\xBFD\xBB\xE8\xF1p\xF8\xB27x\x81.\x871\xEA\f\xAF o\xC28\xC2\x18?\xF8\x85\x14\x81k\xAD\"X\x01\xD0\r\x87\xF8\x89\xF8\xF3\xFFi\t\x04G\xD1\xA8s\x15u\x91\x85n\xA2A\x97c\x19
xF2[HT\x18H\xCDR\b;/\a\xC3\xD7\xFD\xA8\xFB"JrD!\x81Q\xA1(\xC2N'\xBA\x19\x85\x83NdX\x1A\xFC\x9D\xE8G\xA3\xA8\x9B#", :lev
el=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.10/lib/logstash/inputs/file
.rb", :line=>"137", :method=>"run"}←[0m
←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File type=>"gzip", path=>["C:/run/calypso.log.gz"], start_position=>"beginning", sincedb_pa
th=>"gzip", delimiter=>"\n">
Error: Object: ▼ ??mo???&?y♂??►?p♥2n)2??=??%u???K-???c↑♣???T?????????↑/0↨{w?;▬?????▼▬X???m??}?B???:bangbang:Af*"Hf▬
↓?j?3·Z???xx?8??sN??:clubs:???]→ ?j???-??:arrow_up_down:?????N8↑D1:???)→?°←???8???∟????←??5p??Nx|°G?eY?C?↓????0?????7?D???p??7x?.?1?♀?o?8?↑ ??????i :diamonds:G??s§u??n?A?c↓?[HT↑H?;/?????"JrD!?Q?(?N'?↓??NdX→???G???# is not a legal argument to this wrapper, cause
it doesn't respond to "read".
Exception: Java::JavaLang::IllegalArgumentException
Stack: org.jruby.util.IOInputStream.(IOInputStream.java:64)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize(JZlibRubyGzipReader.java:122)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize19(JZlibRubyGzipReader.java:156)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.call(JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.ge
n)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:681)
org.jruby.RubyClass.finvoke(RubyClass.java:549)
org.jruby.runtime.Helpers.invoke(Helpers.java:496)
org.jruby.RubyObject.callInit(RubyObject.java:339)
org.jruby.ext.zlib.JZlibRubyGzipReader.newInstance(JZlibRubyGzipReader.java:85)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.call(JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.gen)

org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
org.jruby.ast.CallOneArgBlockNode.interpret(CallOneArgBlockNode.java:60)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:130)
org.jruby.runtime.Block.yieldSpecific(Block.java:111)
org.jruby.RubyKernel.loop(RubyKernel.java:1511)
org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
org.jruby.ast.FCallOneArgBlockPassNode.interpret(FCallOneArgBlockPassNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhenOneArgNode.when(WhenOneArgNode.java:49)
org.jruby.ast.CaseNode.interpret(CaseNode.java:133)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
org.jruby.ast.FCallNoArgBlockPassNode.interpret(FCallNoArgBlockPassNode.java:27)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:247)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:227)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:211)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
org.jruby.ast.CallTwoArgBlockNode.interpret(CallTwoArgBlockNode.java:62)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
org.jruby.runtime.Block.call(Block.java:101)
org.jruby.RubyProc.call(RubyProc.java:290)
org.jruby.RubyProc.call(RubyProc.java:228)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
java.lang.Thread.run(Thread.java:745) {:level=>:error, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash
-core-1.5.1-java/lib/logstash/pipeline.rb", :line=>"181", :method=>"inputworker"}←[0m
org.jruby.util.IOInputStream.(IOInputStream.java:64)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize(JZlibRubyGzipReader.java:122)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize19(JZlibRubyGzipReader.java:156)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.call(JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.ge
n)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:681)
org.jruby.RubyClass.finvoke(RubyClass.java:549)
org.jruby.runtime.Helpers.invoke(Helpers.java:496)
org.jruby.RubyObject.callInit(RubyObject.java:339)
org.jruby.ext.zlib.JZlibRubyGzipReader.newInstance(JZlibRubyGzipReader.java:85)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.call(JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.gen)

org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
org.jruby.ast.CallOneArgBlockNode.interpret(CallOneArgBlockNode.java:60)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:130)
org.jruby.runtime.Block.yieldSpecific(Block.java:111)
org.jruby.RubyKernel.loop(RubyKernel.java:1511)
org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
org.jruby.ast.FCallOneArgBlockPassNode.interpret(FCallOneArgBlockPassNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhenOneArgNode.when(WhenOneArgNode.java:49)
org.jruby.ast.CaseNode.interpret(CaseNode.java:133)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
org.jruby.ast.FCallNoArgBlockPassNode.interpret(FCallNoArgBlockPassNode.java:27)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:247)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:227)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:211)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
org.jruby.ast.CallTwoArgBlockNode.interpret(CallTwoArgBlockNode.java:62)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
org.jruby.runtime.Block.call(Block.java:101)
org.jruby.RubyProc.call(RubyProc.java:290)
org.jruby.RubyProc.call(RubyProc.java:228)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
java.lang.Thread.run(Thread.java:745)

1 Like

Hi,

I have also got the same requirement, reading .gz file from logstash. Tried with codec gzip_lines, ended up with the same issue unrecoverable error.

Any solution for this? Please do the needfull.

Thanks,
Lakshmi.

Hi,
same requirement and same error.

Any solution?

Thanks,
Max

Same problem here :frowning:

In my case, I found the problem is the input data of gzip_lines. Someone who wants to handle gziped text files should offer the filelist as plugin file's input. This problem happens plugin file provide codec gzip_lines with the data steam of each files. It means codec gzip_lines get a part of a compressed file. The solution is to offer a file which includes file list as plugin file's option path. We need a preprocessor to process gziped files before starting plugin file.

This is an example.

list_file.txt


/path/a.gz
/path/b.gz
/path/c.gz

file {
path => "list_file.txt"
codec => "gzip_lines"
}

1 Like

Thanks for this explanation. Since I'm experiencing the same issue, I gave it another try with your tip.

Unfortunately, it still does not work for me :
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["/home/mducruet/list.txt"], codec=><LogStash::Codecs::GzipLines charset=>"UTF-8">, type=>"akamai-access", debug=>false, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, start_position=>"end", delimiter=>"\n">
Error: Object: /home/akamaiftp/tomsguide_logs/tomsguide_383644.eclf_S.201510310000-2400-0.gz is not a legal argument to this wrapper, cause it doesn't respond to "read". {:level=>:error}

My logstash conf :
input {
file {
path => "/home/mducruet/list.txt"
codec => "gzip_lines"
type => "akamai-access"
}

Content of the list.txt file :
/home/akamaiftp/tomsguide_logs/tomsguide_something.201510310000-2400-0.gz
/home/akamaiftp/tomsguide_logs/tomsguide_something.201510310000-2400-1.gz

I have the same requirement....The logstash reading the gz file but the output is not the "plain" text as in the original file...

I am facing a similar problem, wherein my file is entirely compressed and not line-by-line. Also, the gz files are dynamic, so they cannot be populated in a wrapper file as above. Any solution to get away with this?

Clone Source
git clone https://github.com/logstash-plugins/logstash-codec-gzip_lines

Change gzip_lines.rb
logstash-codec-gzip_lines/lib/logstash/codecs/gzip_lines.rb
line 32 @decoder = Zlib::GzipReader.open(data)

Build Gem File
cd logstash-codec-gzip_lines
gem build logstash-code-gzip_lines.gemspec

Reinstall logstash-codec-gzip_lines
cd logstash
bin/plugin install /opt/logstash-codec-gzip_lines/logstash-codec-gzip_lines-2.0.2.gem

Create list.txt File
Write by yourself gzip_lines.rb and run it. It will create list.txt file.

Sample code:
class GzipLines
attr_accessor :list_path

  public
  def create_list(data_path)
    #unless File.exist?(self.list_path)
    open(self.list_path,"w")
    #end

    folders = entries_list(data_path)

    folders.each do |folder|
      files = file_list(data_path+folder)
      files.each do |file|
        open(self.list_path, "a") { |f| f << data_path+folder+'/'+file.to_s+"\n" }
      end
    end
  end

  def entries_list(basedir)
    Dir.entries(basedir).select {|entry| File.directory? File.join(basedir, entry) and !(entry =='.' || entry =='..')}
  end

  def file_list(folder_path)
    Dir.chdir(folder_path)
    files = Dir.glob('*.gz')
  end
end

    gzip_lines = GzipLines.new
    gzip_lines.list_path = '/opt/logstash/list.txt'
    gzip_lines.create_list('/home/tsanchin/rawdata/')

Config logstash.conf to read list.txt file
file {
type => "gzip"
path => "/opt/logstash/list.txt"
start_position => "beginning"
sincedb_path => "gzip"
codec => gzip_lines
}

2 Likes