Read a gzip file with gzip_lines codec

dima_eam · June 29, 2015, 3:44pm

Hi!

I need to use Logstash for parsing *.gz files. I've downloaded gzip_lines gem and installed it with

logstash-1.5.1\bin>plugin install logstash-codec-gzip_lines-0.1.5.gem

the output said "Installation succesful". After I checked plugin list and found logstash-codec-gzip_lines there. But when I run Logstash with such input declaration:

file {
type => "gzip"
path => "C:/run/log.gz"
start_position => "beginning"
sincedb_path => "gzip"
codec => "logstash-codec-gzip_lines"
}

I always have error message:

"Couldn't find any codec plugin named 'logstash-codec-gzip_lines'. Are you sure this is correct? Trying to lo
ad the logstash-codec-gzip_lines codec plugin resulted in this error: no such file to load -- logstash/codecs/
logstash-codec-gzip_lines"

What should I do to make it work?

Regards

magnusbaeck · June 30, 2015, 6:32am

The "logstash-codec" part of the gem name is added automatically. Try this:

codec => "gzip_lines"

dima_eam · July 2, 2015, 11:49am

Magnus,

thank you very much! LS is working now, but plugin still not. Next problem is

←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File type=>"gzip", path=>["C:/run/calypso.log.gz"], start_position=>"beginning", sincedb_path=>"gzip", delimiter=>"\n">
Error: Object: ▼ some crap ??# is not a legal argument to this wrapper, cause it doesn't respond to "read". {:level=>:error}←[0m

Is anyone got the text from gz file?

Regards

dima_eam · July 2, 2015, 12:07pm

Also, I've got the debug:

←[36m_sincedb_open: reading from gzip {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatc
h-0.6.3/lib/filewatch/tail.rb", :line=>"218", :method=>"_sincedb_open"}←[0m
←[36m_sincedb_open: setting ["2719441811-5625-67567616", 0, 2] to 0 {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/
bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"222", :method=>"_sincedb_open"}←[0m
←[36m_discover_file_glob: C:/run/calypso.log.gz: glob is: ["C:/run/calypso.log.gz"] {:level=>:debug, :file=>"/RUN/logsta
sh-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/watch.rb", :line=>"132", :method=>"_discover_file"}←
[0m
←[36m_discover_file: C:/run/calypso.log.gz: new: C:/run/calypso.log.gz (exclude is []) {:level=>:debug, :file=>"/RUN/log
stash-1.5.1/vendor/bundle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/watch.rb", :line=>"141", :method=>"_discover_file
"}←[0m
←[36m_open_file: C:/run/calypso.log.gz: opening {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gem
s/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"119", :method=>"_open_file"}←[0m
←[36mC:/run/calypso.log.gz: sincedb last value 0, cur size 193626 {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bu
ndle/jruby/1.9/gems/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"146", :method=>"_open_file"}←[0m
←[36mC:/run/calypso.log.gz: sincedb: seeking to 0 {:level=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/g
ems/filewatch-0.6.3/lib/filewatch/tail.rb", :line=>"148", :method=>"open_file"}←[0m
←[36mReceived line {:path=>"C:/run/calypso.log.gz", :text=>"\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x00\xEC\xBDmo\xE4\xC8\xB5
&\xF8y\v\x98\xFF\x10\xDBp\x032n)\xCC\b2\xF8\x92=\x9A\x81*%u\xAB\xBB\xAAK-\xA9\xBA\xC7c\x18\x05*\x93*\xE5\xADT\xA6\x9A\xC
C\xAC*\xB9\xD0\x80\xDD\xBB\x18/0\x17{w\xB1;\x16\xF0\xCE\x9D\xFD\xB4\x1F\x16X\xDB\xEB\xC6m\xD8\xBE}\xFFB\xD6?\xDA\x13Af*
_"Hf\x16\x19\x91j\xD33\xB7Z\x99\xC9\x88xx\xC88\xE7\xC4sN\x9C\xF8\x05\xB5\b\xDB\xB5\xDC]\x1A \xCBj\x89\xFF\x8F-\xE6\xFF
x12\xFD\xE2\xC9\xD3\xB3N8\x18D1:\xFE\xFC\xE8)\x1A\xE0\xB0\e\xDE\x8C\xA28\xC1\xD7\xE1\x1C\xF6\xF1\xD3\xB8\e\xC5\xD95p\xF D\xDBNx|\xB0G\x98eY\xF6C\xD4\x19\xBD\xC1\a\xD1\xCD0\xE9\x8D\xF6\xFE\xC37\xBFD\xBB\xE8\xF1p\xF8\xB27x\x81.\x871\xEA\f\xAF o\xC28\xC2\x18?\xF8\x85\x14\x81k\xAD\"X\x01\xD0\r\x87\xF8\x89\xF8\xF3\xFFi\t\x04G\xD1\xA8s\x15u\x91\x85n\xA2A\x97c\x19
xF2[HT\x18H\xCDR\b;/\a\xC3\xD7\xFD\xA8\xFB"JrD!\x81Q\xA1(\xC2N'\xBA\x19\x85\x83NdX\x1A\xFC\x9D\xE8G\xA3\xA8\x9B#", :lev
el=>:debug, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash-input-file-0.1.10/lib/logstash/inputs/file
.rb", :line=>"137", :method=>"run"}←[0m
←[31mA plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File type=>"gzip", path=>["C:/run/calypso.log.gz"], start_position=>"beginning", sincedb_pa
th=>"gzip", delimiter=>"\n">
Error: Object: ▼ ??mo???&?y♂??►?p♥2n)2??=??%u???K-???c↑♣???T?????????↑/0↨{w?;▬?????▼▬X???m??}?B???Af*"Hf▬
↓?j?3·Z???xx?8??sN?????]→ ?j???-???????N8↑D1:???)→?°←???8???∟????←??5p??Nx|°G?eY?C?↓????0?????7?D???p??7x?.?1?♀?o?8?↑ ??????i G??s§u??n?A?c↓?[HT↑H?;/?????"JrD!?Q?(?N'?↓??NdX→???G???# is not a legal argument to this wrapper, cause
it doesn't respond to "read".
Exception: Java::JavaLang::IllegalArgumentException
Stack: org.jruby.util.IOInputStream.(IOInputStream.java:64)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize(JZlibRubyGzipReader.java:122)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize19(JZlibRubyGzipReader.java:156)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.call(JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.ge
n)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:681)
org.jruby.RubyClass.finvoke(RubyClass.java:549)
org.jruby.runtime.Helpers.invoke(Helpers.java:496)
org.jruby.RubyObject.callInit(RubyObject.java:339)
org.jruby.ext.zlib.JZlibRubyGzipReader.newInstance(JZlibRubyGzipReader.java:85)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.call(JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.gen)

org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
org.jruby.ast.CallOneArgBlockNode.interpret(CallOneArgBlockNode.java:60)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:130)
org.jruby.runtime.Block.yieldSpecific(Block.java:111)
org.jruby.RubyKernel.loop(RubyKernel.java:1511)
org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
org.jruby.ast.FCallOneArgBlockPassNode.interpret(FCallOneArgBlockPassNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhenOneArgNode.when(WhenOneArgNode.java:49)
org.jruby.ast.CaseNode.interpret(CaseNode.java:133)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
org.jruby.ast.FCallNoArgBlockPassNode.interpret(FCallNoArgBlockPassNode.java:27)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:247)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:227)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:211)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
org.jruby.ast.CallTwoArgBlockNode.interpret(CallTwoArgBlockNode.java:62)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
org.jruby.runtime.Block.call(Block.java:101)
org.jruby.RubyProc.call(RubyProc.java:290)
org.jruby.RubyProc.call(RubyProc.java:228)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
java.lang.Thread.run(Thread.java:745) {:level=>:error, :file=>"/RUN/logstash-1.5.1/vendor/bundle/jruby/1.9/gems/logstash
-core-1.5.1-java/lib/logstash/pipeline.rb", :line=>"181", :method=>"inputworker"}←[0m
org.jruby.util.IOInputStream.(IOInputStream.java:64)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize(JZlibRubyGzipReader.java:122)
org.jruby.ext.zlib.JZlibRubyGzipReader.initialize19(JZlibRubyGzipReader.java:156)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.call(JZlibRubyGzipReader$INVOKER$i$0$1$initialize19.ge
n)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:681)
org.jruby.RubyClass.finvoke(RubyClass.java:549)
org.jruby.runtime.Helpers.invoke(Helpers.java:496)
org.jruby.RubyObject.callInit(RubyObject.java:339)
org.jruby.ext.zlib.JZlibRubyGzipReader.newInstance(JZlibRubyGzipReader.java:85)
org.jruby.ext.zlib.JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.call(JZlibRubyGzipReader$INVOKER$s$0$0$newInstance.gen)

org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:206)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:188)
org.jruby.ast.CallOneArgBlockNode.interpret(CallOneArgBlockNode.java:60)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:130)
org.jruby.runtime.Block.yieldSpecific(Block.java:111)
org.jruby.RubyKernel.loop(RubyKernel.java:1511)
org.jruby.RubyKernel$INVOKER$s$0$0$loop.call(RubyKernel$INVOKER$s$0$0$loop.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.FCallNoArgBlockNode.interpret(FCallNoArgBlockNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:204)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:211)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:177)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
org.jruby.ast.FCallOneArgBlockPassNode.interpret(FCallOneArgBlockPassNode.java:32)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhenOneArgNode.when(WhenOneArgNode.java:49)
org.jruby.ast.CaseNode.interpret(CaseNode.java:133)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:177)
org.jruby.runtime.Interpreted19Block.yieldSpecific(Interpreted19Block.java:140)
org.jruby.runtime.Block.yieldSpecific(Block.java:129)
org.jruby.ast.YieldTwoNode.interpret(YieldTwoNode.java:31)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.IfNode.interpret(IfNode.java:116)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:157)
org.jruby.runtime.Block.yield(Block.java:142)
org.jruby.RubyArray.eachCommon(RubyArray.java:1606)
org.jruby.RubyArray.each(RubyArray.java:1613)
org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:149)
org.jruby.ast.FCallNoArgBlockPassNode.interpret(FCallNoArgBlockPassNode.java:27)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:247)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:227)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:211)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:222)
org.jruby.ast.CallTwoArgBlockNode.interpret(CallTwoArgBlockNode.java:62)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:161)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:195)
org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:143)
org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:154)
org.jruby.ast.CallNoArgBlockNode.interpret(CallNoArgBlockNode.java:64)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.RescueNode.executeBody(RescueNode.java:221)
org.jruby.ast.RescueNode.interpret(RescueNode.java:116)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.EnsureNode.interpret(EnsureNode.java:96)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:182)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:203)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:206)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:194)
org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:125)
org.jruby.runtime.Block.call(Block.java:101)
org.jruby.RubyProc.call(RubyProc.java:290)
org.jruby.RubyProc.call(RubyProc.java:228)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:99)
java.lang.Thread.run(Thread.java:745)

Lakshmi · September 11, 2015, 4:34am

Hi,

I have also got the same requirement, reading .gz file from logstash. Tried with codec gzip_lines, ended up with the same issue unrecoverable error.

Any solution for this? Please do the needfull.

Thanks,
Lakshmi.

MaxCor · October 19, 2015, 4:29pm

Hi,
same requirement and same error.

Any solution?

Thanks,
Max

paulkeogh · October 30, 2015, 3:12pm

Same problem here

bar4mi · November 19, 2015, 9:47pm

In my case, I found the problem is the input data of gzip_lines. Someone who wants to handle gziped text files should offer the filelist as plugin file's input. This problem happens plugin file provide codec gzip_lines with the data steam of each files. It means codec gzip_lines get a part of a compressed file. The solution is to offer a file which includes file list as plugin file's option path. We need a preprocessor to process gziped files before starting plugin file.

This is an example.

list_file.txt

/path/a.gz
/path/b.gz
/path/c.gz

file {
path => "list_file.txt"
codec => "gzip_lines"
}

Manu · November 24, 2015, 1:44pm

Thanks for this explanation. Since I'm experiencing the same issue, I gave it another try with your tip.

Unfortunately, it still does not work for me :
A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::File path=>["/home/mducruet/list.txt"], codec=><LogStash::Codecs::GzipLines charset=>"UTF-8">, type=>"akamai-access", debug=>false, stat_interval=>1, discover_interval=>15, sincedb_write_interval=>15, start_position=>"end", delimiter=>"\n">
Error: Object: /home/akamaiftp/tomsguide_logs/tomsguide_383644.eclf_S.201510310000-2400-0.gz is not a legal argument to this wrapper, cause it doesn't respond to "read". {:level=>:error}

My logstash conf :
input {
file {
path => "/home/mducruet/list.txt"
codec => "gzip_lines"
type => "akamai-access"
}

Content of the list.txt file :
/home/akamaiftp/tomsguide_logs/tomsguide_something.201510310000-2400-0.gz
/home/akamaiftp/tomsguide_logs/tomsguide_something.201510310000-2400-1.gz

mvenkat_in · December 3, 2015, 12:08am

I have the same requirement....The logstash reading the gz file but the output is not the "plain" text as in the original file...

Pritesh_Lahoti · December 5, 2015, 12:04pm

I am facing a similar problem, wherein my file is entirely compressed and not line-by-line. Also, the gz files are dynamic, so they cannot be populated in a wrapper file as above. Any solution to get away with this?

TC_Huang · January 12, 2016, 9:42am

Clone Source
git clone https://github.com/logstash-plugins/logstash-codec-gzip_lines

Change gzip_lines.rb
logstash-codec-gzip_lines/lib/logstash/codecs/gzip_lines.rb
line 32 @decoder = Zlib::GzipReader.open(data)

github.com

logstash-plugins/logstash-codec-gzip_lines/blob/master/lib/logstash/codecs/gzip_lines.rb

# encoding: utf-8
require "logstash/codecs/base"
require "logstash/codecs/plain"
require "logstash/json"
require "zlib"

# This codec will read gzip encoded content
class LogStash::Codecs::GzipLines < LogStash::Codecs::Base
  config_name "gzip_lines"


  # The character encoding used in this codec. Examples include "UTF-8" and
  # "CP1252"
  #
  # JSON requires valid UTF-8 strings, but in some cases, software that
  # emits JSON does so in another encoding (nxlog, for example). In
  # weird cases like this, you can set the charset setting to the
  # actual encoding of the text and logstash will convert it for you.
  #
  # For nxlog users, you'll want to set this to "CP1252"

This file has been truncated. show original

Build Gem File
cd logstash-codec-gzip_lines
gem build logstash-code-gzip_lines.gemspec

Reinstall logstash-codec-gzip_lines
cd logstash
bin/plugin install /opt/logstash-codec-gzip_lines/logstash-codec-gzip_lines-2.0.2.gem

Create list.txt File
Write by yourself gzip_lines.rb and run it. It will create list.txt file.

Sample code:
class GzipLines
attr_accessor :list_path

  public
  def create_list(data_path)
    #unless File.exist?(self.list_path)
    open(self.list_path,"w")
    #end

    folders = entries_list(data_path)

    folders.each do |folder|
      files = file_list(data_path+folder)
      files.each do |file|
        open(self.list_path, "a") { |f| f << data_path+folder+'/'+file.to_s+"\n" }
      end
    end
  end

  def entries_list(basedir)
    Dir.entries(basedir).select {|entry| File.directory? File.join(basedir, entry) and !(entry =='.' || entry =='..')}
  end

  def file_list(folder_path)
    Dir.chdir(folder_path)
    files = Dir.glob('*.gz')
  end
end

    gzip_lines = GzipLines.new
    gzip_lines.list_path = '/opt/logstash/list.txt'
    gzip_lines.create_list('/home/tsanchin/rawdata/')

Config logstash.conf to read list.txt file
file {
type => "gzip"
path => "/opt/logstash/list.txt"
start_position => "beginning"
sincedb_path => "gzip"
codec => gzip_lines
}

Topic		Replies	Views
Log file decode error Logstash	3	1275	June 14, 2017
How to efficiently parse log files in logstash? Logstash	3	421	July 16, 2018
Error in installing logstash-plugin Logstash	8	2686	July 4, 2017
Logstash for application.log + application.log.1.gz Logstash	6	152	March 1, 2024
Logstash using codec line is not working Logstash	2	232	August 4, 2023

Read a gzip file with gzip_lines codec

/path/a.gz /path/b.gz /path/c.gz

Related Topics

/path/a.gz
/path/b.gz
/path/c.gz