Issue of protobuf for logstash codec

Elastic Stack Logstash
1

I am trying to use protobuf for logstash codec. I am following :

However, I cannot start my logstash.

bin/logstash -f logstash-test-1.conf --config.reload.automatic --debug

Here is the error:

...
[2017-09-11T18:46:44,887][DEBUG][logstash.inputs.twitter  ] config LogStash::Inputs::Twitter/@rate_limit_reset_in = 300
[2017-09-11T18:46:44,970][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"protobuf", :type=>"codec", :class=>LogStash::Codecs::Protobuf}
[2017-09-11T18:46:44,974][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@class_name = "Twitter"
[2017-09-11T18:46:44,974][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@include_path = ["/Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb"]
[2017-09-11T18:46:44,974][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@id = "16ab258d8e30f65d80c0bb85353e47740f7c9971-2"
[2017-09-11T18:46:44,975][DEBUG][logstash.codecs.protobuf ] config LogStash::Codecs::Protobuf/@enable_metric = true
[2017-09-11T18:46:44,975][DEBUG][logstash.codecs.protobuf ] Including protobuf file: /Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb
[2017-09-11T18:46:44,982][ERROR][logstash.agent           ] Cannot create pipeline {
:reason=>"Twitter is not a class", 
:backtrace=>[
"/Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb:9:in `(root)'", 
"org/jruby/RubyKernel.java:1040:in `require'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/bundle/jruby/1.9/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:1:in `(root)'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:163:in `require_with_metadata_analysis'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:207:in `require_pb_path'", 
"org/jruby/RubyArray.java:1613:in `each'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:71:in `register'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/local_gems/797f65a3/logstash-codec-protobuf-1.0.3/lib/logstash/codecs/protobuf.rb:71:in `register'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/codecs/base.rb:20:in `initialize'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/pipeline.rb:109:in `plugin'", 
"org/jruby/RubyKernel.java:1079:in `eval'", 
"(eval):12:in `initialize'", "/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/pipeline.rb:72:in `initialize'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/pipeline.rb:156:in `initialize'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/runner.rb:314:in `execute'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/logstash-core/lib/logstash/runner.rb:209:in `run'", 
"/Users/fluency03/Workplace/elastic/logstash-5.5.2-1/lib/bootstrap/environment.rb:71:in `(root)'"]}
[2017-09-11T18:46:44,988][DEBUG][logstash.agent           ] starting agent
[2017-09-11T18:46:44,991][DEBUG][logstash.agent           ] Starting puma
[2017-09-11T18:46:44,991][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2017-09-11T18:46:44,991][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2017-09-11T18:46:44,993][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2017-09-11T18:46:44,994][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2017-09-11T18:46:44,994][DEBUG][logstash.api.service     ] [api-service] start

Why I got the error: Twitter is not a class?

Here is my logstash config:

...
output {
  kafka {
    bootstrap_servers => "localhost:9092"
    codec => protobuf {
      class_name => "Twitter"
      include_path => ['/Users/fluency03/Workplace/protobuf/TwitterSimple.pb.rb']
    }
    topic_id => "blockchain-tweets"
  }
#  stdout {
#    codec => rubydebug
#  }
}

This is only giving error when I use Kafka as output of logstash. When I use Kafka as input and add codec of protobuf, it can successfully load the pb.rb class.

Here is my TwitterSimple.proto:

syntax = "proto3";

message Twitter {
  string message = 1;
}

Here is my generated TwitterSimple.pb.rb after ruby-protoc TwitterSimple.proto:

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

# forward declarations
class Twitter < ::ProtocolBuffers::Message; end

class Twitter < ::ProtocolBuffers::Message
  set_fully_qualified_name "Twitter"

  optional :string, :message, 1
end
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.