Protobuf Codec -> Several classes, How I declared them?

Hello everyone,

I'm making a dynatrace's BusinessTransactions integration with logstash + es.

I have a pb.rb generated file with ruby-protoc with three classes:

#!/usr/bin/env ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!

require 'protocol_buffers'

module Export
  module Bt
    # forward declarations
    class BusinessTransactions < ::ProtocolBuffers::Message; end
    class BusinessTransaction < ::ProtocolBuffers::Message; end
    class BtOccurrence < ::ProtocolBuffers::Message; end

    class BusinessTransactions < ::ProtocolBuffers::Message
      set_fully_qualified_name "export.bt.BusinessTransactions"

      repeated ::Export::Bt::BusinessTransaction, :businessTransactions, 1
      optional :int32, :lostTransactions, 2
      optional :string, :serverName, 3
    end

    class BusinessTransaction < ::ProtocolBuffers::Message
      # forward declarations

      # enums
      module Type
        include ::ProtocolBuffers::Enum

        set_fully_qualified_name "export.bt.BusinessTransaction.Type"

        PUREPATH = 0
        USER_ACTION = 1
        VISIT = 2
      end

      set_fully_qualified_name "export.bt.BusinessTransaction"

      required :string, :name, 1
      optional ::Export::Bt::BusinessTransaction::Type, :type, 2
      optional :string, :application, 3
      repeated :string, :measureNames, 4
      repeated :string, :dimensionNames, 5
      repeated ::Export::Bt::BtOccurrence, :occurrences, 6
      optional :string, :systemProfile, 7
    end

    class BtOccurrence < ::ProtocolBuffers::Message
      set_fully_qualified_name "export.bt.BtOccurrence"

      required :int64, :startTime, 1
      optional :int64, :endTime, 2
      optional :string, :purePathId, 5
      repeated :double, :values, 6
      repeated :string, :dimensions, 7
      optional :string, :actionName, 8
      optional :string, :url, 9
      optional :string, :query, 10
      optional :int64, :visitId, 11
      optional :string, :visitTag, 12
      optional :double, :userExperienceIndex, 13
      optional :bool, :converted, 14
      optional :double, :responseTime, 15
      optional :double, :cpuTime, 16
      optional :double, :syncTime, 17
      optional :double, :waitTime, 18
      optional :double, :suspensionTime, 19
      optional :double, :execTime, 20
      optional :double, :duration, 21
      optional :bool, :failed, 22
      optional :int32, :nrOfActions, 23
      optional :string, :clientFamily, 24
      optional :string, :clientIP, 25
      optional :string, :continent, 26
      optional :string, :country, 27
      optional :string, :city, 28
      optional :int32, :failedActions, 29
      optional :int32, :clientErrors, 30
      optional :bool, :exitActionFailed, 31
      optional :bool, :bounce, 32
      optional :string, :osFamily, 33
      optional :string, :osName, 34
      optional :string, :connectionType, 35
      repeated :string, :convertedBy, 36
      optional :double, :clientTime, 37
      optional :double, :networkTime, 38
      optional :double, :serverTime, 39
      optional :int32, :urlRedirectionTime, 40
      optional :int32, :dnsTime, 41
      optional :int32, :connectTime, 42
      optional :int32, :sslTime, 43
      optional :int32, :documentRequestTime, 44
      optional :int32, :documentResponseTime, 45
      optional :int32, :processingTime, 46
    end

  end
end

In the logstash documentation there is an example with only one class. I tried to make at least these attempts:

  • an array but there was an error that the codec couldn't load the file. This was my trying:

    codec => protobuf
    {
    class_name => [ "Export::Bt::BtOccurrence", "Export::Bt::BusinessTransactions", "Export::Bt:BusinessTransaction"]
    include_path => ['/path/to/file/dyna.pb.rb']
    }

  • Just export Bt without declaring classes
    codec => protobuf
    {
    class_name => "Export::Bt"
    include_path => ['/path/to/file/dyna.pb.rb']
    }

  • And if I declare only the BusinessTransactions class, it makes an array with all the data from the others classes without parsing.
    codec => protobuf
    {
    class_name => [ "Export::Bt::BtOccurrence", "Export::Bt::BusinessTransactions", "Export::Bt:BusinessTransaction"]
    include_path => ['/path/to/file/dyna.pb.rb']
    }

How can I declared the three classes in the codec in order to get parsed the entired message?

This is the Error message that I got with the array attempt:

[2018-03-28T06:50:24,329][ERROR][logstash.codecs.protobuf ] Invalid setting for protobuf codec plugin:

  codec {
    protobuf {
      # This setting must be a string
      # Expected string, got ["Export::Bt::BusinessTransactions", "Export::Bt:BusinessTransaction", "Export::Bt:BtOccurrence"]
      class_name => ["Export::Bt::BusinessTransactions", "Export::Bt:BusinessTransaction", "Export::Bt:BtOccurrence"]
      ...
    }
  }

The Error for "Export::Bt"

[2018-03-28T08:10:38,397][WARN ][logstash.codecs.protobuf ] Couldn't decode protobuf: #<NoMethodError: undefined method `parse' for Export::Bt:Module>.
[2018-03-28T08:10:38,398][ERROR][logstash.inputs.http     ] unable to process event. {:request=>{"request_method"=>"POST", "request_path"=>"/", "request_uri"=>"/", "http_version"=>"HTTP/1.1", "content_type"=>"application/octet-stream", "http_x_dynatrace_version"=>"6.5.31.0", "http_user_agent"=>"Java/1.8.0_102", "http_host"=>"len1ixa1.es.wcorp.carrefour.com:9280", "http_accept"=>"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2", "http_connection"=>"keep-alive", "content_length"=>"2"}, :message=>"undefined method `parse' for Export::Bt:Module", :class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-protobuf-1.0.5/lib/logstash/codecs/protobuf.rb:93:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http-3.0.8/lib/logstash/inputs/http.rb:155:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http-3.0.8/lib/logstash/util/http_compressed_requests.rb:27:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/builder.rb:153:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:557:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:404:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `block in spawn_thread'"]}
[2018-03-28T08:10:41,965][WARN ][logstash.codecs.protobuf ] Couldn't decode protobuf: #<NoMethodError: undefined method `parse' for Export::Bt:Module>.
[2018-03-28T08:10:41,966][ERROR][logstash.inputs.http     ] unable to process event. {:request=>{"request_method"=>"POST", "request_path"=>"/", "request_uri"=>"/", "http_version"=>"HTTP/1.1", "content_type"=>"application/octet-stream", "http_x_dynatrace_version"=>"6.5.31.0", "http_user_agent"=>"Java/1.8.0_102", "http_host"=>"len1ixa1.es.wcorp.carrefour.com:9280", "http_accept"=>"text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2", "http_connection"=>"keep-alive", "content_length"=>"2"}, :message=>"undefined method `parse' for Export::Bt:Module", :class=>"NoMethodError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-protobuf-1.0.5/lib/logstash/codecs/protobuf.rb:93:in `decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http-3.0.8/lib/logstash/inputs/http.rb:155:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-http-3.0.8/lib/logstash/util/http_compressed_requests.rb:27:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/rack-1.6.6/lib/rack/builder.rb:153:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:557:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:404:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:270:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/thread_pool.rb:106:in `block in spawn_thread'"]}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.