Error with msgpack input codec : TypeError: wrong argument type Fixnum (expected Hash)

Elastic Stack Logstash
1

Hello,

I am trying to decode msgpack serialized events from a Kafka instance, by using the msgpack input codec. This the configuration file i'm using :

input {
   kafka {
    bootstrap_servers => "${KAFKA_BOOTSTRAP_SERVERS}"
    topics => ["${KAFKA_TOPIC}"]
    codec => msgpack
  }
}

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "${ELASTICSEARCH_INDEX}"
        codec => "json"
    }
}

I have the following error when running logstash :

[2018-10-11T10:34:12,531][WARN ][logstash.codecs.msgpack  ] 
Trouble parsing msgpack input, falling back to plain text 
{:input=>"��test�value", :exception=>#<TypeError: wrong argument type Fixnum (expected Hash)>}
 [2018-10-11T10:34:12,644][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `set' for nil:NilClass
Did you mean?  send>, :backtrace=>["/usr/local/Cellar/logstash/6.4.2/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-codec-msgpack-3.0.7-java/lib/logstash/codecs/msgpack.rb:36:in `decode'", "/usr/local/Cellar/logstash/6.4.2/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.1.1/lib/logstash/inputs/kafka.rb:256:in `block in thread_runner'", "/usr/local/Cellar/logstash/6.4.2/libexec/vendor/bundle/jruby/2.3.0/gems/logstash-input-kafka-8.1.1/lib/logstash/inputs/kafka.rb:255:in `block in thread_runner'"]}

I noticed that the input data "��test�value" which is a simple like this {"test": "value"} is printed weirdly in the console. Whereas I tested to do a quick Python consumer to retrieve this same data from Kafka and everything seems correct when printing it to the console : b'\x81\xa4test\xa5value'.

This make me think the problem come from the way Logstash is encoding data when retrieving it from Kafka, maybe it need an extra step to decode the data before using the msgpack codec ?

Also what do this error "TypeError: wrong argument type Fixnum (expected Hash)" means ?

Additional informations :

  • Logstash version : 6.4.2
  • Operating system : Macos 10.13.4

Thank you for your help

UPDATE 1 :

I tried to replace my kafka input by a simple file input (just to be sure it's not coming from the kafka plugin). I made a Python script writing msgpack serialized data to a file and run with logstash the following configuration :

input {
  file {
    path => "input.log"
    stat_interval => 15
    codec => msgpack
  }
}

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
         codec => "json"
    }
}

And it work as expected ! My data is stashed into a well formated json in elasticsearch.

So it means the problem come from the way the kafka plugin retrieve the data ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.