Codec json is not working

(JMC) #1

I am trying to read data from Kafka (fed by beats) and send to ES. Using codec => "json" does not work at all.

input {
kafka {
bootstrap_servers => ""
topics => ["metricbeats"]
type => "metricbeat"
codec => "json"
client_id => "metricbeat"

(Magnus Bäck) #2

Could you be more specific than "does not work at all"? What do you get? What's in the Logstash log?

(JMC) #3

When I have codec => "json", nothing goes to ES. No error in the log. When I comment it out, everything goes to ES. However, the payload is wrapped in a message field.

(Magnus Bäck) #4

Sounds like you should use the json_lines codec instead so that you split on newline characters.

(JMC) #5

I quickly gave it a shot. No luck.

In the log, I enabled the debugging.
[2017-02-02T00:15:32,711][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@bootstrap_servers = ""
[2017-02-02T00:15:32,713][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@topics = ["metricbeats"]
[2017-02-02T00:15:32,714][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@type = "metricbeat"
[2017-02-02T00:15:32,714][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@codec = <LogStash::Codecs::JSONLines id=>"json_lines_39ef8b82-24e9-42ad-989d-a07d64641341", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">

(JMC) #6

I also enabled the logging at trace level. But there is not much useful information.

loggers = inputkafka,outputelasticsearch,filterjson,codecjson,codecjsonlines
#kafkaconsumer = logstash.inputs.kafka
logger.inputkafka.level = trace = logstash.outputs.elasticsearch
logger.outputelasticsearch.level = trace = logstash.filters.json
logger.filterjson.level = trace = org.apache.kafka.clients.consumer
logger.kafkaconsumer.level = trace = logstash.codecs.json
logger.codecjson.level = trace = logstash.codecs.jsonlines
logger.codecjsonlines.level = trace

(JMC) #7

Any further assistance is appreciated here.

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.