Codec json is not working

Elastic Stack Logstash
1

I am trying to read data from Kafka (fed by beats) and send to ES. Using codec => "json" does not work at all.

input {
kafka {
bootstrap_servers => "10.100.177.100:9999"
topics => ["metricbeats"]
type => "metricbeat"
codec => "json"
client_id => "metricbeat"
}
}

2

Could you be more specific than "does not work at all"? What do you get? What's in the Logstash log?

3

When I have codec => "json", nothing goes to ES. No error in the log. When I comment it out, everything goes to ES. However, the payload is wrapped in a message field.

4

Sounds like you should use the json_lines codec instead so that you split on newline characters.

5

I quickly gave it a shot. No luck.

In the log, I enabled the debugging.
[2017-02-02T00:15:32,711][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@bootstrap_servers = "10.100.177.37:6667"
[2017-02-02T00:15:32,713][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@topics = ["metricbeats"]
[2017-02-02T00:15:32,714][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@type = "metricbeat"
[2017-02-02T00:15:32,714][DEBUG][logstash.inputs.kafka ] config LogStash::Inputs::Kafka/@codec = <LogStash::Codecs::JSONLines id=>"json_lines_39ef8b82-24e9-42ad-989d-a07d64641341", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">

6

I also enabled the logging at trace level. But there is not much useful information.

loggers = inputkafka,outputelasticsearch,filterjson,codecjson,codecjsonlines
#kafkaconsumer
logger.inputkafka.name = logstash.inputs.kafka
logger.inputkafka.level = trace

logger.outputelasticsearch.name = logstash.outputs.elasticsearch
logger.outputelasticsearch.level = trace

logger.filterjson.name = logstash.filters.json
logger.filterjson.level = trace

logger.kafkaconsumer.name = org.apache.kafka.clients.consumer
logger.kafkaconsumer.level = trace

logger.codecjson.name = logstash.codecs.json
logger.codecjson.level = trace

logger.codecjsonlines.name = logstash.codecs.jsonlines
logger.codecjsonlines.level = trace

7

Any further assistance is appreciated here.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.