Logstash Kafka Consumer/Input
Hello! I'm attempting use the kakfa input plugin and thought I would start with some basic testing first. From the looks of it, I'm losing three fields (app, pid, and msgid) when logstash displays the output. Has anyone else ran into this or am I missing something in my logstash configuration for the kafka input plugin? Additional research found that the community recommended using the json codec, but after adding that, the issue still existed. Any help or pointers would be appreciated!
Thank you.
Versions
kafka version: 8.2.2
logstash-input-kafka (2.0.8)```
**Logstash**
/opt/logstash/bin/logstash -e 'input { kafka { zk_connect => "kafkahost01:2181" topic_id => "internal" codec => "json" }} output {stdout {}}'
```Settings: Default pipeline workers: 16
Pipeline main started
2016-04-27T19:37:40.275Z testhost01 ntpd exiting on signal 15
2016-04-27T19:37:40.275Z testhost01 ntpd 4.2.6p5@1.2349-o Thu Feb 11 18:30:40 UTC 2016 (1)
2016-04-27T19:37:40.276Z testhost01 proto: precision = 0.102 usec
2016-04-27T19:37:40.276Z testhost01 ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
2016-04-27T19:37:40.276Z testhost01 Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
2016-04-27T19:37:40.277Z testhost01 Listen and drop on 1 v6wildcard :: UDP 123
2016-04-27T19:37:40.277Z testhost01 Listen normally on 2 lo 127.0.0.1 UDP 123
2016-04-27T19:37:40.277Z testhost01 Listen normally on 3 eth0 10.10.12.29 UDP 123
2016-04-27T19:37:40.278Z testhost01 Listen normally on 4 lo ::1 UDP 123
2016-04-27T19:37:40.278Z testhost01 Listen normally on 5 eth0 fe80::250:56ff:fe96:a6d2 UDP 123
2016-04-27T19:37:40.279Z testhost01 peers refreshed
2016-04-27T19:37:41.022Z testhost01 Listening on routing socket on fd #22 for interface updates```
**Kafka Queue (Console Consumer)**
kafka-console-consumer.sh --zookeeper kakfahost01:2181 --topic internal
```{"priority":29,"version":0,"timestamp":"2016-04-27T15:37:39.399418-04:00","host":"testhost01","app":"ntpd","pid":2314,"msgid":"-","message":" ntpd exiting on signal 15"}
{"priority":29,"version":0,"timestamp":"2016-04-27T15:37:41.428965-04:00","host":"testhost01","app":"ntpd","pid":5591,"msgid":"-","message":" ntpd 4.2.6p5@1.2349-o Thu Feb 11 18:30:40 UTC 2016 (1)"}
{"priority":29,"version":0,"timestamp":"2016-04-27T15:37:41.429667-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" proto: precision = 0.102 usec"}
{"priority":31,"version":0,"timestamp":"2016-04-27T15:37:41.430107-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" ntp_io: estimated max descriptors: 1024, initial socket boundary: 16"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.430191-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.433657-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listen and drop on 1 v6wildcard :: UDP 123"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.433863-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listen normally on 2 lo 127.0.0.1 UDP 123"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.433911-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listen normally on 3 eth0 10.10.12.29 UDP 123"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.433967-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listen normally on 4 lo ::1 UDP 123"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.434036-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listen normally on 5 eth0 fe80::250:56ff:fe96:a6d2 UDP 123"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.434081-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" peers refreshed"}
{"priority":30,"version":0,"timestamp":"2016-04-27T15:37:41.434128-04:00","host":"testhost01","app":"ntpd","pid":5592,"msgid":"-","message":" Listening on routing socket on fd #22 for interface updates"}```