I upgraded logstash from 2.4 to 7.5 and also kafka from 0.10 to 2.5 Im facing few issues here when pushing logs through logstash
issue background information:
- Im using the same logstash config files in /etc/logstash/conf.d/*.conf and I defined the each config file in pipeline file /etc/logstash/pipeline.yml
- Im using logstash-integration-kafka and sending the messages to kafka brokers which defined in output section
- when I start logstash 7.5 i can see in the logstash logs it is started properly without any issues and service is fine but it is sending empty messages to kafka( i have a kakfa UI tool running kafka side) and I also validated logstash config in verbose mode and didn't find any errors.
- To double check kafka side Im running logstash in two hosts with 0.10 and 7.5 I can see the messages from logstash with 0.10 sending to kafka topics but the logstash 7.5 is sending empty messages to topics
- my logstash-integration-kafka for logstash is in version 10.0.0 is this compatible with kafka 2.5?. In plugin documentation, it mentioned to check with kafka documentation but kafka documentation page is not up to date so I'm checking here.