Hi Team,
I have setup logstash to fetch from amazon MSK and then push the messages to Amazon Elasticsearch with Kibana integration. The setup works smoothly in QA and Dev environments but when I use the same logstash image in Production environment it stops working. I can see in the logstash logs, that the messages are fetched from MSK, but nothing is pushed to Elasticsearch. The wierd thing is that the logstash logs are exactly the same in QA, Dev and Prod. I am even using the same config file, with only the Elasticsearch endpoint that changes. Here is the config file.
input {
kafka {
bootstrap_servers => "{MSK_HOST}"
topics => "{TOPIC}"
group_id => "${KAFKA_GRP_ID}"
auto_offset_reset => "latest"
#ssl_truststore_location => "/usr/share/logstash/config/kafka.client.truststore.jks"
security_protocol => "SSL"
decorate_events => true
codec => json
}
}
filter {
ruby {
code => "event.set('[@metadata][kafka][lc_topic]', event.get('[@metadata][kafka][topic]').split(/(?=[A-Z])/).map{|x| x.downcase }.join('') )"
}
}
output {
amazon_es {
hosts => ["${ES_HOST}"]
ssl => true
#ilm_enabled => false
index => "logstash.%{[@metadata][kafka][lc_topic]}.%{+YYYY.MM.dd}"
manage_template => false
aws_access_key_id => ''
aws_secret_access_key => ''
}
}
Any idea why this happens?