# Logstash integration with Amazon Elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-integration-with-amazon-elasticsearch/250232>\
**Category:** Logstash\
**Created:** [September 28, 2020, 4:22pm UTC](https://discuss.elastic.co/t/logstash-integration-with-amazon-elasticsearch/250232 "2020-09-28T16:22:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akhil96](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@Akhil96](https://discuss.elastic.co/u/Akhil96)\
**Post date:** [September 28, 2020, 4:22pm UTC](https://discuss.elastic.co/t/logstash-integration-with-amazon-elasticsearch/250232/1 "2020-09-28T16:22:34Z")

</div>

Hi Team,

I have setup logstash to fetch from amazon MSK and then push the messages to Amazon Elasticsearch with Kibana integration. The setup works smoothly in QA and Dev environments but when I use the same logstash image in Production environment it stops working. I can see in the logstash logs, that the messages are fetched from MSK, but nothing is pushed to Elasticsearch. The wierd thing is that the logstash logs are exactly the same in QA, Dev and Prod. I am even using the same config file, with only the Elasticsearch endpoint that changes. Here is the config file.

input {  
kafka {  
bootstrap\_servers =\> "{MSK\_HOST}" topics =\> "{TOPIC}"  
group\_id =\> "${KAFKA\_GRP\_ID}"  
auto\_offset\_reset =\> "latest"  
#ssl\_truststore\_location =\> "/usr/share/logstash/config/kafka.client.truststore.jks"  
security\_protocol =\> "SSL"  
decorate\_events =\> true  
codec =\> json  
}  
}

filter {  
ruby {  
code =\> "event.set('[@metadata][kafka][lc\_topic]', event.get('[@metadata][kafka][topic]').split(/(?=[A-Z])/).map{|x| x.downcase }.join('') )"  
}  
}

output {  
amazon\_es {  
hosts =\> ["${ES\_HOST}"]  
ssl =\> true  
#ilm\_enabled =\> false  
index =\> "logstash.%{[@metadata][kafka][lc\_topic]}.%{+YYYY.MM.dd}"  
manage\_template =\> false  
aws\_access\_key\_id =\> ''  
aws\_secret\_access\_key =\> ''

}

}

Any idea why this happens?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 28, 2020, 9:12pm UTC](https://discuss.elastic.co/t/logstash-integration-with-amazon-elasticsearch/250232/2 "2020-09-28T21:12:12Z")

</div>

We aren't able to help here sorry, this is a custom plugin AWS has built.

---

<div class="post-metadata">

**Author:** ![Akhil96](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@Akhil96](https://discuss.elastic.co/u/Akhil96)\
**Post date:** [September 29, 2020, 8:13am UTC](https://discuss.elastic.co/t/logstash-integration-with-amazon-elasticsearch/250232/3 "2020-09-29T08:13:28Z")

</div>

it doesnt work with the usual plug-in as well

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 27, 2020, 8:13am UTC](https://discuss.elastic.co/t/logstash-integration-with-amazon-elasticsearch/250232/4 "2020-10-27T08:13:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.