# Logstash can't send logs to aws\_elasticsearch from kubernetes

**URL:** <https://discuss.elastic.co/t/logstash-cant-send-logs-to-aws-elasticsearch-from-kubernetes/159564>\
**Category:** Kibana\
**Created:** [December 5, 2018, 3:22pm UTC](https://discuss.elastic.co/t/logstash-cant-send-logs-to-aws-elasticsearch-from-kubernetes/159564 "2018-12-05T15:22:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mreynoso](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mreynoso](https://discuss.elastic.co/u/mreynoso)\
**Post date:** [December 5, 2018, 3:22pm UTC](https://discuss.elastic.co/t/logstash-cant-send-logs-to-aws-elasticsearch-from-kubernetes/159564/1 "2018-12-05T15:22:43Z")

</div>

Hello,

I'm trying to use Amazon Cognito authentication with Kibana on aws\_ElasticSearch and send logs using logstash from kubernetes.  
Authentication with kibana is working fine, but logstash seems to not be sending logs to ES.

I've created a user using aws\_access\_key\_id and aws\_secret\_access\_key with a AmazonESFullAccess policy to be used by logstash.

This is the policy I have setup on my ES Domain:

{  
"Version": "2012-10-17",  
"Statement": [  
{  
"Effect": "Allow",  
"Principal": {  
"AWS": [  
"arn:aws:iam::xxxxxxxx:role/Cognito\_IDpoolAuth\_Role",  
"arn:aws:iam::xxxxxxxx:user/test"  
]  
},  
"Action": "es:_",  
"Resource": "arn:aws:es:eu-west-2:xxxxxxx:domain/testkibana/_"  
}  
]  
}

logstash conf:

input {  
udp {  
port =\> 5555  
codec =\> json  
}  
}  
output {  
stdout {  
codec =\> rubydebug  
}  
amazon\_es {  
hosts =\> ["[vpc-MYENDPOINT.eu-west-2.es.amazonaws.com](http://vpc-MYENDPOINT.eu-west-2.es.amazonaws.com)"]  
region =\> "eu-west-2"  
aws\_access\_key\_id =\> 'XXXXXXXXX'  
aws\_secret\_access\_key =\> 'XXXXXXXXXXX'  
index =\> "test-%{+YYYY.ww}"  
}  
}

logstash is running on kubernetes as a deployment, also I have created a service for elasticSearch as a ExternalName

spec:  
externalName: [MYENDPOINT.eu-west-2.es.amazonaws.com](http://MYENDPOINT.eu-west-2.es.amazonaws.com)  
sessionAffinity: None  
type: ExternalName

Does anyone have an idea, what's wrong?

Thank you!

---

<div class="post-metadata">

**Author:** ![Aaron\_Caldwell](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aaron_caldwell/32/45755_2.png) [@Aaron\_Caldwell](https://discuss.elastic.co/u/Aaron_Caldwell)\
**Post date:** [December 5, 2018, 3:56pm UTC](https://discuss.elastic.co/t/logstash-cant-send-logs-to-aws-elasticsearch-from-kubernetes/159564/2 "2018-12-05T15:56:11Z")

</div>

Hello!

This can be a little hard for us to pinpoint since it's leveraging Amazon's security product. Do you have access to the Elasticsearch logs? I believe they're located at `/var/log/elasticsearch/elasticsearch/*.log (where * is your cluster ID)` on AWS. In parallel, it may be worth reviewing their [Cognito console documentation](https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-console.html).

I should mention, we also offer [cloud hosting](https://www.elastic.co/cloud) for which we can offer closer guidance on specific requirements!

Regards,  
Aaron

---

<div class="post-metadata">

**Author:** ![mreynoso](https://avatars.discourse-cdn.com/v4/letter/m/58f4c7/32.png) [@mreynoso](https://discuss.elastic.co/u/mreynoso)\
**Post date:** [December 7, 2018, 12:26pm UTC](https://discuss.elastic.co/t/logstash-cant-send-logs-to-aws-elasticsearch-from-kubernetes/159564/3 "2018-12-07T12:26:06Z")

</div>

Hello,

It's now working fine.  
My configuration is fine, the issue is the latest aws\_logstash\_out plugin version has a bug.

Plugin is installing a index template after a successful connection to ElasticSearch, but it fails installing the template.

I had to modified the file `/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb` from the aws-logstash-output plugin config

`path = "_template/#{name}"`

adding "/" at the beginning

`path = "/_template/#{name}"`

Kind Regards,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2019, 12:26pm UTC](https://discuss.elastic.co/t/logstash-cant-send-logs-to-aws-elasticsearch-from-kubernetes/159564/4 "2019-01-04T12:26:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.