Logstash can't send logs to aws_elasticsearch from kubernetes

Hello,

I'm trying to use Amazon Cognito authentication with Kibana on aws_ElasticSearch and send logs using logstash from kubernetes.
Authentication with kibana is working fine, but logstash seems to not be sending logs to ES.

I've created a user using aws_access_key_id and aws_secret_access_key with a AmazonESFullAccess policy to be used by logstash.

This is the policy I have setup on my ES Domain:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::xxxxxxxx:role/Cognito_IDpoolAuth_Role",
"arn:aws:iam::xxxxxxxx:user/test"
]
},
"Action": "es:",
"Resource": "arn:aws:es:eu-west-2:xxxxxxx:domain/testkibana/
"
}
]
}

logstash conf:

input {
udp {
port => 5555
codec => json
}
}
output {
stdout {
codec => rubydebug
}
amazon_es {
hosts => ["vpc-MYENDPOINT.eu-west-2.es.amazonaws.com"]
region => "eu-west-2"
aws_access_key_id => 'XXXXXXXXX'
aws_secret_access_key => 'XXXXXXXXXXX'
index => "test-%{+YYYY.ww}"
}
}

logstash is running on kubernetes as a deployment, also I have created a service for elasticSearch as a ExternalName

spec:
externalName: MYENDPOINT.eu-west-2.es.amazonaws.com
sessionAffinity: None
type: ExternalName

Does anyone have an idea, what's wrong?

Thank you!

Hello!

This can be a little hard for us to pinpoint since it's leveraging Amazon's security product. Do you have access to the Elasticsearch logs? I believe they're located at /var/log/elasticsearch/elasticsearch/*.log (where * is your cluster ID) on AWS. In parallel, it may be worth reviewing their Cognito console documentation.

I should mention, we also offer cloud hosting for which we can offer closer guidance on specific requirements!

Regards,
Aaron

Hello,

It's now working fine.
My configuration is fine, the issue is the latest aws_logstash_out plugin version has a bug.

Plugin is installing a index template after a successful connection to ElasticSearch, but it fails installing the template.

I had to modified the file /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb from the aws-logstash-output plugin config

path = "_template/#{name}"

adding "/" at the beginning

path = "/_template/#{name}"

Kind Regards,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.