Logstash can't send logs to aws_elasticsearch from kubernetes


#1

Hello,

I'm trying to use Amazon Cognito authentication with Kibana on aws_ElasticSearch and send logs using logstash from kubernetes.
Authentication with kibana is working fine, but logstash seems to not be sending logs to ES.

I've created a user using aws_access_key_id and aws_secret_access_key with a AmazonESFullAccess policy to be used by logstash.

This is the policy I have setup on my ES Domain:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::xxxxxxxx:role/Cognito_IDpoolAuth_Role",
"arn:aws:iam::xxxxxxxx:user/test"
]
},
"Action": "es:",
"Resource": "arn:aws:es:eu-west-2:xxxxxxx:domain/testkibana/
"
}
]
}

logstash conf:

input {
udp {
port => 5555
codec => json
}
}
output {
stdout {
codec => rubydebug
}
amazon_es {
hosts => ["vpc-MYENDPOINT.eu-west-2.es.amazonaws.com"]
region => "eu-west-2"
aws_access_key_id => 'XXXXXXXXX'
aws_secret_access_key => 'XXXXXXXXXXX'
index => "test-%{+YYYY.ww}"
}
}

logstash is running on kubernetes as a deployment, also I have created a service for elasticSearch as a ExternalName

spec:
externalName: MYENDPOINT.eu-west-2.es.amazonaws.com
sessionAffinity: None
type: ExternalName

Does anyone have an idea, what's wrong?

Thank you!


(Aaron Caldwell) #2

Hello!

This can be a little hard for us to pinpoint since it's leveraging Amazon's security product. Do you have access to the Elasticsearch logs? I believe they're located at /var/log/elasticsearch/elasticsearch/*.log (where * is your cluster ID) on AWS. In parallel, it may be worth reviewing their Cognito console documentation.

I should mention, we also offer cloud hosting for which we can offer closer guidance on specific requirements!

Regards,
Aaron


#3

Hello,

It's now working fine.
My configuration is fine, the issue is the latest aws_logstash_out plugin version has a bug.

Plugin is installing a index template after a successful connection to ElasticSearch, but it fails installing the template.

I had to modified the file /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb from the aws-logstash-output plugin config

path = "_template/#{name}"

adding "/" at the beginning

path = "/_template/#{name}"

Kind Regards,