Logstash intergation with AWS Elasticsearch

Hello,

I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0

Here is my Logstash sample configuration file :-

input {
    file {
    path => "/var/log/httpd/access_log"
    type => "apache-access"
    start_position => "beginning"
  }
}
filter {
  if [type] == "apache-access" {
grok {
  match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
  }
}
output {
  elasticsearch {
  hosts => "xxxx-yyyy-oul45pxbkudcpzz7w7l5222od4.us-east-1.es.amazonaws.com:443"
  ssl => "true"
  manage_template => false
  }
}

I cannot see any Indices in AWS Elasticsearch.
I am not sure if I am missing something. Configuration seems to be plain and simple.
Also there is no log generated on Logstash server.

When I issue the command /bin/logstash -f 01-logstash.conf it gives me proper output. This seems Logstash is working but not sending data to AWS Elasticsearch Cluster.

Is new data being added to the log file you're monitoring? start_position => "beginning" only matters for previously unseen files.

Cranking up the log level with --verbose or --debug could give additional clues.

1 Like

Hey @magnusbaeck thanks for your reply. I solved the problem. There was permission issue, we need to provide proper IAM Policies for AWS Elasticsearch Cluster. Now every thing is working.