Logstash intergation with AWS Elasticsearch

Elastic Stack Logstash
1

Hello,

I am using AWS Elasticsearch service to configure Elasticsearch Cluster and there is a separate server where I have installed Logstash 2.1.0

Here is my Logstash sample configuration file :-

input {
    file {
    path => "/var/log/httpd/access_log"
    type => "apache-access"
    start_position => "beginning"
  }
}
filter {
  if [type] == "apache-access" {
grok {
  match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
  }
}
output {
  elasticsearch {
  hosts => "xxxx-yyyy-oul45pxbkudcpzz7w7l5222od4.us-east-1.es.amazonaws.com:443"
  ssl => "true"
  manage_template => false
  }
}

I cannot see any Indices in AWS Elasticsearch.
I am not sure if I am missing something. Configuration seems to be plain and simple.
Also there is no log generated on Logstash server.

When I issue the command /bin/logstash -f 01-logstash.conf it gives me proper output. This seems Logstash is working but not sending data to AWS Elasticsearch Cluster.

2

Is new data being added to the log file you're monitoring? start_position => "beginning" only matters for previously unseen files.

Cranking up the log level with --verbose or --debug could give additional clues.

1 Like
3

Hey @magnusbaeck thanks for your reply. I solved the problem. There was permission issue, we need to provide proper IAM Policies for AWS Elasticsearch Cluster. Now every thing is working.