Hello, I'm having this issue:
When I run: sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/apache.conf works perfectly and logstash send logs to AWS ES.
When I run as a service using sudo initctl logstash start logstash starts correctly without errors but no data is sent to AWS ES.
This is really frustrating because we choose AWS ES instead of Elastic Cloud..
Currently we are running AWS ES v7.1, Kibana version 7.1.1 and Logstash version is 7.4 as the compatibility matrix says it supported link. Also we are sending data with Logstash from an Ubuntu 14.04 with the amazon_es_output plugin.
Here are the two different outputs logs when running logstash from command and as a service:
logstash_logs.log
My logstash.yml file is by default and sits in /etc/logstash/
My apache.conf is as follows:
input {
file {
path => "/var/log/apache2/*.log"
}
}
filter {
if [path] =~ "access" {
mutate { replace => { type => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
} else if [path] =~ "error" {
mutate { replace => { type => "apache_error" } }
} else {
mutate { replace => { type => "random_logs" } }
}
}
output {
if [type] in ["apache_access","random_logs","apache_error"] {
if [response] =~ /^2\d\d/ {
amazon_es {
hosts => ["vpc-xxxx.region.es.amazonaws.com"]
region => "us-east-1"
aws_access_key_id => ''
aws_secret_access_key => ''
index => "apache-access-logs-%{+YYYY.MM.dd}"
}
}
}
}
Any help would be appreciated.
Marcos.