using ELK stack 5.3. I'm getting the following error message: [ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}
I just added x-pack security with AD to the stack. I can login from kibana using AD credentials, I can curl values using the internal credentials 'logstash_system'.
curl -u logstash_system:xyz123pass 'http://localhost:9200/_xpack/security/_authenticate?pretty=true'
{
"username" : "logstash_system",
"roles" : [
"logstash_system"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}
So that works. These are my logstash config files:
cat /etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
user => logstash_system
password => xyz123pass
}
}
cat /etc/logstash/conf.d/10-syslog-filter.conf
filter {
user => logstash_system
password => xyz123pass
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
cat /etc/logstash/conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
hosts => "http://localhost:9200"
sniffing => true
manage_template => false
index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
user => logstash_system
password => xyz123pass
}
}
Everything was working fine before I added the x-pack security. This is the actual logstash config:
cat /etc/logstash/logstash.yml| grep -v '#'
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: "http://localhost:9200"
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: xyz123pass
Can someone help out with this issue?