Logstash Failed to create monitoring event

[faustf (Fabrizio)]

Hy guys

After I installed x-pack for monitoring logstash - elasticsearch and kibana, I've the following error on logstash:

[2017-02-07T10:18:05,181][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

I followed the installation guide, but maybe I did somenthing wrong

logstash.yml

......
......
xpack.monitoring.elasticsearch.url: "http://localhost:9200"
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "changeme"

/etc/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
  },
  user => logstash_internal
  password => changeme
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  },
  user => logstash_internal
  password => changeme
}

Also, now, in Kibana I don't see any new metricbeat data
(the metricbeat instances are configured to send the data to logstash instance)

I didn't change any default password
thank you

[Dan_Markhasin]

Having run into the same issue today, in my case it was caused by the fact that logstash itself failed to start correctly due to wrong configuration.
However, instead of stopping and exiting, the process remained alive and continuously printing this error to the log.

Make sure your logstash configuration is correct and that there are no other errors when it starts.

[Charles.w (Charles Casadei)]

I too ran into this issue today. Again, this was caused by an invalid pipeline configuration.

I do believe that logstash should fail to start completely instead of trying to send metrics to its monitoring cluster.

[jjstephe (Jon Stephenson)]

I've had this same issue, but there is nothing wrong with the configuration - nothing has changed and I was simply using the same mysql java connector I had used on tons of 1G+ files before... but now the process stays alive and doesn't exit when complete.

I believe I am seeing the same thing with CSV imports I have been doing.

Any thoughts on connector issues after this update?

[abdalians (Saqi)]

I am seeing the exact same issue as well.
logstash 5.2.2

[masik (Alexandr Usha)]

Need to change user to elastic and passwd changeme in output to elasticsearch
he need all grant access to create indexes

and add to elastic search

action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*

or you can add

xpack.security.enabled: false
to disable auth in elasticsearch