Can't start logstash after x-pack installation

using elk 5.4.

After installing x-pack for elk stack 5.4, I'm now getting the error:

Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

kibana and elasticsearch start fine, but logstash is having problems.

I'm using the logstash_system credentials in logstash.yml, and I have tested the creds against the api:

curl -u logstash_system 'http://localhost:9200/_xpack/security/_authenticate?pretty=true'
Enter host password for user 'logstash_system':
"username" : "logstash_system",
"roles" : [
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
"enabled" : true

What gives?

it could be many things, try running your config files one by one in debug to see the details

ps there are alot of topics like this they might be worth a look :slight_smile:

Looks like LS is failing to create the metrics it uses for monitoring. Did you setup things as per

I did Mark. When I look at monitoring from kibana, I can see both kibana and elasticsearch, but not LS. I'm using using the elastic credentials to test out, since elastic is a super user. /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: [""]
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "xxxxxxxx"

I can even see monitoring indices being created. Which I'm retrieving using the same credentials as I'm using in LS.

curl -XGET 'localhost:9200/_cat/indices?v&pretty' -u elastic:xxxxxxxx| grep monitor

yellow open .monitoring-kibana-2-2017.05.21 Tmoh5hNTRTuTH1s88MP4XQ 1 1 8638 0 1.9mb 1.9mb
1yellow open .monitoring-logstash-2-2017.05.17 GvOG_05mQomF0chFHPIX9Q 1 1 223 0 160.4kb 160.4kb
0yellow open .monitoring-data-2 cXGS3hAJRxujGmf4SON_-A 1 1 4 1 18kb 18kb
0yellow open .monitoring-es-2-2017.05.24 CX5y8js4REKrAVX-sNotdQ 1 1 218509 5910 67.3mb 67.3mb
yellow open .monitoring-kibana-2-2017.05.20 I5CAPkm2RKe4fGuiXpgbnA 1 1 8639 0 1.9mb 1.9mb
1yellow open .monitoring-es-2-2017.05.18 SSx3mVxlQQiC4C0tsNR5ow 1 1 1576263 7910 503.8mb 503.8mb
4yellow open .monitoring-kibana-2-2017.05.23 bmeaLa3URiGFNQrwZzw8UA 1 1 3366 0 906.8kb 906.8kb
0yellow open .monitoring-kibana-2-2017.05.24 Dm-dK4A0RXC-gDABFHWzqA 1 1 852 0 283.8kb 283.8kb
6yellow open .monitoring-kibana-2-2017.05.18 rjMf9UvxQoeNpfCsFn6e5w 1 1 8638 0 1.9mb 1.9mb
1yellow open .monitoring-kibana-2-2017.05.19 Rfgwfzy1TSSmpcj2WASgkA 1 1 8639 0 1.9mb 1.9mb
yellow open .monitoring-es-2-2017.05.20 eM1fuicaQnS4TDnyRBoUVw 1 1 1996475 7327 595.5mb 595.5mb

Pretty wild.

Can you paste the entire error?

this is the whole message:

[2017-05-24T05:35:54,472][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

does this message prevent logstash from starting, or does it stop appearing during runtime?

it prevents logstash from starting up. The message keeps on coming every few seconds.

is it possible for you to show me the pipeline configuration being used in logstash?

I went ahead and reinstalled everything from scratch. renamed config and install folders alike. Everything works as expected now. too bad I couldn't reproduced what the issue was, but happy things are now normal. Thanks for your help guys.

1 Like

glad you're sorted out! please ping this thread if it happens again

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.