Can't start logstash after x-pack installation

Elastic Stack Logstash
1

using elk 5.4.

After installing x-pack for elk stack 5.4, I'm now getting the error:

Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

kibana and elasticsearch start fine, but logstash is having problems.

I'm using the logstash_system credentials in logstash.yml, and I have tested the creds against the api:

curl -u logstash_system 'http://localhost:9200/_xpack/security/_authenticate?pretty=true'
Enter host password for user 'logstash_system':
{
"username" : "logstash_system",
"roles" : [
"logstash_system"
],
"full_name" : null,
"email" : null,
"metadata" : {
"_reserved" : true
},
"enabled" : true
}

What gives?

2

it could be many things, try running your config files one by one in debug to see the details

ps there are alot of topics like this they might be worth a look :slight_smile:

3

Looks like LS is failing to create the metrics it uses for monitoring. Did you setup things as per https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html

4

I did Mark. When I look at monitoring from kibana, I can see both kibana and elasticsearch, but not LS. I'm using using the elastic credentials to test out, since elastic is a super user.

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: ["http://127.0.0.1:9200"]
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "xxxxxxxx"

I can even see monitoring indices being created. Which I'm retrieving using the same credentials as I'm using in LS.

curl -XGET 'localhost:9200/_cat/indices?v&pretty' -u elastic:xxxxxxxx| grep monitor

yellow open .monitoring-kibana-2-2017.05.21 Tmoh5hNTRTuTH1s88MP4XQ 1 1 8638 0 1.9mb 1.9mb
1yellow open .monitoring-logstash-2-2017.05.17 GvOG_05mQomF0chFHPIX9Q 1 1 223 0 160.4kb 160.4kb
0yellow open .monitoring-data-2 cXGS3hAJRxujGmf4SON_-A 1 1 4 1 18kb 18kb
0yellow open .monitoring-es-2-2017.05.24 CX5y8js4REKrAVX-sNotdQ 1 1 218509 5910 67.3mb 67.3mb
yellow open .monitoring-kibana-2-2017.05.20 I5CAPkm2RKe4fGuiXpgbnA 1 1 8639 0 1.9mb 1.9mb
1yellow open .monitoring-es-2-2017.05.18 SSx3mVxlQQiC4C0tsNR5ow 1 1 1576263 7910 503.8mb 503.8mb
4yellow open .monitoring-kibana-2-2017.05.23 bmeaLa3URiGFNQrwZzw8UA 1 1 3366 0 906.8kb 906.8kb
0yellow open .monitoring-kibana-2-2017.05.24 Dm-dK4A0RXC-gDABFHWzqA 1 1 852 0 283.8kb 283.8kb
6yellow open .monitoring-kibana-2-2017.05.18 rjMf9UvxQoeNpfCsFn6e5w 1 1 8638 0 1.9mb 1.9mb
1yellow open .monitoring-kibana-2-2017.05.19 Rfgwfzy1TSSmpcj2WASgkA 1 1 8639 0 1.9mb 1.9mb
yellow open .monitoring-es-2-2017.05.20 eM1fuicaQnS4TDnyRBoUVw 1 1 1996475 7327 595.5mb 595.5mb

Pretty wild.

5

Can you paste the entire error?

6

this is the whole message:

[2017-05-24T05:35:54,472][ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

7

does this message prevent logstash from starting, or does it stop appearing during runtime?

8

it prevents logstash from starting up. The message keeps on coming every few seconds.

9

is it possible for you to show me the pipeline configuration being used in logstash?

10

I went ahead and reinstalled everything from scratch. renamed config and install folders alike. Everything works as expected now. too bad I couldn't reproduced what the issue was, but happy things are now normal. Thanks for your help guys.

1 Like
11

glad you're sorted out! please ping this thread if it happens again

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.