\>, @created\_at=2017-05-09 13:43:32 -0400\>} [2017-05-09T13:43:32,094][ERROR][logstash.inputs.metrics] Failed to create monitoring event {:message=\>"For path: events", :error=\>"LogStash::Instrument::MetricStore::MetricNotFound", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:225:in `get_recursively'", "org/jruby/RubyArray.java:1613:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:224:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:235:in`get\_recursively'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:224:in`get\_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:95:in `get'", "org/jruby/ext/thread/Mutex.java:149:in`synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:94:in `get'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:108:in`get\_shallow'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric\_store.rb:157:in `extract_metrics'", "org/jruby/RubyArray.java:1613:in`each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:133:in`extract\_metrics'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.4.0-java/lib/monitoring/inputs/metrics.rb:191:in `format_global_event_count'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.4.0-java/lib/monitoring/inputs/metrics.rb:80:in`build\_event'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.4.0-java/lib/monitoring/inputs/metrics.rb:60:in `update'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/x-pack-5.4.0-java/lib/monitoring/inputs/metrics.rb:35:in`configure\_snapshot\_poller'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/safe_task_executor.rb:24:in`execute'", "com/concurrent\_ruby/ext/SynchronizationLibrary.java:174:in `synchronize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/safe_task_executor.rb:19:in`execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/timer\_task.rb:307:in `execute_task'", "org/jruby/RubyProc.java:281:in`call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/safe\_task\_executor.rb:24:in `execute'", "com/concurrent_ruby/ext/SynchronizationLibrary.java:174:in`synchronize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/safe\_task\_executor.rb:19:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/ivar.rb:170:in`safe\_execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/scheduled\_task.rb:285:in `process_task'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/timer_set.rb:157:in`process\_tasks'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/java_executor_service.rb:94:in`run'", "Concurrent$$JavaExecutorService$$Job\_754842726.gen:13:in `run'"]} [2017-05-09T13:43:36,825][DEBUG][logstash.pipeline] Pushing flush onto pipeline
# configs:
# cat /etc/logstash/logstash.yml
* * *
path.data: "/var/lib/logstash"
path.config: "/etc/logstash/conf.d"
path.logs: "/var/log/logstash"
xpack:
monitoring:
elasticsearch:
username: logstash\_system
password: SecretPassword
pipeline:
batch:
size: 25
delay: 5
# cat /etc/logstash/conf.d/basic\_ls\_config
# input { udp { port =\> 2055 codec =\> netflow } } output { elasticsearch { hosts =\> 127.0.0.1 user =\> logstash\_internal password =\> AnotherSecretPassword } }
# I can authenticate with both logstash users, both internal and system:
# curl -u logstash\_system:SecretPassword '[http://localhost:9200/\_xpack/security/\_authenticate?pretty=true](http://localhost:9200/_xpack/security/_authenticate?pretty=true)'
{
"username" : "logstash\_system",
"roles" : [
"logstash\_system"
],
"full\_name" : null,
"email" : null,
"metadata" : {
"\_reserved" : true
},
"enabled" : true
}
# curl -u logstash\_internal:AnotherSecretPassword localhost:9200/\_xpack/security/\_authenticate?pretty=true
# { "username" : "logstash\_internal", "roles" : ["logstash\_writer"], "full\_name" : null, "email" : null, "metadata" : { }, "enabled" : true }
# But I can not GET localhost:9200/\_xpack/monitoring:
# curl -u elastic '[http://localhost:9200/\_xpack/monitoring/?pretty=true](http://localhost:9200/_xpack/monitoring/?pretty=true)'
# Enter host password for user 'elastic': { "error" : { "root\_cause" : [{ "type" : "illegal\_argument\_exception", "reason" : "No endpoint or operation is available at [monitoring]" } ], "type" : "illegal\_argument\_exception", "reason" : "No endpoint or operation is available at [monitoring]" }, "status" : 400 }
## Please, suggest, how can I debug and fix this problem.
Regards,
Sergey
---
### Author:  [@seriv](https://discuss.elastic.co/u/seriv)
#### Post date: [May 9, 2017, 8:59pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/2 "2017-05-09T20:59:32Z")
I have changed output to be 'stdout{ codec =\> rubydebug }' and getting the same errors about input metrics. What is wrong here with logstash? Is it netflow? Will try now some other input.
---
### Author:  [@jordansissel](https://discuss.elastic.co/u/jordansissel)
#### Post date: [May 9, 2017, 9:05pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/3 "2017-05-09T21:05:55Z")
This looks like a bug in logstash x-pack.
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [May 9, 2017, 9:59pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/4 "2017-05-09T21:59:21Z")
@seriv to build on what Jordan's saying, this is a bug in xpack's monitoring code. This shouldn't affect your own pipeline's code.
Are there any other errors before or after this one in the log?
---
### Author:  [@seriv](https://discuss.elastic.co/u/seriv)
#### Post date: [May 9, 2017, 9:59pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/5 "2017-05-09T21:59:34Z")
Sorry, I was wrong: i missed the difference in command-line arguments --path.settings and --path.config to the logstash. Now tried all 4 combinations:
1. input { stdin {} } output {stdout {codec =\> rubydebug}}
2. input { udp { port =\> 2055, codec =\> netflow } output {stdout {codec =\> rubydebug}}
3. input { stdin {} output { elasticsearch { hosts =\> 127.0.0.1, user =\> logstash\_internal ,password =\> Secret }
4. input { udp { port =\> 2055, codec =\> netflow } output { elasticsearch { hosts =\> 127.0.0.1, user =\> logstash\_internal ,password =\> Secret }
First two were working fine, while 3 and 4 did not work. So, codec netflow and input udp are cleared, the problem is in my configuration for connection to elasticsearch. Strange that the easiest way to see the problem is to try curling URL '[http://127.0.0.1:9200/\_xpack/monitoring/?pretty=true](http://127.0.0.1:9200/_xpack/monitoring/?pretty=true)', with even superuser privileges I got error:
{
"error" : {
"root\_cause" : [
{
"type" : "illegal\_argument\_exception",
"reason" : "No endpoint or operation is available at [monitoring]"
}
],
"type" : "illegal\_argument\_exception",
"reason" : "No endpoint or operation is available at [monitoring]"
},
"status" : 400
}
---
### Author:  [@jordansissel](https://discuss.elastic.co/u/jordansissel)
#### Post date: [May 9, 2017, 10:46pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/6 "2017-05-09T22:46:25Z")
> [@seriv](#):
>
> "reason" : "No endpoint or operation is available at [monitoring]"
Is x-pack monitoring enabled on your Elasticsearch cluster?
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [May 10, 2017, 12:06am UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/7 "2017-05-10T00:06:23Z")
No, it appears that there's something about that exact config that kills xpack.
Running `bin/logstash -e 'input { udp { port => 2055, codec => netflow } output { elasticsearch { hosts => 127.0.0.1, user => logstash_internal ,password => Secret } '` with xpack will repro.
---
### Author:  [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)
#### Post date: [May 10, 2017, 12:21am UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/8 "2017-05-10T00:21:24Z")
The problem is that there are two errors. @seriv 's last config is broken. The first log line is:
`[2017-05-09T19:18:15,431][ERROR][logstash.agent] Cannot create pipeline {:reason=>"Expected one of #, {, ., } at line 1, column 27 (byte 27) after input { udp { port => 2055"}`
However, the metrics pipeline still starts, despite the main pipeline not starting, and then logs
`Failed to create monitoring event {:message=>"For path: events", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}`
This is just confusing UX.
I propose that Logstash should exhibit the following behaviors to fix this.
1. Logstash should just die if the config given is invalid and config reloading is not enabled.
2. The metrics pipeline should not be dependent on other pipelines existing or not.
@jordansissel WDYT?
---
### Author:  [@seriv](https://discuss.elastic.co/u/seriv)
#### Post date: [May 10, 2017, 1:48am UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/9 "2017-05-10T01:48:03Z")
Thanks!
problem fixed by enclosing 127.0.0.1 with single quotes!
---
### Author:  [@seriv](https://discuss.elastic.co/u/seriv)
#### Post date: [May 10, 2017, 2:35pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/10 "2017-05-10T14:35:36Z")
## Although, there was another small issue. lagstash\_internal user belongs to logstash\_writer group as per examples in the documentation and thus have a right to create indexes with logstash-\* prefix, but in the logs: [WARN][logstash.outputs.elasticsearch] Failed action. {:status=\>404, :action=\>["index", {:\_id=\>nil, :\_index=\>"logstash-2017.05.10", :\_type=\>"logs", :\_routing=\>nil}, 2017-05-10T01:44:52.491Z 128.8.127.153 %{message}], :response=\>{"index"=\>{"\_index"=\>"logstash-2017.05.10", "\_type"=\>"logs", "\_id"=\>nil, "status"=\>404, "error"=\>{"type"=\>"index\_not\_found\_exception", "reason"=\>"no such index and [action.auto\_create\_index] ([.security,.monitoring\*,.watches,.triggered\_watches,.watcher-history\*]) doesn't match", "index\_uuid"=\>"_na_", "index"=\>"logstash-2017.05.10"}}} Apperantly, a line in elasticsearch.yaml explicitly permits auto-creation of indexes, like: 'action.auto\_create\_index' =\> '.security,.monitoring\*,.watches,.triggered\_watches,.watcher-history\*', is actually prohibiting any other index to be created. Strange, is not it?
Sergey.
---
### Author:  [@Gude.ravi](https://discuss.elastic.co/u/Gude.ravi)
#### Post date: [May 18, 2017, 3:36pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/11 "2017-05-18T15:36:36Z")
I am having the same issue. is there a solution on this ?
[2017-05-18T11:29:49,809][ERROR][logstash.inputs.metrics] Failed to create monitoring event {:message=\>"For path: events", :error=\>"LogStash::Instrument::MetricStore::MetricNotFound"}
path.data: /var/lib/logstash
xpack.monitoring.elasticsearch.url: "url:9200"
xpack.monitoring.elasticsearch.username: "elastic"
##tried default logstash\_writer
xpack.monitoring.elasticsearch.password: "changeme"
example conf file
input {
beats{
port=\> 5000
}
heartbeat {
interval =\> 30
type =\> 'heartbeat'
enable\_metric =\> false
}
}
output {
elasticsearch {
hosts =\> ["url:9200"]
index =\> "logstash-health-%{+YYYY.MM.dd}"
user =\> logstash\_internal
password =\> whatever
}
}
---
### Author:  [@craftytech1](https://discuss.elastic.co/u/craftytech1)
#### Post date: [May 23, 2017, 5:29pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/12 "2017-05-23T17:29:15Z")
I'm in a similar situation. I'm using the config above for testing purposes and I'm getting the same result.
[ERROR][logstash.inputs.metrics] Failed to create monitoring event {:message=\>"For path: events", :error=\>"LogStash::Instrument::MetricStore::MetricNotFound"}
Does anyone have elk 5.4 working w/ x-pack? it seems everyone is having the same issue.
---
### Author:  [@Vinuja1](https://discuss.elastic.co/u/Vinuja1)
#### Post date: [May 26, 2017, 1:27pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/13 "2017-05-26T13:27:16Z")
the same for me : " icStore::MetricNotFound"}
[2017-05-26T09:26:41,001][ERROR][logstash.inputs.metrics] Failed to create mon
itoring event {:message=\>"For path: events", :error=\>"LogStash::Instrument::Metr
icStore::MetricNotFound"}"
---
### Author:  [@saikrishnagaddipati](https://discuss.elastic.co/u/saikrishnagaddipati)
#### Post date: [June 6, 2017, 10:21pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/14 "2017-06-06T22:21:52Z")
+1 Having the same issue. was anyone successful in fixing this issue?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 4, 2017, 10:22pm UTC](https://discuss.elastic.co/t/logstash-again-is-failing-to-create-monitoring-event/85147/15 "2017-07-04T22:22:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.