Logstash JMX input plugin configuration error

Hello,

I am seeking some guidance regarding the configuration of the logstash jmx input plugin. I apologize for the formatting/wording of this as I'm quite certain it's wrong. Hopefully you can overlook this and provide assistance in spite of my ignorance.

I have a single logstash docker container running logstash 6.2.4

The Dockerfile is formatted similar to:

FROM docker.elastic.co/logstash/logstash:6.2.4
RUN /usr/share/logstash/bin/logstash-plugin install logstash-input-jmx
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
RUN rm -f /usr/share/logstash/pipeline/pipelines.yml
RUN rm -f /usr/share/logstash/config/logstash.yml
COPY --chown=logstash:logstash logstash.conf /usr/share/logstash/pipeline/
COPY --chown=logstash:logstash pipelines.yml /usr/share/logstash/pipeline/
COPY --chown=logstash:logstash config1 config2 config3 /usr/share/logstash/pipeline/
COPY --chown=logstash:logstash logstash.yml /usr/share/logstash/config/

I chose to follow the example configuration provided on: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jmx.html

My logstash.conf:

input {
jmx {
path => "/usr/share/logstash/pipeline/lmsgrad"
polling_frequency => 600
type => "jmx"
nb_thread => 4
}
}

output {
elasticsearch {
hosts => ["http://elasticsearch:9200"]
index => "jmx-logs-%{+YYYY.MM.dd}"
}
}

My pipelines.yml:

  • pipeline.id: main
    path.config: "/usr/share/logstash/pipeline"
    path.config: "/usr/share/logstash/pipeline/config1"
    path.config: "/usr/share/logstash/pipeline/config2"
    path.config: "/usr/share/logstash/pipeline/config3"

My config1-3 is similar to:

{
//Required, JMX listening host/ip
"host" : "10.10.10.10",
//Required, JMX listening port
"port" : 9012,
//Optional, the username to connect to JMX
"username" : "user",
//Optional, the password to connect to JMX
"password": "pass",
//Optional, use this alias as a prefix in the metric name. If not set use _
"alias" : "test.homeserver.elasticsearch",
//Required, list of JMX metrics to retrieve
"queries" : [
{
//Required, the object name of Mbean to request
"object_name" : "java.lang:type=Memory",
//Optional, use this alias in the metrics value instead of the object_name
"object_alias" : "Memory"
}, {
"object_name" : "java.lang:type=Runtime",
//Optional, set of attributes to retrieve. If not set retrieve
//all metrics available on the configured object_name.
"attributes" : [ "Uptime", "StartTime" ],
"object_alias" : "Runtime"
}, {
//object_name can be configured with * to retrieve all matching Mbeans
"object_name" : "java.lang:type=GarbageCollector,name=",
"attributes" : [ "CollectionCount", "CollectionTime" ],
//object_alias can be based on specific value from the object_name thanks to ${}.
//In this case ${type} will be replaced by GarbageCollector...
"object_alias" : "${type}.${name}"
}, {
"object_name" : "java.nio:type=BufferPool,name=
",
"object_alias" : "${type}.${name}"
} ]
}

My logstash.yml is similar to:

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://elasticsearch:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: changeme

Files have been modified to protect the hosts & passwords as required by common sense.

Running this container with the above configuration provides me with the following error:

[2018-07-16T22:22:54,430][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

This unfortunately does not provide a lot of guidance regarding where the error is.

Would anyone happen to have any suggestions?

Thanks.

This is your problem, I think. A quick test suggests that if you have more than one path.config line then the last one wins. So this is really equivalent to

- pathline.id: main
  path.config: "/usr/share/logstash/pipeline/config3"

And that tells it to read every file in that directory (not just .conf files) and concatenate them before parsing the config. Given that it fails at the very first character, the output of

ls -l /usr/share/logstash/pipeline/config3

or

find /usr/share/logstash/pipeline/config3 -type f -exec cat {} \; | sed 5q

might be informative. If that does not get you there then enable debug logging and start logstash with '--config.debug' to see what configuration it is trying to parse and then walk the cat back from there.

1 Like

Thanks for the response!

The entries for "config1,config2,config3" are actually files not folders. So the commands suggested returns the file or the first 5 lines of the file listed above under My config1-3.

I actually had enabled debugging and it allowed me to see that if I didn't add the last three "path.config" entries then the files I placed in the "/usr/share/logstash/pipeline" folder they were ignored because they didn't match the glob pattern.

After removing the 3 additional path.config entries I am still seeing it read all of the config files (as previously, not sure why):

[2018-07-17T00:54:44,219][DEBUG][logstash.api.service ] [api-service] start,
[2018-07-17T00:54:44,286][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/usr/share/logstash/CONTRIBUTORS", "/usr/share/logstash/Gemfile", "/usr/share/logstash/Gemfile.lock", "/usr/share/logstash/LICENSE", "/usr/share/logstash/NOTICE.TXT", "/usr/share/logstash/bin", "/usr/share/logstash/config", "/usr/share/logstash/data", "/usr/share/logstash/lib", "/usr/share/logstash/logstash-core", "/usr/share/logstash/logstash-core-plugin-api", "/usr/share/logstash/modules", "/usr/share/logstash/pipeline", "/usr/share/logstash/tools", "/usr/share/logstash/vendor"]},
[2018-07-17T00:54:44,289][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/config1"},
[2018-07-17T00:54:44,303][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/config2"},
[2018-07-17T00:54:44,308][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/config3"},
[2018-07-17T00:54:44,309][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/logstash.conf"},
[2018-07-17T00:54:44,310][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/pipelines.yml"},

Unfortunately I still see the same error message generated before the container restarts:

[2018-07-17T01:10:18,406][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x6af6813d run>"},
[2018-07-17T01:10:18,455][DEBUG][logstash.agent ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main},
[2018-07-17T01:10:18,454][DEBUG][logstash.inputs.metrics ] Metric: input started
[2018-07-17T00:54:47,698][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]},

Any additional suggestions or data that I could provide from the logs?

I have been able to setup other plugins successfully but this one appears to be eluding me.

I appreciate any feedback or assistance you might be able to provide.

As I said, run with --config.debug and see what it is trying to parse. Is it possible your configs have byte order marks in them?

1 Like

Thanks for the clarification! I thought just setting the "log_level: debug" environment variable was all the more verbose the logging could get. I looked up how to initiate the process with "--config.debug" and now I'm seeing more data but I'm not seeing anything additional that appears useful.

Taking your advice regarding byte order marks, I re-wrote all the configs manually in vim on the docker build server before building the image and I'm not seeing any changes.

Here is the full log:
https://pastebin.com/WRBLdPTc

OK, so the problem is that in your pipelines.yml you have told logstash to reads all the files in the directory /usr/share/logstash/pipeline as part of the logstash configuration. But you have also put your jmx plugin configuration file in that directory (as well as pipelines.yml), so logstash treats the JMX JSON as part of its configuration and fails to parse it.

I think the solution is a simple as updating the pipelines.yml to be

- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline/logstash.conf"
1 Like

Excellent!

I didn't realize that it would be reading the files in alphabetical order and jmx comes before logstash naturally.

So i moved the jmx.conf to the '/usr/share/logstash/data' folder after attempting to modify the pipelines.yml to point directly at the path you suggested and that appears to have worked!

Because now I'm getting a different error message:

[2018-07-19T02:13:30,245][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 16, column 1 (byte 253) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

I examined all of my files and discovered none of them have 16 lines.

I am not sure which file it's referring to now. I would assume that since it's the first error, it would be referencing the logstash.conf as that appears to be the first file it's parsing:

[2018-07-19T02:13:26,704][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/logstash.conf"},
[2018-07-19T02:13:26,711][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/usr/share/logstash/pipeline/pipelines.yml"},
[2018-07-19T02:13:26,610][DEBUG][logstash.api.service ] [api-service] start,
[2018-07-19T02:13:26,691][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/usr/share/logstash/CONTRIBUTORS", "/usr/share/logstash/Gemfile", "/usr/share/logstash/Gemfile.lock", "/usr/share/logstash/LICENSE", "/usr/share/logstash/NOTICE.TXT", "/usr/share/logstash/bin", "/usr/share/logstash/config", "/usr/share/logstash/data", "/usr/share/logstash/lib", "/usr/share/logstash/logstash-core", "/usr/share/logstash/logstash-core-plugin-api", "/usr/share/logstash/modules", "/usr/share/logstash/pipeline", "/usr/share/logstash/tools", "/usr/share/logstash/vendor"]},
[2018-07-19T02:13:26,734][DEBUG][logstash.config.pipelineconfig] -------- Logstash Config ---------,
[2018-07-19T02:13:26,739][DEBUG][logstash.config.pipelineconfig] Config from source {:source=>LogStash::Config::Source::MultiLocal, :pipeline_id=>:main},
[2018-07-19T02:13:26,745][DEBUG][logstash.config.pipelineconfig] Config string {:protocol=>"file", :id=>"/usr/share/logstash/pipeline/logstash.conf"},
[2018-07-19T02:13:26,747][DEBUG][logstash.config.pipelineconfig]

I'm confused where it's finding the extra lines. My config files have 10,14,4 and 2 lines each.

Unless it's referring to a default config file that I did not modify but is no longer compatible with my configuration.

Any thoughts?

My gut says that it is concatenating the files together in alphabetical order before attempting to parse, which could throw off the line number output by the parser. This output could definitely be more helpful, and if that's the case, I'll open a ticket in the morning to improve the experience.

1 Like

As I said, it is concatentating all the files in the directory, which would include the logstash.yml. Use

- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline/logstash.conf"
1 Like

I apologize for the confusion.

This is with the pipelines.yml containing the following:

  • pipeline.id: main
    path.config: "/usr/share/logstash/pipeline/logstash.conf"

As you recommended earlier.

OK, if you do the --config.debug thing again I can take a look later.

1 Like

Here is a link to the output log with the recommended configuration:

https://pastebin.com/dBFQEjAD

How strange! You can see at lines 101 and 102 that it decides to read logstash.conf and pipelines.yml as part of the logstash conf, even though, as we can see at 127 that you have

path.config: "/usr/share/logstash/pipeline/logstash.conf"

At 131 it starts printing the merged configuration, and at line 147 (i.e. line 16 of the config) it hits

- pipeline.id: main

EDIT: Just realized what the problem is. It does not know you are trying to use pipelines, so it is looking in path.config for configurations. pipelines.yml MUST be in path.settings, which for you is /usr/share/logstash/config

[2018-07-19T16:17:12,323][DEBUG][logstash.runner ] path.settings: "/usr/share/logstash/config",

1 Like

Fantastic!

Now it's reading the pipelines.yml and launching the JMX input plugin but it's not finding my config file:

[2018-07-19T23:17:28,570][DEBUG][logstash.inputs.jmx      ] config LogStash::Inputs::Jmx/@path = "/usr/share/logstash/data/zmx.conf",
[2018-07-19T23:17:28,571][DEBUG][logstash.inputs.jmx      ] config LogStash::Inputs::Jmx/@polling_frequency = 600,
[2018-07-19T23:17:28,571][DEBUG][logstash.inputs.jmx      ] config LogStash::Inputs::Jmx/@type = "jmx",
[2018-07-19T23:17:28,571][DEBUG][logstash.inputs.jmx      ] config LogStash::Inputs::Jmx/@nb_thread = 4,
[2018-07-19T23:17:28,802][INFO ][logstash.inputs.jmx      ] Initialize 4 threads for JMX metrics collection,
[2018-07-19T23:17:28,817][DEBUG][logstash.inputs.jmx      ] Wait config to retrieve from queue conf,
[2018-07-19T23:17:28,831][DEBUG][logstash.inputs.jmx      ] Wait config to retrieve from queue conf,
[2018-07-19T23:17:28,838][DEBUG][logstash.inputs.jmx      ] Wait config to retrieve from queue conf,
[2018-07-19T23:17:28,840][INFO ][logstash.inputs.jmx      ] Loading configuration files in path {:path=>"/usr/share/logstash/data/zmx.conf"},
[2018-07-19T23:17:28,841][DEBUG][logstash.inputs.jmx      ] Wait config to retrieve from queue conf,
[2018-07-19T23:17:28,845][ERROR][logstash.inputs.jmx      ] No such file or directory - No such directory: /usr/share/logstash/data/zmx.conf,

Is there any designated location for jmx plugin config files? Maybe that's the solution to my problem.

Here's the full log file:

https://pastebin.com/hLg9eHeK

The pastebin output you posted earlier suggest that there is actually a /usr/share/logstash/zmx.conf

1 Like

Sorry for the confusion, I don't actually know when it's parsing the jmx configuration.

My first attempt to resolve the issue of allowing logstash to stop parsing the jmx.conf I had placed in the pipelines folder was to rename it (zmx.conf) assuming it was parsing the files in alphabetical order while it was in the pipelines folder. Next attempt moved it to the '/usr/share/logstash/' folder in an attempt to place it in a location where it should have permissions to the file even though I had executed the command: 'COPY --chown=logstash:logstash zmx.conf /usr/share/logstash/' from the Dockerfile to build the image.

I am not sure what causes the behavior I'm seeing in the container using the plugin and it's a fairly high priority project I'm working on so I'm actively seeking a solution.

I have the current configuration files hosted here:
http://pasted.co/11a583e4

Your logstash.yml is a copy of logstash.conf. That's going to get you a whole bunch of errors.

1 Like

Correct.

Unfortunately I failed to paste the contents of the logstash.yml and just ended up pasting the contents of the logstash.conf again as it was already on my clipboard.

I have an updated link:
http://pasted.co/ded88e65

And with that configuration what result are you seeing?

1 Like

Are there reserved directories for plugin configuration files I should be using?

Recent log file capture:
http://pasted.co/ce514d0f

[2018-07-23T18:02:15,971][ERROR][logstash.inputs.jmx      ] No such file or directory - No such directory: /usr/share/logstash/data/zmx.conf,
[2018-07-23T18:02:15,978][ERROR][logstash.inputs.jmx      ] org/jruby/RubyDir.java:146:in `initialize',
org/jruby/RubyDir.java:383:in `foreach',
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-jmx-3.0.6/lib/logstash/inputs/jmx.rb:326:in `run',
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:514:in `inputworker',
/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:507:in `block in start_input',