JMX plugin guidance

hburnswell · June 18, 2018, 3:31am

All,

I'm looking for some guidance on using the JMX input plugin to gather multiple JMX streams. Ideally I was thinking to create unique indexes for each JMX stream by using:

if "spefic tag" in [tags] {..}

However, I have been unable to get it to work. The only thing I am seemingly able to use for the output into ES is:

if [type] == "jmx" {..}

By using [type] everything seems like it will need to go into the same index.

Is that the idea? Should all JMX ingests go into the same index even it there are dozens of them?

Can anyone provide some guidance / best practices?

Thanks in advance,

HB

magnusbaeck · June 18, 2018, 5:59am

By using [type] everything seems like it will need to go into the same index.

Why?

Is that the idea? Should all JMX ingests go into the same index even it there are dozens of them?

Why not? It's the same kind of data with (presumably) the same set of fields regardless of which JVM you're monitoring so it's quite natural to keep them all in the same index.

hburnswell · June 18, 2018, 1:25pm

Thank you for the response.

Here is my config file from /etc/logstash/conf.d:

input {

    jmx {

            path => "/etc/logstash/json"
            polling_frequency => 15
            type => "jmx"
            nb_thread => 4

    }

}

output {

    if [type] == "jmx" {

            elasticsearch {

                    hosts => ["node1:9200", "node2:9200", "node3:9200"]
                    index => "dev-jmx-%{+YYYY.MM.dd}"
                    user => "elastic"
                    password => "****"

            }

    }

}

Ideally, I would want to separate indexes for Dev, QA, and PROD. I need to create the "if statement" because I have several other *.conf files in /etc/logstash/conf.d and my understanding is that they all are compressed into a single LS configuration upon start up. That seemingly requires that I use if statements on all of the output blocks so that the varying data go into the proper indexes.

Is this not the case? I've not found a different way to do it.

As mentioned, I'd like to separate DEV, QA, and PROD. But as a novice I was thinking possibly per application as well. If not, what fields would/could one use to differentiate environment or application for visualizations?

Thanks.

magnusbaeck · June 18, 2018, 1:32pm

Ideally, I would want to separate indexes for Dev, QA, and PROD. I need to create the "if statement" because I have several other *.conf files in /etc/logstash/conf.d and my understanding is that they all are compressed into a single LS configuration upon start up.

Yes.

That seemingly requires that I use if statements on all of the output blocks so that the varying data go into the proper indexes.

Yes, it sounds like you need to wrap all your outputs in some kind of conditionals.

If not, what fields would/could one use to differentiate environment or application for visualizations?

That's up to you. There are no standard fields for that.

hburnswell · June 18, 2018, 2:13pm

Thanks. I'm trying to figure out how to designate unique indexes.

If I use [type] == "jmx" all JMX streams will go into the same index. I've tried to use [tags] in the JSON conf as:

{
"host" : "jmxnode1",
"port" : 9400,
"tags" : "dev-jmx",
"queries" : [
{
"object_name" : "java.lang:type=Memory",
"object_alias" : "Memory"
}, {
"object_name" : "java.lang:type=Runtime",
"attributes" : [ "Uptime", "StartTime" ],
"object_alias" : "Runtime"
}, {
"object_name" : "java.lang:type=GarbageCollector,name=",
"attributes" : [ "CollectionCount", "CollectionTime" ],
"object_alias" : "${type}.${name}"
}, {
"object_name" : "java.nio:type=BufferPool,name=",
"object_alias" : "${type}.${name}"
} ]
}

And then the conditional for the output block:

output {

if "dev-jmx" in [tags] {

        elasticsearch {

                hosts => ["node1:9200", "node2:9200", "node3:9200"]
                index => "dev-jmx-%{+YYYY.MM.dd}"
                user => "elastic"
                password => "****"

        }

}

}

However, the tag doesn't seem to be picked up in the output.

Any ideas why the tag isn't picked up for the conditional?
Is there a different way to allow for a unique conditional to work?

Thanks.

magnusbaeck · June 18, 2018, 6:23pm

Any ideas why the tag isn't picked up for the conditional?

Does the event actually have a "dev-jmx" tag?

I don't see any support in the documentation that the JMX configuration JSON can have a "tags" key. However, the jmx input plugin itself support a tags option (like all plugins do), and more importantly add_field. So, I suggest you change your config to

jmx {
  path => "..."
  ...
  add_field => {
    "environment" => "dev"
  }
}

and your elasticsearch output configuration to:

elasticsearch {
  index => "%{environment}-jmx-%{+YYYY.MM.dd}"
  ...
}

If you want to pull JMX metrics from JVMs in other environments you can just make sure to keep the environment field up to date and it'll get included in the index name.

hburnswell · June 19, 2018, 12:55am

@magnusbaeck Thank you for the response.

I was just trying to add the [tags] key to the JSON config. It would make sense that it's not being recognized if it is not an available key.

I appreciate your suggestion of using "add_field" to create an environment distinction. However, in this case there is still only one JMX index being created at a time. I guess I don't really need separate indices as long as I could visualize based upon separate JVM's but I feel like it would make it easier.

Is there a better way to monitor JVM's? We have single systems that run 4 or 5 JVM's for different purposes, what can we do with a dozen or so servers?

I am going about this wrong?

magnusbaeck · June 19, 2018, 6:03am

I appreciate your suggestion of using "add_field" to create an environment distinction. However, in this case there is still only one JMX index being created at a time.

One per environment.

I guess I don't really need separate indices as long as I could visualize based upon separate JVM's but I feel like it would make it easier.

I don't see why. If you want to split things up further on you can always do that later. Keep in mind that indexes are a resource with non-zero cost.

hburnswell · June 20, 2018, 11:20pm

@magnusbaeck Thank you for your help on my inquiry, much appreciated.

I managed to do what I intended as follows:

input {

    jmx {

            path => "/etc/logstash/json-dev-jmx"
            polling_frequency => 15
            type => "jmx"
            nb_thread => 4
            tags => "dev-jmx"

    }

    jmx {

            path => "/etc/logstash/json-qa-jmx"
            polling_frequency => 15
            type => "jmx"
            nb_thread => 4
            tags => "qa-jmx"

    }

}

output {

    if [type] == "jmx" {

            if "dev-jmx" in [tags] {

                    elasticsearch {

                            hosts => ["node1:9200", "node2:9200", "node3:9200"]
                            index => "dev-jmx-%{+YYYY.MM.dd}"
                            user => "elastic"
                            password => "*****"

                    }

            }

            else if "qa-jmx" in [tags] {

                   elasticsearch {

                            hosts => ["node1:9200", "node2:9200", "node3:9200"]
                            index => "qa-jmx-%{+YYYY.MM.dd}"
                            user => "elastic"
                            password => "*****"

                    }

            }

    }

}

I am going to experiment a bit on multi and single indexes but the option is there.

Thanks again..

system · July 18, 2018, 11:29pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.