I'm looking for some guidance on using the JMX input plugin to gather multiple JMX streams. Ideally I was thinking to create unique indexes for each JMX stream by using:
if "spefic tag" in [tags] {..}
However, I have been unable to get it to work. The only thing I am seemingly able to use for the output into ES is:
if [type] == "jmx" {..}
By using [type] everything seems like it will need to go into the same index.
Is that the idea? Should all JMX ingests go into the same index even it there are dozens of them?
Can anyone provide some guidance / best practices?
By using [type] everything seems like it will need to go into the same index.
Why?
Is that the idea? Should all JMX ingests go into the same index even it there are dozens of them?
Why not? It's the same kind of data with (presumably) the same set of fields regardless of which JVM you're monitoring so it's quite natural to keep them all in the same index.
if [type] == "jmx" {
elasticsearch {
hosts => ["node1:9200", "node2:9200", "node3:9200"]
index => "dev-jmx-%{+YYYY.MM.dd}"
user => "elastic"
password => "****"
}
}
}
Ideally, I would want to separate indexes for Dev, QA, and PROD. I need to create the "if statement" because I have several other *.conf files in /etc/logstash/conf.d and my understanding is that they all are compressed into a single LS configuration upon start up. That seemingly requires that I use if statements on all of the output blocks so that the varying data go into the proper indexes.
Is this not the case? I've not found a different way to do it.
As mentioned, I'd like to separate DEV, QA, and PROD. But as a novice I was thinking possibly per application as well. If not, what fields would/could one use to differentiate environment or application for visualizations?
Ideally, I would want to separate indexes for Dev, QA, and PROD. I need to create the "if statement" because I have several other *.conf files in /etc/logstash/conf.d and my understanding is that they all are compressed into a single LS configuration upon start up.
Yes.
That seemingly requires that I use if statements on all of the output blocks so that the varying data go into the proper indexes.
Yes, it sounds like you need to wrap all your outputs in some kind of conditionals.
If not, what fields would/could one use to differentiate environment or application for visualizations?
That's up to you. There are no standard fields for that.
Any ideas why the tag isn't picked up for the conditional?
Does the event actually have a "dev-jmx" tag?
I don't see any support in the documentation that the JMX configuration JSON can have a "tags" key. However, the jmx input plugin itself support a tags option (like all plugins do), and more importantly add_field. So, I suggest you change your config to
elasticsearch {
index => "%{environment}-jmx-%{+YYYY.MM.dd}"
...
}
If you want to pull JMX metrics from JVMs in other environments you can just make sure to keep the environment field up to date and it'll get included in the index name.
I was just trying to add the [tags] key to the JSON config. It would make sense that it's not being recognized if it is not an available key.
I appreciate your suggestion of using "add_field" to create an environment distinction. However, in this case there is still only one JMX index being created at a time. I guess I don't really need separate indices as long as I could visualize based upon separate JVM's but I feel like it would make it easier.
Is there a better way to monitor JVM's? We have single systems that run 4 or 5 JVM's for different purposes, what can we do with a dozen or so servers?
I appreciate your suggestion of using "add_field" to create an environment distinction. However, in this case there is still only one JMX index being created at a time.
One per environment.
I guess I don't really need separate indices as long as I could visualize based upon separate JVM's but I feel like it would make it easier.
I don't see why. If you want to split things up further on you can always do that later. Keep in mind that indexes are a resource with non-zero cost.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.