Wasn't sure whether to post this question in Logstash forum or ElasticSearch since it can technically be in either.
Scenario: Multiple applications logging to a single ELK stack. Each application's logs will differ in format. All application logs will log to the same index (logstash-%date).
Question: How does one maintain index/mapping templates for each application stack in a sustainable and organized manner? Meaning if the log structure changes, it should be easy to change the mapping for that specific app without affecting other apps. It should also be versionable if possible.
I was thinking for this, best use would be to store the templates on logstash servers in a config directory where they are applied by logstash on the first message of a new index. The flat files could be versioned and easily modified/replaced.
Would love to hear how other people are handling this and what options and possibilities there are that I may be missing.
How does one maintain index/mapping templates for each application stack in a sustainable and organized manner? Meaning if the log structure changes, it should be easy to change the mapping for that specific app without affecting other apps.
If all applications log to the same indexes keep in mind that the mappings need to be the same for a given field, i.e. a field named x will have mapping y regardless of the type.
I was thinking for this, best use would be to store the templates on logstash servers in a config directory where they are applied by logstash on the first message of a new index. The flat files could be versioned and easily modified/replaced.
Not sure why you're talking about plural here—if all applications share the same index then there will only be one template.
I prefer disabling Logtash's template management and storing the templates directly in ES but I don't think there are any significant advantages (or disadvantages) in doing so.
Sorry for the confusion with the multiple templates. You are right that there will be a single index, and thus a single template. Perhaps I may be over complicating things a bit, so while i have your attention maybe i can ask this question first:
Is there a way to have kibana search through all indexes at once? For instance if we had iis-%date tomcat-%date mysql-%date, could kibana still do a single query look up through all of them at once in order to track a transaction through all stacks?
Now on to answering some of the concerns you brought up...Yes a single index and thus a single template. Each app will have different fields, however, just the nature of the beast. some fields may overlap such as timestamps and success/fail fields, but not all fields will be the same.
I keep using plural because i was thinking the template could be broken down into multiple pieces instead of one large file, same way the logstash configuration is. In the config we have 00_input.conf 10_iis.conf 20_tomcat.conf 30_mysql.confg 99_output.conf. I was expecting to be able to do a similar thing for the template where each piece is separated out.
The reason is changes to tomcat app are frequent, and fields may be added or removed as needed and thus the template would have to change. I am attempting to plan ahead and make this as easy going forward as possible and establish some procedures and standards.
Am I way over complicating this? Thanks for your help!
Is there a way to have kibana search through all indexes at once? For instance if we had iis-%date tomcat-%date mysql-%date, could kibana still do a single query look up through all of them at once in order to track a transaction through all stacks?
Yes, as long as there's a single index pattern that selects all those indexes (and no indexes you don't want to search). You may want to prefix all your log indexes.
I keep using plural because i was thinking the template could be broken down into multiple pieces instead of one large file, same way the logstash configuration is. In the config we have 00_input.conf 10_iis.conf 20_tomcat.conf 30_mysql.confg 99_output.conf. I was expecting to be able to do a similar thing for the template where each piece is separated out.
Okay. Sure, but merging those files will be up to you.