Hi!
How to define index name in multi-ls-on-hosts -> rabbit -> ls -> kibana chain to be further used in kibana index pattern?
It's not really clear what you are after here, can you try asking it a different way?
I have a service logging in local ls. This ls pushes messages to rabbitmq, and the last one gathers messages from multiple similar services. Then rmq pushes messages to another ls, and the last one forwards them to elasticsearch used by kibana. Kibana has got settings "indices" to configure index patterns.
Say, for two given services i'd want to push their messages to indexes jazz@yyyy.mm.dd to be then able to separate them in kibana from other indexes using jazz@* index pattern.
Which configuration(s) must deal with jazz name?
Is my attempt to elaborate the question is still ugly to be understood? 
Have a look at my answer here - Can I create different index patterns for different logs types in logstash? - it's basically the same thing.
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html will also be helpful.
I guess we still say about different use cases. Hope, this scheme will clarify my intention.
Assuming you have something in your data that can be used to determine which index it should go to, you can achieve this by using 2 elasticsearch output blocks (one for each index) covered by conditionals. If you current do not have this, you can simply add a tag or a field at the Logstash instances that collects the data and use this.
1 Like
No, I haven't. The information must be some way provided in the local (on the left side) LSs configurations. And there can be potentially infinite amount of foos and bars on the left side.
[quote="Christian_Dahlqvist, post:7, topic:66614, full:true"]If you current do not have this, you can simply add a tag or a field at the Logstash instances that collects the data and use this.
[/quote]Aha, this way seems to be appropriate, thanks, will try to dig in this direction.
Group events into a few indices based on how similar the event structures are. Be careful to not create a large amount of small, time-based indices and shards as this is very inefficient and is very likely cause problems down the road. You should easily be able to find examples of the issues around having too many indices and shards in these forums as it is a relatively common mistake.
Thanks, there isn't any reason at my case to get really many indexes, and I'm aware of the road to the hell - that is to use shards with partitioning. So the next step is to understand how to map a field (which was set with add_field) to ES index name.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.