Create index from tags (basic question ?)

Elastic Stack Logstash
#1

Hello,

I would like to create index to split data regarding the application for example. Is-it a good idea to use tags for that ? If yes, how can I convert tags from array to string ? Because when I want to use tags, logstash tell me that I can't put tags which is an array in index which is a string.

Is-it possible to create a loop like foreach on tags ??

Thanks.

#2

Do you really want to create indexes based on all tags? That's probably not a good idea. Can you give an example of the kind of tags you have?

#3

The need is to split index on some criteria. For management, for curator, for data space allocation, ...
And give the capability to the sender to set the name of the application.

For example:

  • All messages received from this queue are APPLICATION
  • And tags contains APP_TOTO
    Then index is:
  • APPLICATION-APP_TOTO-%{+YYYY.MM.dd}

But I don't want to fix APP_TOTO, i don't want this piece of code: if 'APP_TOTO' in [tags] ...
I would like something more dynamic. Not a good idea ?

#4

You need to be very careful of shard explosion (kagillion shard problem) here.

#5

I think it's a mistake to use tags in this way. Why not put the application name in its own field? It needs to have a singular value anyway and you don't lock the use of the tags to what you need to build up your index name. A concrete problem would be if you have a grok filter that fails for certain kinds of input. Suddenly those events end up in another index with an extra "_grokparsefailure" in its name.

#6

I am sorry but I didn't find the field application that you mention.
Currently I receive logs from syslog and winlogbeat and I don't find this field.
Is-it a custom field? For example if in one application I setup log4j to use syslog, How to setup the field application?

#7

You currently use a custom tag to indicate which application an event comes from. My point is that you might just as well use a custom field to accomplish the same thing.

I can't give any specific advice without knowing how the application tags end up in the events.

#8