Sending unique filename pattern to unique Index?

Elastic Stack Logstash
1

Hi,

How can I send different log files to different indexs?

For example:
I read with the "filebeat" a file called: abc.full.log and a file called xyz.cdr and send both to LogStash. How can I make that abc.full.log will go to Index A and xyz.cdr will got to Index B.

Many thanks,
Tomer

2

That's most likely a bad idea since you don't want to have too many indexes, but to answer your question both Logstash and Filebeat populate events with the path to the file an event was read from. In the Filebeat case I think that field is source. You can then use e.g. a grok filter to extract the piece you want to use directly in the index name.

1 Like
3

Hi,

Thank you for the answer and the advise.

Regarding the advice, isn't it better to have indexs for every type of data. The reasoning behind the question is if we query the ES very frequently, wouldn't it be easier for the ES if we query a unique index? Therefor there should be unique indexs for every type of information?

4

Regarding the advice, isn't it better to have indexs for every type of data. The reasoning behind the question is if we query the ES very frequently, wouldn't it be easier for the ES if we query a unique index? Therefor there should be unique indexs for every type of information?

It might be beneficial (it depends on the kind of queries you make), but it needs to be balanced against the overhead of having lots of indexes.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.