Multiple index creation from single configuration file

Elastic Stack Logstash
1

I am trying to generate multiple indices in one .conf file under a single tag(which is also defined in filebeat.yml) with different grok filters along with respective index generation.

The whole scenario is ,
I have installed one filebeat instance on the server from which I want to collect my logs. So,this Instance is collecting logs from the multiple prospectors which are kept on the same server as well as under the same path , Filebeat properly collecting logs from both of the sources and passes the data to my logstash server(and to my .config file on port 5044).
So now, the issue I am facing with this scenario is ES is only able to generate one Index of one source file's log data. It is somehow bypassing the other source file's log data. Any errors are not observed at any level. I have no idea why this is happening (Because not getting any errors in logs). Guide me on this issue.

2

You need to show us what your logstash configuration looks like.

1 Like
3

In general you would want something like:

if "tag1" in [tags] {  # those are tags you can set in filebeat
    elasticsearch {
        host => ["localhost:9200"]
        index => "custom-index-1"
    }
else if "tag2" in [tags] {
    elasticsearch {
       host => ["localhost:9200"]
       index => "custom-index-2"
    }
}

Unfortunately this is kind of bloated because the if could be only encapsulating the different indices but I somehow can't get that to work this way.
Hope this helps.

4

The very first idea implemented for this scenario was this only, But yes, if works only for encapsulating the different indices and not for multiple tag matching at logstash level in single .conf file I think. So, the second idea was generating multiple indices only under one tag with some simple if conditions.(reminding that ,logs are coming from different source file which are on the same server). But unfortunately that idea didn't work as well.
Thanks for your help topel.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.