Multiple Elasticsearch types output for a same log

Elastic Stack Logstash
1

Hello everyone,

I'm working on a project using ELK and I would like to know if it's possible for Logstash to store information of a same document but in different Elasticsearch types ?

Because my explanation is not very clear, I give you the following example that I have in mind :

Imagine a log : {timestamp: X, toto:1, titi:2}
I would like to know if it's possible for Logstash when it's receiving this log to do :

  • Store timestamp and 1 in Elasticsearch type "toto"
  • Store timestamp and 2 in Elasticsearch type "titi"
    in the same time ...

Of course, it could have more types than 2.

I hope to be clear enough

Thank you

2

Where are you defining the types?

3

Well, for me I define only one type that I use in output and I build it in filters...
But it seems to me that we can't put data into multiple types ?

4

Sure you can, but why?

output {
 elasticsearch {
    hosts => ["host:9220"]
    index => INDEX
    document_type => TYPE1
  }
 elasticsearch {
    hosts => ["host:9220"]
    index => INDEX
    document_type => TYPE2
  }
}
5

If I understand you correctly, you want to split an event like

{"timestamp": "X", "toto": 1, "titi:" 2}

into two events,

{"timestamp": "X", "toto": 1}

and

{"timestamp": "X", "titi:" 2}

and send each event to a separate index? If yes, use a clone filter to clone the original event and use other filters to modify each copy as desired.

6

This is what I want to do but imagine now that I have multiple types according to some fields that I can identify, is there a kind of loop that I can do depending on the number of types I identify in the message ?

7

The Logstash configuration language doesn't support loops. You might need to use a ruby filter.

8

Ok, that is what I was thinking.
Well, I will reconsider my template to see if I can do things in an other way :slight_smile:

Thank you for your replies