Transforming data WHILE indexing


(Kavya Sabu) #1

How can we transform data while it gets indexed? Some examples please.


(David Pilato) #2

The best thing to do is to transform data before it gets indexed!

So use logstash for example.


(Kavya Sabu) #3

Ok thanks! In the elasticsearch docs it's mentioned that "As of now there really isn’t a feature to use in its place other than transforming the document in the client application." Could you elaborate on this sentence please. What exactly do they mean by transforming in client application?

This is the link: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-transform.html


(David Pilato) #4

Exactly what I meant. That you need to transform your document before sending it to elasticsearch.


(Kavya Sabu) #5

So we use the same method for transforming given in the examples in the above link but we only have to do it in logstash?


(Isabel Drost-Fromm) #6

Not quite - you can use Logstash to achieve the same results as with that method. You can find some typical examples for what transforming data with Logstash looks like here: https://www.elastic.co/blog/little-logstash-lessons-part-using-grok-mutate-type-data

Out of curiosity: What type of data transformation are you looking for exactly? What type of data are you planning to index?


(Kavya Sabu) #7

I'll be using log data. I'll have to perform some operations such as converting data from bits to bytes/ seconds to mins etc.


(Isabel Drost-Fromm) #8

Without having deep Logstash knowledge (might be better to ask in their group in discuss) to me that sounds like something that should be well doable in Logstash.


(Kavya Sabu) #9

Thank you anyway.


(system) #10