Logstash configuration- How to do dynamic mapping?

I have one requirement to create dynamic mapping in my Logstash configuration to map two different columns. Please find below details: -
Below is the ruby code I'm using in my logstash config:

![image|690x351](upload://r5E8Q9BVhusDRM5TRNVv66mCnHH.png)

In this, I have hardcoded "doc_lob" on the basis of "user". But I have to make this dynamic for "n" number of users to accomodate, so that everytime I don't need to change the code. It can be done by passin the values via an excel sheet or something!

I hope I am clear in explaining my requirement. Can you plz help me with any lead on this?

Thanks,
Nisha

Hello,

It is not clear what you are trying to do, can you share your logstash config using the </> code format option? Do not share images.

I have one requirement to create dynamic mapping in my Logstash configuration to map two different columns. Please find below details: -
Below is the ruby code I'm using in my logstash config:
ruby {
            code =>'table = event.get("[hive][table]");
            user = event.get("[hive][ugi]");
            event.set("[dataconsumption][access_flag]",table);

            if(user == "biuser@NPRD.BIGD.BE")
                event.set("[dataconsumption][doc_lob]","Data & Analytics");

            elsif(user == "hive/nprd.bigd.BE")
                event.set("[dataconsumption][doc_lob]","Marketing Automation");

            elsif(user == "hue/el755.nprd.bigd.be")
                event.set("[dataconsumption][doc_lob]","Analytics Boost");

            elsif(user == "impala.nprd.bigd.BE")
                event.set("[dataconsumption][doc_lob]","Business insights");

            elsif(user == "bhdg@PROD.BE")
                event.set("[dataconsumption][doc_lob]","Product & network Intelligence");
			else
                event.set("[dataconsumption][doc_lob]","Others");
            end'
	}
	
In this, I have hardcoded "doc_lob" on the basis of "user". But I have to make this dynamic for "n" number of users to accomodate, so that everytime I don't need to change the code. It can be done by passin the values via an excel sheet or something!

I hope I am clear in explaining my requirement. Can you plz help me with any lead on this?

Thanks,
Nisha

@leandrojmp Could you please let me know if this thing(dynamic lookup) is possible in Kibana?

If I understood correctly you are trying to populate a field based on the content of another field?

One way to do that in logstash is using the translate filter where you would have a dictionary with key-value pairs.

The keys in this dictionary would be the value of the first field, and the value of this dictionary would be the one that you want to populate into the new field.

So, based in your example, you would need a dictionary like this in an external file.

"biuser@NPRD.BIGD.BE": "Data & Analytics"
"hive/nprd.bigd.BE": "Marketing Automation"
"hue/el755.nprd.bigd.be": "Analytics Boost"
"impala.nprd.bigd.BE": "Business insights"
"bhdg@PROD.BE": "Product & network Intelligence"

Then you would need this translate filter in your pipeline

translate {
	field => "user"
	destination => "[dataconsumption][doc_lob]"
	dictionary_path => "/path/to/the/dictionary/file.yml"
	refresh_interval => 300
	fallback => "Others"
}

What this filter do is check if the value of the user field exists as a key in the dictionary file, if it exists, it will get the value for that key and set it as the value of the field [dataconsumption][doc_lob], if it does not exists, it will set the value of the field [dataconsumption][doc_lob] as Others, because the fallback option is set with this value.

For example, if the user field has the value impala.nprd.bigd.BE, after the event pass through the translate filter, it will have the field [dataconsumption][doc_lob] with the value Business insights.

You will need to build a process to update the dictionary file, the refresh_interval in the filter will tell logstash to check for changes in the file after that interval has passed.

There are other filters that can be used to enrich your data like the jdbc_static and the memcached, but the translate filter is the easiest one to use.

1 Like

Thanks @leandrojmp! Yesterday, I tried Translate only and it served the purpose well.
Thanks for your reply.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.