How to add custom filed to a existing elastic data source?

(Sabyasachi Mallick) #1

i have an existing elastic data source with around 42000 rows. and also i am updating it daily.I want to add a custom filed , whose value will depend on two other fields .
e.g i have 2 existing field P and Q
i want to add R which ll depend on values of P and Q ,
i know i can do it by If ..else mutate in logstash filter while adding or updating data . But the problem is, i have more than 100 different conditions to be taken care. so writing 100 if..else statement is time consuming and bad idea.
is there any way to it from python, so that i can use list or dictionary to do mapping or
Is there any better solution to it?

Any help ll be appreciated !!

(Magnus Bäck) #2

Tell us more about your configuration. Are you currently using Logstash to feed ES? Are you using the jdbc input? Please give an example of the kind of conditional and mutate filter you'd have to write. I suspect we can use a translate filter for this.

Consider moving this topic to the Logstash category.

(Sabyasachi Mallick) #3

yea!! I am using logstash to feed ES .
and also JDBC to fetch data from MySql DB in one case and in another case i am pulling data from csv files .
But in both the case i am using logstash to feed data into ES.
I have two fileds "product name" and "milestone name" . i want to create a new filed "release name" which is based on product and milestone filed.
if [product name]=="x" AND [milestone name]=="Y"{mutate{add_filed=>["release name", "1.0"]}}
else mutate{add_filed=>["release name", "2.0"]}

i have 100 different products and mile stones are there , so it is a bad idea to write 100's of conditional statements.

(Magnus Bäck) #4

Those conditionals could be generated by a script.

Alternatively you could combine product name and milestone name into a single field and use that field to look up a value in a table provided by a translate filter.

mutate {
  add_field => {
    "[@metadata][release_name_key]" => "%{product name}_%{milestone name}"
translate {
  dictionary_path => "/path/to/file.csv"
  field => "[@metadata][release_name_key]"
  destination => "release name"

/path/to/file.csv could then look like this:


(Sabyasachi Mallick) #5

That worked like a charm !! Thanks a lot
one more thing. while using file for hashing , how can i give default vale?
If ther is filed which does not match to any of the row in hash file , then i wan to give a default value for all such type of fields . how can i do that?

i think , one way to do is add the filed before with a default value then apply translate filter.
or is there any way to give default value inside translate filter?

(Magnus Bäck) #6

i think , one way to do is add the filed before with a default value then apply translate filter.

Yes, exactly. Just make sure to set override => true in the translate filter.

(Sabyasachi Mallick) #7

sure...Thanks for your quick help !1 :slight_smile:

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.