Fetching data from a csv file and updating another file accordingly

[jatin6418 (jatin jain)]

hey
I am having two files one is a log file and one is a csv file. From log file I am reading the itemid and I want to add the name of this item which is stored in the csv file can someone help me .

input{

 file{
     type => "log"
     path =>"/home/dummy.log"
     start_position =>"beginning"
     sincedb_path => "/dev/null"
     }
   

 file{
     type => "csv"
     path =>"/home/output.csv"
     start_position =>"beginning"
     sincedb_path => "/dev/null"
     }

}

filter {

if [type] == "log" {

grok {
match => [
"message","(?<item_id>[\w-/]+)"
]
}

}

if [type] == "csv"{
csv{
separator => ","
columns => ["item_id", "item_name"]
}

   } 

}

[sjabiulla (JB)]

1. Read only log file in input. Remove the csv file type from input.

2. Extract the item id from the log using grok filter as you already did.

3. Now use the translate filter to map the item id with name . Translate filter in your scenario will look like below.

   translate {
   field => "item_id"
   destination => "item_name"
   dictionary_path => "/home/output.csv"
   }

[jatin6418 (jatin jain)]

Thnx a lot sir

I wanted to ask one more question

if my grook filter looks like
(?[\d-]+)\s(?[\w-/]+)

(here viewid represents the item_id)

and my csv file has the following structure
community_id,item_id,community_name,item_name

then how should i write the translate filter
Is there any restriction that my csv file should have only two columns in which mapping needs to be done?

[sjabiulla (JB)]

Yes, csv can only have two columns. Hence you can have csv as below. The below has only 2 columns, but the second column has all info that you need.

item_id,community_id|community_name|item_name

You can use below filters now.

translate {
field => "item_id"
destination => "item_details"
dictionary_path => "/home/output.csv"
}

mutate {
 split => { "item_details" => "|" }
 add_field => { "community_id" => "%{item_details[0]}" }
 add_field => { "community_name" => "%{item_details[1]}" }
 add_field => { "item_name" => "%{item_details[2]}" }
}

Now you will have comunity_id, community_name, item_name that are mapped by item_id via csv in the output

[jatin6418 (jatin jain)]

thnxx a lot for your help it worked for me .