Fetching data from a csv file and updating another file accordingly

Elastic Stack Logstash
1

hey
I am having two files one is a log file and one is a csv file. From log file I am reading the itemid and I want to add the name of this item which is stored in the csv file can someone help me .

input{

 file{
     type => "log"
     path =>"/home/dummy.log"
     start_position =>"beginning"
     sincedb_path => "/dev/null"
     }
   

 file{
     type => "csv"
     path =>"/home/output.csv"
     start_position =>"beginning"
     sincedb_path => "/dev/null"
     }

}

filter {

if [type] == "log" {

grok {
match => [
"message","(?<item_id>[\w-/]+)"
]
}

}

if [type] == "csv"{
csv{
separator => ","
columns => ["item_id", "item_name"]
}

   }

}

2

  1. Read only log file in input. Remove the csv file type from input.

  2. Extract the item id from the log using grok filter as you already did.

  3. Now use the translate filter to map the item id with name . Translate filter in your scenario will look like below.

    translate {
    field => "item_id"
    destination => "item_name"
    dictionary_path => "/home/output.csv"
    }

4

Thnx a lot sir

I wanted to ask one more question

if my grook filter looks like
(?[\d-]+)\s(?[\w-/]+)

(here viewid represents the item_id)

and my csv file has the following structure
community_id,item_id,community_name,item_name

then how should i write the translate filter
Is there any restriction that my csv file should have only two columns in which mapping needs to be done?

5

Yes, csv can only have two columns. Hence you can have csv as below. The below has only 2 columns, but the second column has all info that you need.

item_id,community_id|community_name|item_name

You can use below filters now.

translate {
field => "item_id"
destination => "item_details"
dictionary_path => "/home/output.csv"
}

mutate {
 split => { "item_details" => "|" }
 add_field => { "community_id" => "%{item_details[0]}" }
 add_field => { "community_name" => "%{item_details[1]}" }
 add_field => { "item_name" => "%{item_details[2]}" }
}

Now you will have comunity_id, community_name, item_name that are mapped by item_id via csv in the output

1 Like
6

thnxx a lot for your help it worked for me .

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.