Using csv file as dictionary for translate plugin

[leeyu]

Hi all,

I am new to elastic.
Today I was trying to perform translate on a csv file.
Including the dictionary in the conf works fine but logstash always shut down if I try to load the dictionary from a csv file.

Can anyone help me with this issue?
Thanks

My translate code
</>
translate {
dictionary_path => "/data/ELK_data/map.csv"
field => "event_id"
destination => "event_name"
}
</>

map.csv
</>
event_id, event_name
1,"Event #1"
2,"Event #2"
3,"Event #3"
4,"Event #4"
</>

target.csv
</>
id,event_id
1,4
2,3
3,2
4,1
</>

[Badger]

Did you parse your CSV file using

csv { columns => [ "id", "event_id" ] }

[leeyu]

Hi Badger,
Thanks for reply,
I used autodetect_column_names and autogenerate_column_names.
I think there should be sth wrong with my csv.

[Badger]

Can you post a complete example of a configuration that is reproducible. </> is not markdown, it is an icon on the toolbar that tells the browser to apply markdown. If you select a multiline block of text then clicking on </> will indent the text by 4 spaces and change

event_id, event_name
1,"Event #1"
2,"Event #2"
3,"Event #3"
4,"Event #4"

to

event_id, event_name
1,"Event #1"
2,"Event #2"
3,"Event #3"
4,"Event #4"

This matters a lot when your data, for example, has multiple spaces. It is the difference between

Foo bar
Baz ...

where the browser compresses multiple spaces into one, and

Foo          bar
Baz          ...

and sometimes spaces (or tabs) matter a lot. </> does not work on single lines.

[leeyu]

I apologize for the inconvenience. Here should be sufficient.

filter

filter{
  csv {
      autodetect_column_names => "true"
      autogenerate_column_names => "true"
      skip_header => "true"
      separator => ","
   }

  translate {
    field => "event_id"
    destination => "event_name"
    override => true
    dictionary_path => ["/data/mapping-table/map_v2.csv"]
   } 
}

map_v2.csv

1, Event1
2, Event2
3, Event3
4, Event4

target.csv

id, event_id
1, 4
2, 3
3, 2
4, 1

[Badger]

Great response! I don't think you have a field called [event_id]. I think you have a field called

[ event_id]

with an imbedded space. I do not have time to check right now, but it is possible that changing the option on your translate filter to

field => " event_id"

would make it work, and fixing up your CSV files would probably be a better approach.

[leeyu]

Haha, I also have no access to the server right now.
I will be able to verify it tomorrow.
Thank you very much!!!