Add field from csv to event log send by logstash

[magnusbaeck (Magnus Bäck)]

Your configuration looks fine and when I try it out myself with Logstash 2.4.0 it does what it's supposed to do:

$ cat table.yml 
---
C583: jane Doe
C090: John Doe
C587: Michael Jackson
$ cat test.config 
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
filter {
  translate {
    dictionary_path => "/tmp/trash.Nuac/table.yml"
    field => "[event_data][TargetUserName]"
    destination => "[username]"
    remove_field => [ "@version", "@timestamp" ]
    override => true
  }
}
$ echo '{"event_data": {"TargetUserName": "C587"}}' | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
    "event_data" => {
        "TargetUserName" => "C587"
    },
          "host" => "lnxolofon",
      "username" => "Michael Jackson"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

[Grenouille06]

Thank you Magnus it works, I can see the new field in my console with "Name FirstName" In it.

Last question please. In kibana I Can't see the values in my new field :

This field is present in your elasticsearch mapping but not in any documents in the search results. You may still be able to visualize or search on it.

I reloaded the index pattern and my new field is searchable and analyzed. New field and values appears in terminal but won't store in ES.

This is my conf file output

filter{
translate {
dictionary_path => "/etc/logstash/mutate/ExportADLDS.yml"
field => "[event_data][TargetUserName]"
destination => "[DisplayName]"
remove_field => [ "@version", "@timestamp" ]
override => true
}
}
output {
elasticsearch {
hosts => "192.168.18.15:9200"
manage_template => false
enable_metric => false
action => "update"
document_id => "[DisplayName]"
index => "logstash-security-2017.03.14"
template => "/etc/logstash/conf.d/mapping/security.json"
}
stdout {codec => rubydebug}
}

Thank you

[Grenouille06]

Hi all. Something new :

When I launch the conf file above, I can see in the terminal, like I said before the new field "DisplayName" : "Content_of_my_yaml_file". But nothing is add in my event strored in ES.

In the log files I saw that line at the beginning of the process : it seems that ES won't see the content of my yaml file as a String but biValues string.

Error Code: 0xC000006A], :response=>{"update"=>{"_index"=>"logstash-security-2017.03.15-08:11", "_type"=>"wineventlog", "_id"=>"[NCADisplayName]", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[wineventlog][[NCADisplayName]]: document missing", "index_uuid"=>"W197WWOKS6GGEYGNpACAPg", "shard"=>"2", "index"=>"logstash-security-2017.03.15-08:11"}}}}
2017-03-16 09:19:15,691 [main]>worker2 ERROR An exception occurred processing Appender plain_console java.lang.ClassCastException: org.logstash.bivalues.StringBiValue cannot be cast to java.lang.String
at org.logstash.Event.toString(Event.java:315)
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_to_s(JrubyEventExtLibrary.java:209)
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$0$0$ruby_to_s.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$0$0$ruby_to_s.gen)
at org.jruby.RubyClass.finvoke(RubyClass.java:624)
at org.jruby.runtime.Helpers.invoke(Helpers.java:502)
at org.jruby.RubyBasicObject.inspect(RubyBasicObject.java:1042)
at org.jruby.RubyKernel.inspect(RubyKernel.java:2079)
at org.jruby.RubyKernel$INVOKER$s$0$0$inspect.call(RubyKernel$INVOKER$s$0$0$inspect.gen)
at org.jruby.RubyClass.finvoke(RubyClass.java:624)
at org.jruby.runtime.Helpers.invoke(Helpers.java:502)
at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:356)
at org.jruby.RubyObject.inspect(RubyObject.java:533)
at org.jruby.RubyArray.inspectAry(RubyArray.java:1480)
at org.jruby.RubyArray.inspect(RubyArray.java:1510)
at org.jruby.RubyArray$INVOKER$i$0$0$inspect.call(RubyArray$INVOKER$i$0$0$inspect.gen)
at org.jruby.RubyClass.finvoke(RubyClass.java:624)
at org.jruby.runtime.Helpers.invoke(Helpers.java:502)
at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:356)
at org.jruby.RubyObject.inspect(RubyObject.java:533)
at org.jruby.RubyHash$5.visit(RubyHash.java:816)
at org.jruby.RubyHash.visitLimited(RubyHash.java:648)
at org.jruby.RubyHash.visitAll(RubyHash.java:634)
at org.jruby.RubyHash.inspectHash19(RubyHash.java:811)
at org.jruby.RubyHash.inspect19(RubyHash.java:848)
at org.jruby.RubyHash.to_s19(RubyHash.java:910)
at org.jruby.RubyHash$INVOKER$i$0$0$to_s19.call(RubyHash$INVOKER$i$0$0$to_s19.gen)
at org.jruby.RubyClass.finvoke(RubyClass.java:624)
at org.jruby.runtime.Helpers.invoke(Helpers.java:502)
at org.jruby.RubyObject.toString(RubyObject.java:331)
at java.lang.String.valueOf(String.java:2994)
at java.lang.StringBuilder.append(StringBuilder.java:131)
at org.logstash.log.StructuredMessage.getFormattedMessage(StructuredMessage.java:61)
at org.apache.logging.log4j.core.pattern.MessagePatternConverter.format(MessagePatternConverter.java:84)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)

[Grenouille06]

I don't know why but now everything works fine.

Thanks Magnus for advices and example conf