# Add field from csv to event log send by logstash

**URL:** <https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677>\
**Category:** Logstash\
**Created:** [March 7, 2017, 2:13pm UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677 "2017-03-07T14:13:44Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 15, 2017, 6:25am UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677/21 "2017-03-15T06:25:14Z")

</div>

Your configuration looks fine and when I try it out myself with Logstash 2.4.0 it does what it's supposed to do:

```nohighlight
$ cat table.yml 
---
C583: jane Doe
C090: John Doe
C587: Michael Jackson
$ cat test.config 
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
filter {
  translate {
    dictionary_path => "/tmp/trash.Nuac/table.yml"
    field => "[event_data][TargetUserName]"
    destination => "[username]"
    remove_field => ["@version", "@timestamp"]
    override => true
  }
}
$ echo '{"event_data": {"TargetUserName": "C587"}}' | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
    "event_data" => {
        "TargetUserName" => "C587"
    },
          "host" => "lnxolofon",
      "username" => "Michael Jackson"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

```

---

<div class="post-metadata">

**Author:** ![Grenouille06](https://avatars.discourse-cdn.com/v4/letter/g/48db29/32.png) [@Grenouille06](https://discuss.elastic.co/u/Grenouille06)\
**Post date:** [March 15, 2017, 10:51am UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677/22 "2017-03-15T10:51:56Z")

</div>

Thank you Magnus it works, I can see the new field in my console with "Name FirstName" In it.

Last question please. In kibana I Can't see the values in my new field :

> This field is present in your elasticsearch mapping but not in any documents in the search results. You may still be able to visualize or search on it.

I reloaded the index pattern and my new field is searchable and analyzed. New field and values appears in terminal but won't store in ES.

This is my conf file output

> filter{  
> translate {  
> dictionary\_path =\> "/etc/logstash/mutate/ExportADLDS.yml"  
> field =\> "[event\_data][TargetUserName]"  
> destination =\> "[DisplayName]"  
> remove\_field =\> ["@version", "@timestamp"]  
> override =\> true  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> "192.168.18.15:9200"  
> manage\_template =\> false  
> enable\_metric =\> false  
> action =\> "update"  
> document\_id =\> "[DisplayName]"  
> index =\> "logstash-security-2017.03.14"  
> template =\> "/etc/logstash/conf.d/mapping/security.json"  
> }  
> stdout {codec =\> rubydebug}  
> }

Thank you

---

<div class="post-metadata">

**Author:** ![Grenouille06](https://avatars.discourse-cdn.com/v4/letter/g/48db29/32.png) [@Grenouille06](https://discuss.elastic.co/u/Grenouille06)\
**Post date:** [March 16, 2017, 8:27am UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677/23 "2017-03-16T08:27:17Z")

</div>

Hi all. Something new :

When I launch the conf file above, I can see in the terminal, like I said before the new field "DisplayName" : "Content\_of\_my\_yaml\_file". But nothing is add in my event strored in ES.

In the log files I saw that line at the beginning of the process : it seems that ES won't see the content of my yaml file as a String but biValues string.

> Error Code: 0xC000006A], :response=\>{"update"=\>{"\_index"=\>"logstash-security-2017.03.15-08:11", "\_type"=\>"wineventlog", "\_id"=\>"[NCADisplayName]", "status"=\>404, "error"=\>{"type"=\>"document\_missing\_exception", "reason"=\>"[wineventlog][[NCADisplayName]]: document missing", "index\_uuid"=\>"W197WWOKS6GGEYGNpACAPg", "shard"=\>"2", "index"=\>"logstash-security-2017.03.15-08:11"}}}}  
> 2017-03-16 09:19:15,691 [main]\>worker2 _ **ERROR An exception occurred processing Appender plain\_console java.lang.ClassCastException: org.logstash.bivalues.StringBiValue cannot be cast to java.lang.String** _  
> at org.logstash.Event.toString(Event.java:315)  
> at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby\_to\_s(JrubyEventExtLibrary.java:209)  
> at org.logstash.ext.JrubyEventExtLibrary$RubyEvent$INVOKER$i$0$0$ruby\_to\_s.call(JrubyEventExtLibrary$RubyEvent$INVOKER$i$0$0$ruby\_to\_s.gen)  
> at org.jruby.RubyClass.finvoke(RubyClass.java:624)  
> at org.jruby.runtime.Helpers.invoke(Helpers.java:502)  
> at org.jruby.RubyBasicObject.inspect(RubyBasicObject.java:1042)  
> at org.jruby.RubyKernel.inspect(RubyKernel.java:2079)  
> at org.jruby.RubyKernel$INVOKER$s$0$0$inspect.call(RubyKernel$INVOKER$s$0$0$inspect.gen)  
> at org.jruby.RubyClass.finvoke(RubyClass.java:624)  
> at org.jruby.runtime.Helpers.invoke(Helpers.java:502)  
> at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:356)  
> at org.jruby.RubyObject.inspect(RubyObject.java:533)  
> at org.jruby.RubyArray.inspectAry(RubyArray.java:1480)  
> at org.jruby.RubyArray.inspect(RubyArray.java:1510)  
> at org.jruby.RubyArray$INVOKER$i$0$0$inspect.call(RubyArray$INVOKER$i$0$0$inspect.gen)  
> at org.jruby.RubyClass.finvoke(RubyClass.java:624)  
> at org.jruby.runtime.Helpers.invoke(Helpers.java:502)  
> at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:356)  
> at org.jruby.RubyObject.inspect(RubyObject.java:533)  
> at org.jruby.RubyHash$5.visit(RubyHash.java:816)  
> at org.jruby.RubyHash.visitLimited(RubyHash.java:648)  
> at org.jruby.RubyHash.visitAll(RubyHash.java:634)  
> at org.jruby.RubyHash.inspectHash19(RubyHash.java:811)  
> at org.jruby.RubyHash.inspect19(RubyHash.java:848)  
> at org.jruby.RubyHash.to\_s19(RubyHash.java:910)  
> at org.jruby.RubyHash$INVOKER$i$0$0$to\_s19.call(RubyHash$INVOKER$i$0$0$to\_s19.gen)  
> at org.jruby.RubyClass.finvoke(RubyClass.java:624)  
> at org.jruby.runtime.Helpers.invoke(Helpers.java:502)  
> at org.jruby.RubyObject.toString(RubyObject.java:331)  
> at java.lang.String.valueOf(String.java:2994)  
> at java.lang.StringBuilder.append(StringBuilder.java:131)  
> at org.logstash.log.StructuredMessage.getFormattedMessage(StructuredMessage.java:61)  
> at org.apache.logging.log4j.core.pattern.MessagePatternConverter.format(MessagePatternConverter.java:84)  
> at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)

---

<div class="post-metadata">

**Author:** ![Grenouille06](https://avatars.discourse-cdn.com/v4/letter/g/48db29/32.png) [@Grenouille06](https://discuss.elastic.co/u/Grenouille06)\
**Post date:** [March 16, 2017, 10:01am UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677/24 "2017-03-16T10:01:34Z")

</div>

I don't know why but now everything works fine.

Thanks Magnus for advices and example conf

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2017, 10:01am UTC](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677/25 "2017-04-13T10:01:43Z")

</div>

Closed on April 13, 2017, 10:01am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/add-field-from-csv-to-event-log-send-by-logstash/77677.md?page=1)