Append IP Addresses to "related.ip" ECS Field

I would like to take advantage of the "related.ip" field for pivoting around different sources. I have everything running through Logstash, and I am attempting to append all IP fields to related.ip. Unfortunately, I am only getting one IP on each event. I have tried merge, copy, and add_field. Per the documentation, "copy" will overwrite the target field, so that explains the issue with that attempt. Merge and add_field should both convert related.ip to an array, should they not?

Here is my attempt with add_field. Again, this is unfortunately resulting in only the source.ip field being added to related.ip; assumedly because it is the last addition.

filter {
  mutate {
    add_field => {"[related][ip]" => "%{[client][ip]}"}
    add_field => {"[related][ip]" => "%{[destination][ip]}"}
    add_field => {"[related][ip]" => "%{[host][ip]}"}
    add_field => {"[related][ip]" => "%{[observer][ip]}"}
    add_field => {"[related][ip]" => "%{[network][forwarded_ip]}"}
    add_field => {"[related][ip]" => "%{[server][ip]}"}
    add_field => {"[related][ip]" => "%{[source][ip]}"}
  }
}

I would appreciate any thoughts on how I should append all these IP addresses to related.ip.

Thank you in advance!

Hi Eric,

I found your qestion as I came today to the same problem.
I found a way to resolve it with following ruby code:

ruby {
    code => 'event.set("[related][ip]", [(event.get("[host][ip]")), (event.get("[source][ip]")), (event.get("[destination][ip]"))])'
}

This way I'm able to search in kibana just over one entry of array, that I'm inserting into related.ip, ex: related.ip:2.3.4.5

Important thing to remember: if something won't be working and you will find your index is not updated with new documents, take a look into logstash logs, if you will find something similiar to:

Aug  2 15:23:48 elk1 logstash: [2019-08-02T15:23:48,581][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dynamic-auditd", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x19609875>], :response=>{"index"=>{"_index"=>"dynamic-auditd", "_type"=>"_doc", "_id"=>"EYzuUmwBG7O9jprBDYaH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [related.ip] tried to parse field [related.ip] as object, but found a concrete value"}}}}

stop logstash, remove index pattern, remove index, and start logstash again, and create index pattern again.
Cheers :slight_smile:

@kisamot

Thank you for the reply! I actually just tried again, and it is working now with just a mutate filter. Surprised I did not get an email alert that you replied, but I just came here to post that I resolved the issue.

Here is what is my working conf:

6002_postprocess_related_ip.conf

filter {
  if [client][ip]       {mutate {add_field => {"[related][ip]" => "%{[client][ip]}"}}}
  if [destination][ip]  {mutate {add_field => {"[related][ip]" => "%{[destination][ip]}"}}}
  if [host][ip]         {mutate {add_field => {"[related][ip]" => "%{[host][ip]}"}}}
  if [observer][ip]     {mutate {add_field => {"[related][ip]" => "%{[observer][ip]}"}}}
  if [network][ip]      {mutate {add_field => {"[related][ip]" => "%{[network][ip]}"}}}
  if [server][ip]       {mutate {add_field => {"[related][ip]" => "%{[server][ip]}"}}}
  if [source][ip]       {mutate {add_field => {"[related][ip]" => "%{[source][ip]}"}}}
}
1 Like