I would like to take advantage of the "related.ip" field for pivoting around different sources. I have everything running through Logstash, and I am attempting to append all IP fields to related.ip. Unfortunately, I am only getting one IP on each event. I have tried merge, copy, and add_field. Per the documentation, "copy" will overwrite the target field, so that explains the issue with that attempt. Merge and add_field should both convert related.ip to an array, should they not?
Here is my attempt with add_field. Again, this is unfortunately resulting in only the source.ip field being added to related.ip; assumedly because it is the last addition.
This way I'm able to search in kibana just over one entry of array, that I'm inserting into related.ip, ex: related.ip:2.3.4.5
Important thing to remember: if something won't be working and you will find your index is not updated with new documents, take a look into logstash logs, if you will find something similiar to:
Aug 2 15:23:48 elk1 logstash: [2019-08-02T15:23:48,581][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"dynamic-auditd", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x19609875>], :response=>{"index"=>{"_index"=>"dynamic-auditd", "_type"=>"_doc", "_id"=>"EYzuUmwBG7O9jprBDYaH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [related.ip] tried to parse field [related.ip] as object, but found a concrete value"}}}}
stop logstash, remove index pattern, remove index, and start logstash again, and create index pattern again.
Cheers
Thank you for the reply! I actually just tried again, and it is working now with just a mutate filter. Surprised I did not get an email alert that you replied, but I just came here to post that I resolved the issue.
Here is what is my working conf:
6002_postprocess_related_ip.conf
filter {
if [client][ip] {mutate {add_field => {"[related][ip]" => "%{[client][ip]}"}}}
if [destination][ip] {mutate {add_field => {"[related][ip]" => "%{[destination][ip]}"}}}
if [host][ip] {mutate {add_field => {"[related][ip]" => "%{[host][ip]}"}}}
if [observer][ip] {mutate {add_field => {"[related][ip]" => "%{[observer][ip]}"}}}
if [network][ip] {mutate {add_field => {"[related][ip]" => "%{[network][ip]}"}}}
if [server][ip] {mutate {add_field => {"[related][ip]" => "%{[server][ip]}"}}}
if [source][ip] {mutate {add_field => {"[related][ip]" => "%{[source][ip]}"}}}
}
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.