Unable to remove field form nested object during mutate

Elastic Stack Logstash
1

Hi:
I am using logstash 1.5.3 and trying to reindex an existing index to a new index with modified mapping. To test if this works on nested fields, I gave the following config a shot :

input {
   elasticsearch {
     hosts => ["localhost"]
     port => "9200"
     index => "index1"
     scroll => "5m"
     docinfo => true
   }
}

filter {
  mutate {
    remove_field => ["@timestamp","@version","[entities][kb]"]
  }
}

output {
   elasticsearch {
     host => "localhost"
     port => 9300
     protocol => "transport"
     cluster => "CTRCT-AB8"
     index => "index_mod"
     document_type => "entities2"
     document_id => "%{[@metadata][_id]}"
   }

Where entities is a nested field and kb is a nested field within. However, this approach is not removing the field.
What ends up happening is the new mapping is created where the nested field is no longer a nested field (not a problem) and the field still exists (big problem). I am able to get rid of the top level nested field ("entities") but not any sub fields or sub nested fields of "entities"

I think the syntax for accessing/mutating/modifying nested fields is right (various posts have alluded to documentation, but I cannot find documentation regarding how to specify nested fields). Any help in this regards is very much appreciated.

Thanks much

Ramdev

2

I'm not able to reproduce what you're describing, or I'm not understanding what the problem is.

$ cat test.config
input { stdin { codec => json } }
output { stdout { codec => rubydebug } }
filter {
  mutate {
    remove_field => ["@timestamp","@version","[entities][kb]"]
  }
}
$ echo '{"message": "test", "entities": {"kb": "foo"} }' | /opt/logstash/bin/logstash -f test.config
Logstash startup completed
{
     "message" => "test",
    "entities" => {},
        "host" => "hallonet"
}
Logstash shutdown completed
3

Hi Magnus:
Thanks for the response. What I have found is in my index, I have a nested field which in turn has a nested field. I wanted to promote the inner nested field to the top level (so as to flatten the structure out). So Iwas using mutate to remove the inner nested field and add it back as a top level field.

However, I was unable to do so on the field as it was defined. When I created the following config:

input {
   elasticsearch {
     hosts => ["localhost"]
     port => "9200"
     index => "index"
     scroll => "5m"
     docinfo => true
   }
}

filter {
  mutate {
    rename => {"[entities]" => "[mentions_toplevel]"}
    add_field => {
        "[extracted_mentions][stop]" => "[mentions_toplevel][mentions][stop]"
        "[extracted_mentions][start]" => "[mentions_toplevel][mentions][start]"
        "[extracted_mentions][tag]" => "[mentions_toplevel][mentions][tag]"
        "[extracted_mentions][position_category]" => "[mentions_toplevel][mentions][position_category]"
    }
    remove_field => ["@timestamp","@version","[mentions_toplevel][mentions]"]
  }
}

output {
   elasticsearch {
     host => "localhost"
     port => 9300
     protocol => "transport"
     cluster => "cluster"
     index => "index_mod"
     document_type => "entities2"
     document_id => "%{[@metadata][_id]}"
   }
   stdout {
     codec => "dots"
   }
}

I was able to get the desired effect. For some reason I am unable to remove the inner nested field from the original field definition.

Thanks

Ramdev

4

Hi,
I think that your case is related to:

Maybe you would like to upgrade to 1.5.5 and give it another try.

Regards