Delete a field in filter, but use it in output

Elastic Stack Logstash
#1

Hi everybody,

I am using:
input {
jdbc {
}
}

filter {
elasticsearch {
remove_field => ["index_type"]
}
}

output {
elasticsearch{
index => "myindex"
document_type => "%{index_type}"
document_id => "%{customerno}"
}
}
I have a field in the input that has the name of the document_type, but I don't want that this field come in the document. If I use the filter to delete the field "index_type" I can not use it in the output to setting the document_type.

In resum I don't want the field in the domucument_source but I need it to configure the document_type

How I can do this?

Thanks a lot for your help!!

Regards.

Jorge von Rudno

2 Likes
#2

You can copy the field into event metadata prior to removing it, which will allow you to use it in outputs without having it in the document itself.

1 Like
#3

Thanks a lot Chirstian, I will do this!!!

Regards

Jorge

#4

Hi Christian,

Please what have I wrong in this configuration. I get an error:

#5

Sorry I send without complete:

Error: Expected one of #, => at line 18, column 9 (byte 531) after filter {
elasticsearch {
"[@metadata][index_type]"
{:level=>:error}

The config file is this:

filter {
elasticsearch {
"[@metadata][index_type]"
remove_field => ["index_type"]
}
}

output {
elasticsearch{
index => "myindex"

document_type => "%{index_type}"

document_type => "%{[@metadata][index_type]}" 
document_id => "%{customerno}"

}
}

#6

Why are you using the elasticsearch filter to modify fields? Even though it may be possible to use it as it inherits the base operations, the mutate filter is generally the natural choice for altering the structure of your event. First use add_field to copy the data to the metadata field and then remove the field you no longer want in your event. In order to guarantee that one step is done before the other I believe you will need to specify 2 mutate blocks. Something like this:

{
    mutate { add_field => { "[@metadata][index_type]" => "%{index_type}" } }
    mutate { remove_field => ["index_type"] }
}
2 Likes
#7

Hi Christian thanks a lot!! I will use this and tell you about the result!!

#8

Hi Christian, Today I have tested your suggestion and all work perfectly. Thanks a lot!!

#9