Copy a field without making it a hash value?

Allie_Yang · September 13, 2016, 4:33am

Hi I have an array field. And i want to copy it without turning it to a hash data type, which is what "add_field" does. Is there any ideas? Tks!

PS: the reason i want to do it is because i now have field search_terms = [0]good, [1] day; and i want to ALSO have a new field search_query = "good day". Both fields at same time.

magnusbaeck · September 13, 2016, 5:49am

Well, you could always use a ruby filter to make low-level manipulations of events. But I don't get what you mean by this:

And i want to copy it without turning it to a hash data type, which is what "add_field" does

Could you give an example?

Allie_Yang · September 13, 2016, 5:49pm

Sure...for example, i have an array field search_terms = [0]good, [1] day. If i add_field "temp" => "%{search_terms}", this field temp will not be able to join, since it is a hash data type. It can not be converted in to any other format, only the hash format as: good,day.

magnusbaeck · September 13, 2016, 6:04pm

Your use of the word "hash" is incorrect. There is no hash involved here. This shows why concrete examples with configuration snippets and copy/paste from terminal windows is so useful and reduces confusion.

If you want to join the search_terms array with spaces instead of commas without overwriting the original field you can use a ruby filter. Or, maybe using the mutate filter's gsub option to replace the commas in the joined string with spaces would work.

Allie_Yang · September 13, 2016, 6:36pm

Hi I want to have search_terms both in an array format and as a phrase. If i join search_terms, i would not have it as array format. If I copy search_term using add_field, the copied value is a hash data type, based on the online documentation, and become "good,day" only, not changeable to any other format...(https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html)

Below is my config FYI:
originally k = good++day
mutate {
gsub => [ "k", "[++]", " "]
split => { "k" => " " }
add_field => { "search_terms"=> "%{k}" }
join => { "search_terms" => " "} ## search_terms = good,day, as a hash data type, per the documentation
gsub => [ "search_terms", "[,]", " "] ## nothing will be done since gsub doesn't work on hash type
}

Allie_Yang · September 13, 2016, 6:39pm

As for Ruby, is it similar like below? Would you point out the syntax error pls...coz i never write ruby:

Ruby{
init => "search_terms" = "%{k}"
}

magnusbaeck · September 13, 2016, 7:51pm

The reason your mutate filter doesn't work as expected is that the options aren't applied in the order you specify them. The order is instead defined here:

github.com

logstash-plugins/logstash-filter-mutate/blob/v3.1.1/lib/logstash/filters/mutate.rb#L202-L217


      
          def filter(event)
            rename(event) if @rename
            update(event) if @update
            replace(event) if @replace
            convert(event) if @convert
            gsub(event) if @gsub
            uppercase(event) if @uppercase
            lowercase(event) if @lowercase
            strip(event) if @strip
            remove(event) if @remove
            split(event) if @split
            join(event) if @join
            merge(event) if @merge
          
            filter_matched(event)
          end

In other words, the gsub will happen first, then the join (and finally the add_field). You have to use multiple mutate filters.

If I copy search_term using add_field, the copied value is a hash data type, based on the online documentation

You are interpreting the documentation incorrectly.

As for Ruby, is it similar like below?

No. Something like

ruby {
  code => "
    event['search_terms'] = event['k'].join(' ')
  "
}

should work though.