How to make Ruby selectively remove the top level field from a specific nested field pattern

Elastic Stack Logstash
1

My Logstash instance will be receiving JSON records containing a wide variety of nested fields whose names start with [data][ecs].... and I need to trim the [data] prefix off of all of those field names before indexing. There will be other non-ecs [data] field names that must retain the [data] prefix.

For example, [data][ecs][source][ip] needs to change to [ecs][source][ip]
but [data][org][name] needs to stay the same.

From my research so far, it sounds like this would be a job for Ruby, and this example seems to hint at the soution:

but I'm not strong enough with Ruby to go the distance from there.

Would anyone be willing to at least get me off to a strong start on getting Ruby to enumerate through all field names in the current record that start with [data][ecs] and to trim off the [data] prefix from each of them?

I sure would appreciate it!
Kevin

2

After deeper digging, I found some better examples and I think I very nearly have the solution.

3

What I have so far is:

   #
   # Move all [data][ecs]... field names up to [ecs]...
   #
   ruby {
      code => "
         begin
            keys = event.to_hash.keys
            keys.each{|key|
                if ( key =~ /^\[data\]\[ecs\]/ )
                   newkey = key.gsub(/\\[data\\]\\[ecs\\]/, '[ecs]')
                   event.set(newkey, event.remove(key))
                end
            }
         rescue Exception => e
            event.set('logstash_ruby_exception', 'underscores: ' + e.message)
         end
      "
   }

At this point it is throwing no exception but no fields starting with [data][ecs] are being renamed to not start with [data].

Is there something wrong about how I am escaping the square brackets? I've tried several variations on that and it either throws an exception or simply renames no fields.

4

I sure would appreciate a hand on this. I believe the ability to dynamically adjust the position of certain nodes inside the field tree structure of records being processed by Logstash would be of broader value to the community than just my use case. I just wish I were more experienced with ruby. Please help if you can.

Kevin

5

I'm pretty sure that event.to_hash gives you a multidimensional data structure, so there would be a key "data", not "[data][ecs]". But I think you would only need ruby for this kind of action if your target is the root level of the event (many threads for this, e.g.: Move subarrays to document root). If there is a target field, simply renaming your field should work:

mutate {
    rename => {
        "[data][ecs]" => "ecs"
    }
}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.