How to remove part of name in multiple fields with Ruby in logstash?

[alytkowski]

I have a source of metrics, which are received by logstash and sent out to Elastic later. There are multiple fields starting with "prometheus.metrics.ems".
Can someone help with the ruby code or some other method to cut down this part of title from all of the fields? Is it possible?

[Wolfram_Haussig]

Hi,

How many fields/groups have to be renamed? The easiest would be to use the rename option of the mutate filter:

filter {
      mutate {
        # Renames the 'HOSTORIP' field to 'client_ip'
        rename => { "prometheus.metrics.ems.field1" => "field1" }
      }
    }

I think this also works with nested fields so you shouldn't need to rename each field manually.

Best regards
Wolfram

[alytkowski]

Hi!
I require to change more than 50 fields, that's why I was looking for a more automated method.

[Badger]

Maybe something like this (which I have not tested)

ruby {
    code => '
        event.to_hash.each { |k,v|
            if k =~ "^prometheus.metrics.ems"
                newK = k.sub(/^prometheus.metrics.ems/, "")
                event.remove(k)
                event.set(newK, v)
            end
        }
    '
}

[alytkowski]

@Badger

        event.to_hash.each { |k,v|
            if k =~ "^prometheus.metrics.ems"
                newK = k.sub(/^prometheus.metrics.ems/, "")
                event.remove(k)
                event.set(newK, v)
            end
        }

Unfortunately this gives out below error in the logs:
[ERROR][logstash.filters.ruby ][main] Ruby exception occurred: type mismatch: String given

[alytkowski]

After multiple trials with ruby, I've decided to go with the manual rename approach, for now at least. I've contructed the rename filter for all of the fields and.... it doesn't work. Logstash starts up normally, without errors, but the fields aren't renamed in Kibana. They're exactly the same as before. Could you advise on why is that?
Here's my config for this:
mutate {
add_field => { "EMS-HOST" => "test-host" }
rename => { "prometheus.labels.queue" => "queue" }
rename => { "prometheus.metrics.ems:queue:consumerCount" => "queue:consumerCount" }
}

I've added the "EMS-HOST" field just to test if logstash is using the mutate field. The field has been added as a result to all metrics incoming, but the renaming part is not done. There are much more renames in my config, I haven't included them not to spam here.

[Wolfram_Haussig]

Is the field named prometheus.labels.queueor is it a structure like this?

- prometheus
  - labels
    - queue

If it is a structure you need to change the dots to objects:

mutate {
add_field => { "EMS-HOST" => "test-host" }
rename => { "[prometheus][labels][queue]" => "queue" }
}

[alytkowski]

This worked. Thank you very much!

[alytkowski]

@alytkowski

prometheus.metrics.ems:queue:consumerCount

How to go about this one? I've tried multiple variations and none are working ;/
for example:
rename => { "[prometheus][metrics][ems]:[queue]:[consumerCount]" => "queue:consumerCount" }
rename => { "[prometheus][metrics][ems][queue][consumerCount]" => "queue:consumerCount" }
rename => { "[prometheus][metrics][ems]:queue:consumerCount" => "queue:consumerCount" }

[Wolfram_Haussig]

I think the colon is part of a normal name, right?

- prometheus
  - metric
    - ems:queue:consumerCount

In this case it would be:

mutate {
add_field => { "EMS-HOST" => "test-host" }
rename => { "[prometheus][metric][ems:queue:consumerCount]" => "queue:consumerCount" }
}