Logstash access dynamic field with ruby filter

Elastic Stack Logstash
1

Hello

I have nested field looks like this

[field1][field2][field3]

I want to remove field1 and field2 so the expected output will be

[field3]

I already manage to remove field1 with this ruby filter

    ruby {
        code => '
            event.get("field1").each { |k, v|
                event.set(k,v)
            }
            event.remove("field1")
        '
    }

But I still cannot remove field2 because field2 is dynamic. How can I remove dynamic field2?

Thank you

2

Maybe I am not getting your question but why not just do something like.

mutate {
 rename => ["[field1][field2][field3]", "field3" ]
 remove_field => [ "field1" ]
}
3

Yes that will work for field1 but field2 is dynamic

How can I remove dynamic field?

4

How about

        event.get("field1").each { |k, v|
            v.each { |j, u|
                event.set(j,u)
            }
        }
5

Thanks It works

But some of my data looks like this

[field1][field2.1][field3] => "1"
[field1][field2.2][field3] => "2"

So in one document I have the same name for [field3]

With my current config the output will be like this
[field3] => "1"

value "2" will be replaced

So my expected output like
[field3] => ["1", "2"] become like array

or automatically renamed
[field3.1] => "1"
[field3.2] => "2"

Thank you

6

Have you actually tested that? If a field exists then I would expect event.set to change it to be an array.

If necessary you could do

    event.get("field1").each { |k, v|
        a = []
        key = nil
        v.each { |j, u|
            key = j
            a << u
        }
        event.set(key, a)
    }
7

Yes already tested, Here is the result

ruby filter

ruby {
         code => '
           event.get("field1").each { |k, v|
            v.each { |j, u|
                event.set(j,u)
            }
       }
        '
    }

Result field3 value replaced

{
  "_index": "email-2021.03.08",
  "_type": "_doc",
  "_id": "GMdTEHgBiOEFcJ6DutV8",
  "_version": 1,
  "_score": null,
  "_source": {
    "@version": "1",
    "field3": "2",
    "@timestamp": "2021-03-08T05:33:45.174Z",
    "host": "control-station7",
    "message": "{\"field1\":{\"field2A\":{\"field3\":\"1\"},\"field2B\":{\"field3\":\"2\"}}}",
    "path": "/home/elastic/logstash-7.9.3/email2.log",
    "field1": {
      "field2A": {
        "field3": "1"
      },
      "field2B": {
        "field3": "2"
      }
    }
  },
  "fields": {
    "@timestamp": [
      "2021-03-08T05:33:45.174Z"
    ]
  },
  "sort": [
    1615181625174
  ]
}
8

And here is the result for second ruby filter

ruby {
     code => '
       event.get("field1").each { |k, v|
       a = []
       key = nil
       v.each { |j, u|
          key = j
          a << u
       }
      event.set(key, a)
 }
      '
}

Result field3 value replaced

{
  "_index": "email-2021.03.08",
  "_type": "_doc",
  "_id": "QsdLEHgBiOEFcJ6DYdQN",
  "_version": 1,
  "_score": null,
  "_source": {
    "field3": [
      "2"
    ],
    "field1": {
      "field2A": {
        "field3": "1"
      },
      "field2B": {
        "field3": "2"
      }
    },
    "@timestamp": "2021-03-08T05:24:37.942Z",
    "host": "control-station7",
    "message": "{\"field1\":{\"field2A\":{\"field3\":\"1\"},\"field2B\":{\"field3\":\"2\"}}}",
    "@version": "1",
    "path": "/home/elastic/logstash-7.9.3/email2.log"
  },
  "fields": {
    "@timestamp": [
      "2021-03-08T05:24:37.942Z"
    ]
  },
  "sort": [
    1615181077942
  ]
}
9

Sorry, that should be

a = []
key = nil
event.get("field1").each { |k, v|
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.