How to move nested key value to root level and then rename elements root name?

Elastic Stack Logstash
1

How to move nested key value to root level and then rename elements root name

I have 4 core cpu and below is a sample of details for 2 cpu core from the source.

{
  "_source": {
    "cpu_usage": {
      "sys/host-info/2": {
        "sys/hostInfo/2/cpuInfo": {
          "sys/hostInfo/2/cpuInfo/1": {
            "user": 212142779,
            "system": 42193851
          },
          "sys/hostInfo/2/cpuInfo/0": {
            "user": 71408201,
            "system": 20886608
          }
        },
        "memoryUsed": 2745607080,
        "memoryTotal": 6712508416
      }
    }
  },
  "fields": {
    "cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo.sys/hostInfo/2/cpuInfo/1.system": [
      42193851
    ],
    "cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo.sys/hostInfo/2/cpuInfo/0.user": [
      71408201
    ],
    "cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo.sys/hostInfo/2/cpuInfo/0.system": [
      20886608
    ],
    "cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo.sys/hostInfo/2/cpuInfo/1.user": [
      212142779
    ]
  }
}

I would like to move contents under "cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo.sys/hostInfo/2/cpuInfo" to root and then rename "0." into "cpu0." and "1." into "cpu1"

I have tried below logstash filter and I was able move elements into "cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo" into root

filter {
  if [cpu_usage] {
   ruby {
      code => '
          event.get("[cpu_usage][sys/host-info/2]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[cpu_usage][sys/host-info/2]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo] {
   ruby {
      code => '
          event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[sys/hostInfo/2/cpuInfo]")
      '
    }
  }
}
2

In your first filter you probably want

event.remove("[cpu_usage]")

if [sys/host-info/2] is the only entry in it. In the second filter you can do something like

            event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
                newK = k.sub(/sys\/hostInfo\/\d\//, "" )
                event.set(newK,v)
            }

which will result in

  "cpuInfo/1" => {
    "system" => 42193851,
      "user" => 212142779
},

you can change the .sub call to make that look like anything you want.

3

sub is not working.

I tried below three scenario:

Scenario 1 with the third filter to rename the folder structure not including cpuInfo:

filter {
  if [cpu_usage] {
   ruby {
      code => '
          event.get("[cpu_usage][sys/host-info/2]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[cpu_usage][sys/host-info/2]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo] {
   ruby {
      code => '
          event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[sys/hostInfo/2/cpuInfo]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo] {
   ruby {
      code => '
	      event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
                newK = k.sub(/sys\/hostInfo\/\d\//, "" )
                event.set(newK,v)
            }
          event.remove("[sys/hostInfo/2]")
      '
    }
  }
}

elements are still under "sys/hostInfo/2/cpuInfo" folder. they didn't move to root folder "cpuInfo". i.e., the third filter did nothing.

Scenario 2 with the third filter to rename the folder structure which includes cpuInfo:

filter {
  if [cpu_usage] {
   ruby {
      code => '
          event.get("[cpu_usage][sys/host-info/2]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[cpu_usage][sys/host-info/2]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo] {
   ruby {
      code => '
          event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[sys/hostInfo/2/cpuInfo]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo] {
   ruby {
      code => '
          event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
          newK = k.sub(/sys\/hostInfo\/\d\/cpuInfo/, "cpu" )
          event.set(newK,v)
          }
          event.remove("[sys/hostInfo/2/cpuInfo]")
      '
    }
  }
}

elements are still under "sys/hostInfo/2/cpuInfo" folder. they didn't move to root folder "cpu". i.e., third filter did nothing.

Scenario 3 with the third filter to rename the folder structure which includes "cpuInfo/0" (This works but it doesn't create "cpu0" root folder but moved objects under "cpuInfo/0" to root (I cant use this logic for "cpu1" as it would overwrite cpu0 objects at the root)

filter {
  if [cpu_usage] {
   ruby {
      code => '
          event.get("[cpu_usage][sys/host-info/2]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[cpu_usage][sys/host-info/2]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo] {
   ruby {
      code => '
          event.get("[sys/hostInfo/2/cpuInfo]").each { |k,v|
          event.set(k,v)
          }
          event.remove("[sys/hostInfo/2/cpuInfo]")
      '
    }
  }

  if [sys/hostInfo/2/cpuInfo/0] {
   ruby {
      code => '
          event.get("[sys/hostInfo/2/cpuInfo/0]").each { |k,v|
          newK = k.sub(/sys\/hostInfo\/\d\/cpuInfo\/0/, "cpu0" )!
          event.set(newK,v)
          }
          event.remove("[sys/hostInfo/2/cpuInfo/0]")
      '
    }
  }
}

Below is a snippet of the elements in the output under root with above scenario 3:

{
  "_source": {
    "memoryUsed": 2850744048,
    "memoryTotal": 6712508416,2,
     "user": 73105642,
     "system": 21314119,
    "sys/hostInfo/2/cpuInfo/1": {
      "user": 220777464,
      "system": 44046558
    }   
  },
  "fields": {
    "memoryUsed": [
      2850744048
    ],
    "memoryTotal": [
      6712508416
    ],
    "user": [
      73105642
    ],
    "system": [
      21314119
    ],
    "sys/hostInfo/2/cpuInfo/1.user": [
      220777464
    ],
    "sys/hostInfo/2/cpuInfo/1.system": [
      44046558
    ]
  }
}

The above elements "user" and "system" under root are the elements of "cpuInfo/0"

4

I think in every case you are making execution of the filter conditional upon the existence of a field that does not exist.

5

Below is a sample path to a field "system" which has "sys/hostInfo/2/cpuInfo" repeated 3 times. That is why I used 3 filter for delete the path.

cpu_usage.sys/host-info/2.sys/hostInfo/2/cpuInfo.sys/hostInfo/2/cpuInfo/1.system
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.