Nested JSON use field as value

[zacesa]

Hi,

I have a json input as per below;

"stats": {
"src": {
  "port": {
    "6666": 1
  },
  "rep": 1,
  "country": {
    "US": 1
  },
  "ip": {
    "1.1.1.1": {
      "rep": 1,
      "uuid": "NULL",
      "country": "US",
      "count": 1
    }
  }
},
"dst": {
  "port": {
    "443": 1
  },
  "rep": 0,
  "country": {
    "US": 1
  },
  "ip": {
    "2.2.2.2": {
      "rep": 0,
      "uuid": "0x29bfa15ae6da11e7a1ac000cfb542710",
      "country": "US",
      "count": 1
    }
  }
},
"events": 1
  },

I'd like to do a geoip of the [src][ip] however, the value of [src][ip] is in fact, "1.1.1.1": { "rep": 1, "uuid": "NULL", "country": "US", "count": 1 }.

I tried to copy [src][ip] to a new field called src_ip but, it copies the whole thing. Is there any way to remove the other fields under src_ip?

I've already tried,

 mutate {
        remove_field => [ "[src_ip][rep]", "[src_ip][uuid]", "[src_ip][count]", "[src_ip][country]" ]
    }

but it didn't work.

[guyboertje]

Unfortunately, we do not have a mechanism to refer to an anonymous key in the nested object. That is, we don't have [src][ip^K:1] as in "grab the first key name of the nested object in [src][ip]. Obviously [src][ip][1.1.1.1] is a reference to the nested object {"rep": 1, ... "count": 1}

You could try to transform the JSON before its pushed into LS. https://github.com/ColinEberhardt/json-transforms is a possible OS project.

[zacesa]

I've gone with another method to get the data into a field.

I used;

grok {
match => { "stats" => "\"src\"\:\{.*?\"ip\"\:\{.*?\"%{IP:src_ip}\"" }
}

However, there may be multiple source IP so, how do I use the same grok pattern, to get multiple matches on the same line?

[guyboertje]

Nice idea. This done before the JSON filter, I presume?

You can have multiple patterns if the match value is an array and break_on_match is false, the src_ip will become an array of the IPs extracted as a root key and value. Later you can add that array back into the nested hash/object. P.S. you can use single quote to avoid the escaping of double quotes.

I could only extract 3 IP addresses though (first, second & last)

input {
  generator {
    message => '{
  "src": {
    "port": {
      "6666": 1
    },
    "rep": 1,
    "country": {
      "US": 1
    },
    "ip": {
      "1.1.1.1": {
        "rep": 1,
        "uuid": "NULL",
        "country": "US",
        "count": 1
      },
      "2.1.1.1": {
        "rep": 1,
        "uuid": "NULL",
        "country": "US",
        "count": 1
      },
      "3.1.1.1": {
        "rep": 1,
        "uuid": "NULL",
        "country": "US",
        "count": 1
      }
    }
  }
}'
    count => 1
  }
}

filter {
  grok {
    match => { "message" => ['{[^"]+"%{IP:src_ip}"', '},[^"]+"%{IP:src_ip}":\s{[^}]+},\n', '},[^"]+"%{IP:src_ip}":\s{[^}]+}\n'] }
    break_on_match => false
  }
  json {
    source => "message"
    remove_field => ["message"]
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Results:

{
    "@timestamp" => 2018-01-25T11:25:19.920Z,
           "src" => {
           "port" => {
            "6666" => 1
        },
            "rep" => 1,
             "ip" => {
            "3.1.1.1" => {
                  "count" => 1,
                   "uuid" => "NULL",
                    "rep" => 1,
                "country" => "US"
            },
            "1.1.1.1" => {
                  "count" => 1,
                   "uuid" => "NULL",
                    "rep" => 1,
                "country" => "US"
            },
            "2.1.1.1" => {
                  "count" => 1,
                   "uuid" => "NULL",
                    "rep" => 1,
                "country" => "US"
            }
        },
        "country" => {
            "US" => 1
        }
    },
          "host" => "Elastics-MacBook-Pro.local",
      "@version" => "1",
      "sequence" => 0,
        "src_ip" => "1.1.1.1"
}