Unnest JSON

Elastic Stack Logstash
1

Good Day,
I am trying to unnest a JSON log and I can't seem to get it to work. When I try to unnest I get the already unnested JSON showing up 3 times and I while I am able to unnest the JSON I can't write the actual field value to the newly unnested JSON.
So if I unnest and try to write the existing value of the SpanId field it just displays the actual logstash code

Log

{
  "lvl": "info",
  "eid": 1,
  "en": "RequestStarting",
  "tim": "2023-10-05T10:07:19.157174Z",
  "msg": "Request starting HTTP/2 GET https://FQDN/space/lodwe/independentFile/1lki000000000000k7002c000000  - -",
  "cat": "Microsoft.AspNetCore.Hosting.Diagnostics",
  "scp": [
    {
      "SpanId": "569b3b4281694978",
      "TraceId": "984d20903a7ad189ab4e7d25e9e60a9b",
      "ParentId": "b7pa9b27b48e0726"
    },
    {
      "ConnectionId": "0HMKB5NAOAQU1"
    },
    {
      "RequestId": "0HMKP9NAOAQU7:000008OU",
      "RequestPath": "/space/ lodwe  /independentFile/1bca000000000000c7002c000000"
    }
  ],
  "ste": [
    {
      "Protocol": "HTTP/2",
      "Method": "GET",
      "ContentType": null,
      "ContentLength": null,
      "Scheme": "https",
      "Host": "FQDN",
      "PathBase": "",
      "Path": "/space/lodwe/independentFile/1lki000000000000k506001d000000",
      "QueryString": ""
    }
  ]
}

and I use the following

input {
   tcp {
   port => 8090
   tags => ["test"] 
   codec => json { }
  }
}

filter
{

split { field => "[scp]" }

mutate {
  add_field => {
   "SpanId" => "%{[scp][SpanId]"
}
    remove_field => [ "[scp]" ]
}

split { field => "[ste]" }

mutate {
    remove_field => [ "[ste]" ]
}


date {
        locale => "$LANG"
        match => [ "tim", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ" ]
        target => "@timestamp"
        remove_field => ["msg","tim","bep","@version","RequestPath","tags"]
      }	    

}

output {
#  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }

}

I get the following output

{
          "port" => 50574,
            "en" => "RequestStarting",
          "host" => "localhost",
    "@timestamp" => 2023-10-05T10:07:19.157Z,
           "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
        "SpanId" => "%[scp][SpanId]",
           "eid" => 1,
           "lvl" => "info"
}
{
          "port" => 50574,
            "en" => "RequestStarting",
          "host" => "localhost",
    "@timestamp" => 2023-10-05T10:07:19.157Z,
           "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
        "SpanId" => "%[scp][SpanId]",
           "eid" => 1,
           "lvl" => "info"
}
{
          "port" => 50574,
            "en" => "RequestStarting",
          "host" => "localhost",
    "@timestamp" => 2023-10-05T10:07:19.157Z,
           "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
        "SpanId" => "%[scp][SpanId]",
           "eid" => 1,
           "lvl" => "info"
}
2

You are missing a closing }. I would rewrite that as

    if [scp][SpanId] {
        mutate { add_field => { "SpanId" => "%{[scp][SpanId]}" } }
    }
    mutate { remove_field => [ "[scp]" ] }

I also see no reason to split [ste] since you immediately delete the field.

3

I tried the above code and it doesnt seem to match the [scp] field and then removed [scp] entirely.....as expected. If I comment out the below "remove_field" and run again I am seeing [0] but not sure if that needs to be accounted for.

mutate { remove_field => [ "[scp]" ] }


{
           "eid" => 1,
           "scp" => [
        [0] {
            "ParentId" => "b7pa9b27b48e0726",
              "SpanId" => "569b3b4281694978",
             "TraceId" => "984d20903a7ad189ab4e7d25e9e60a9b"
        },
        [1] {
            "ConnectionId" => "0HMKB5NAOAQU1"
        },
        [2] {
              "RequestId" => "0HMKP9NAOAQU7:000008OU",
            "RequestPath" => "/space/ lorte  /independentFile/1bca000000000000c7002c000000"
        }
    ],
    "@timestamp" => 2023-10-05T10:07:19.157Z,
            "en" => "RequestStarting",
           "lvl" => "info",
          "port" => 39920,
           "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
           "ste" => [
        [0] {
                     "Path" => "/space/lorte/independentFile/1lki000000000000k7002c000000",
                 "PathBase" => "",
                     "Host" => "FQDN",
            "ContentLength" => nil,
                   "Method" => "GET",
              "ContentType" => nil,
                   "Scheme" => "https",
                 "Protocol" => "HTTP/2",
              "QueryString" => ""
        }
    ],
          "host" => "localhost"
}
4

You still need the split { field => "scp" }.

5

Thanks. Its mostly working except that for some reason its not unnesting a small section of the log and for some reason the result is duplicated 3 times.
The below is still not being unnested and I believe its bracket placement?

{
             "scp" => {
        "ConnectionId" => "0HMKB5NAOAQU1"
    },
{
           "scp" => {
        "RequestPath" => "/space/ lorte  /independentFile/1bca000000000000c7002c000000",
          "RequestId" => "0HMKP9NAOAQU7:000008OU"
    },

The full result is below and duplicated 3 times

{
            "port" => 52352,
             "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
          "Scheme" => "https",
    "ConnectionId" => "0HMKB5NAOAQU1",
      "@timestamp" => 2023-10-05T10:07:19.157Z,
             "lvl" => "info",
        "Protocol" => "HTTP/2",
             "eid" => 1,
          "Method" => "GET",
            "Host" => "FQDN",
             "scp" => {
        "ConnectionId" => "0HMKB5NAOAQU1"
    },
            "host" => "localhost",
            "Path" => "/space/lorte/independentFile/1lki000000000000k7002c000000",
              "en" => "RequestStarting"
}
{
          "port" => 52352,
           "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
       "TraceId" => "984d20903a7ad189ab4e7d25e9e60a9b",
        "Scheme" => "https",
    "@timestamp" => 2023-10-05T10:07:19.157Z,
           "lvl" => "info",
      "Protocol" => "HTTP/2",
           "eid" => 1,
      "ParentId" => "b7pa9b27b48e0726",
        "Method" => "GET",
          "Host" => "FQDN",
           "scp" => {
         "TraceId" => "984d20903a7ad189ab4e7d25e9e60a9b",
        "ParentId" => "b7pa9b27b48e0726",
          "SpanId" => "569b3b4281694978"
    },
          "host" => "localhost",
          "Path" => "/space/lorte/independentFile/1lki000000000000k7002c000000",
            "en" => "RequestStarting",
        "SpanId" => "569b3b4281694978"
}
{
          "port" => 52352,
           "cat" => "Microsoft.AspNetCore.Hosting.Diagnostics",
        "Scheme" => "https",
    "@timestamp" => 2023-10-05T10:07:19.157Z,
           "lvl" => "info",
      "Protocol" => "HTTP/2",
           "eid" => 1,
        "Method" => "GET",
          "Host" => "FQDN",
           "scp" => {
        "RequestPath" => "/space/ lorte  /independentFile/1bca000000000000c7002c000000",
          "RequestId" => "0HMKP9NAOAQU7:000008OU"
    },
          "host" => "localhost",
          "Path" => "/space/lorte/independentFile/1lki000000000000k7002c000000",
            "en" => "RequestStarting"
}
6

The split filter will transform one event with an array of three hashes into three events, so that seems to be working as expected. If you want to move anything that is in [scp] to the top level then see this thread.

7

Wow thanks so much this worked!!! Have been going at this for a while now.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.