Array of nested json objects

[rajsolanki (rajesh)]

Hope this is not a repeat.

I have a document like following in json format.

{
  "request_body": "",
  "bytes_sent": 16,
  "response_body": "",
  "response_http_headers": [
      {
        "Content-Type": "application/json"
      },
      {
        "Date": "Thu, 05 Mar 2020 19:17:35 GMT"
      },
      
      {
        "Content-Length": "459"
      },
      {
        "Via": "1.1 AgAAANE7P4c-"
      },
      {
        "hostname": "myhost.com"
      },
      {
        "msgid": "414d51205343535131202020202020205e56fb652147c3b6"
      },
      {
        "Access-Control-Allow-Origin": "*"
      },
      {
        "Access-Control-Allow-Methods": "POST"
      },
      {
        "X-RateLimit-Limit": "name=default,10000;"
      },
      {
        "X-RateLimit-Remaining": "name=default,9996;"
      }
    ],
    "@timestamp": "2020-03-05T19:17:33.962Z"
   
}

so in my logstash config i have

if "myhost.com" in [response_http_headers]

{
mutate{
add_field => { "Raj" => "found myhost" }
}
}

but it does not detect

any clue on what am i doing wrong ? or what i m trying thats not possible ?

Raj

[rajsolanki (rajesh)]

here is output of logstash stdout codec json

},
                  "message" => "{ \"request_body\": \"\", \"bytes_sent\": 16, \"response_body\": \"\", \"response_http_headers\": [ { \"Content-Type\": \"application/json\" }, { \"Date\": \"Thu, 05 Mar 2020 19:17:35 GMT\" }, { \"Content-Length\": \"459\" }, { \"Via\": \"1.1 AgAAANE7P4c-\" }, { \"hostname\": \"myhost.com\" }, { \"msgid\": \"414d51205343535131202020202020205e56fb652147c3b6\" }, { \"Access-Control-Allow-Origin\": \"*\" }, { \"Access-Control-Allow-Methods\": \"POST\" }, { \"X-RateLimit-Limit\": \"name=default,10000;\" }, { \"X-RateLimit-Remaining\": \"name=default,9996;\" } ], \"@timestamp\": \"2020-03-05T19:17:33.962Z\" }",
                     "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
    "response_http_headers" => [
        [0] {
            "Content-Type" => "application/json"
        },
        [1] {
            "Date" => "Thu, 05 Mar 2020 19:17:35 GMT"
        },
        [2] {
            "Content-Length" => "459"
        },
        [3] {
            "Via" => "1.1 AgAAANE7P4c-"
        },
        [4] {
            "hostname" => "myhost.com"
        },
        [5] {
            "msgid" => "414d51205343535131202020202020205e56fb652147c3b6"
        },
        [6] {
            "Access-Control-Allow-Origin" => "*"
        },
        [7] {
            "Access-Control-Allow-Methods" => "POST"
        },
        [8] {
            "X-RateLimit-Limit" => "name=default,10000;"
        },
        [9] {
            "X-RateLimit-Remaining" => "name=default,9996;"
        }
    ],
                    "agent" => {
                  "id" => "c32a8203-358f-425e-a91f-aece2ca98ef7",
                "type" => "filebeat",
             "version" => "7.3.1",
        "ephemeral_id" => "a8fec82e-9b3e-4fa6-b025-36c9ec37a4dd",
            "hostname" => "XXXXXX"
    },
                   "fields" => {
        "log_type" => "XXXXXX",
        "log_farm" => "XXXXXXX"
    },
                 "@version" => "1",
               "bytes_sent" => 16,
             "request_body" => "",
               "@timestamp" => 2020-03-05T19:17:33.962Z,
                      "log" => {
        "offset" => 4384,
          "file" => {
            "path" => "XXXXXXXXX"
        }
    }

[Badger]

You could do it in ruby

ruby {
      code => '
          event.get("[response_http_headers]").each { |h|
              h.each { |k, v|
                  if v =~ /myhost.com/i
                      event.set( "Raj", "found myhost")
                  end
              }
          }
     '
}

[rajsolanki (rajesh)]

thanks Badger.

so instead of your solution i did this.

 ruby {
               code => '
                       event.set("response_http_headers_tmp",event.get("response_http_headers").to_json)
                     '
            }
 if "myhost.com" in [response_http_headers_tmp]
            {
                 mutate{
                 add_field => { "Raj" => "found myhost" }
                }
           }

then i m removing response_http_headers_tmp . so performance wise which is better ?

[Badger]

@rajsolanki

so performance wise which is better ?

You would have to test that.

[rajsolanki (rajesh)]

@Badger

=~ /myhost.com/i

Thanks. I think i will stick with my logic and see monitoring data for few days then switch to your logic. Also i know its watcher question but its related to same topic . I have a watcher alert in this index for HTTP status 422 or 500. i populate request body and response body in my API upon failures. so watcher is triggering correctly and sends alert but email doesnt have value of req body and res body . but in email attachment of json it has all the values. I m using correct values. Only thing is req body and res body is json . So do i need to do any translation ?

my email looks like this.

Request : https://myhost.com/myapi/api
Status : **422 Error**
Endpoint_url :N/A
Total_Errors: 1
RequestBody:
ResponseBody:
TimeStamp: 2020-03-05T13:38:34.883Z|

in attached document my response body looks like this

"response_body":"{ "error":{ "id":"2" , "customer" : "mycust"}}"

and in my watcher i m using {{ctx.payload.hits.response_body}}.

[rajsolanki (rajesh)]

any update Friends ?