Logstash parse mixed json text

Elastic Stack Logstash
1

please can any one help me to parse below mixed json txt from below logentry.
need to extract request json fields & response json fields

2020-01-06 01:02:10.869 +01:00 [Information] WIN HTTP Response at 2020-01-06T01:02:10.8598940+01:00. Response Time 2041.7975 ms.
Request:
{
"Method": "GET",
"Url": "http:\dummyurl.com",
"Headers": {
"Cache-Control": [
"no-cache, no-store, max-age=0"
],
"Connection": [
"close"
],
"Pragma": [
"no-cache"
],
"Via": [
"1 HTTP/1.1 :455"
],
"Accept": [
"application/json, text/plain, /"
],
"Accept-Language": [
"en-US,en;q=0.9"
],
"token": [
"xxxxx"
],
"Cookie": [
"XXXXX"
],
"Host": [
"test.com"
],
"If-Modified-Since": [
"Mon, 20 Jul 1995 05:00:00 GMT"
],
"Referer": [
"http://dummy.com/test"
],
"User-Agent": [
"Safari/537.36"
],
"iv-genera": [
"1R"
],
"groups": [
""USERS""
],
"remote-address": [
"1.X.1.3"
],
"user": [
"123456789"
],
"sec-ch-ua-mobile": [
"?0"
],
"true-client-ip": [
"1.1.1.2"
],
"sec-fetch-site": [
"same-origin"
],
"server_name": [
"test123"
],
"sec-fetch-dest": [
"empty"
],
"trace": [
"ywdwdahda"
]
},
"Body": ""
}
Response:
{
"StatusCode": 200,
"Headers": {
"Content-Type": [
"text/json"
],
"test-val": [
"1"
],
"Set-Cookie": [
"HttpOnly"
],

"test-Control": [
  "test"
]

},
"Body": ""
}

2

Assuming your JSON is valid (it is not, as shown) then you could use a multiline codec to put together all the lines of one message. Perhaps

codec => multiline { pattern => "^\d{4}-\d{2}-\d{2} " negate => true what => previous auto_flush_interval => 2 multiline_tag => "" }

Then

    dissect { mapping => { "message" => "%{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{+[@metadata][timestamp]} [" } }
    if [@metadata][timestamp] { date { match => [ "[@metadata][timestamp]", "YYYY-MM-dd HH:mm:ss.SSS ZZ" ] } }

    if "HTTP Response" in [message] {
        grok { match => { "message" => "Request:%{GREEDYDATA:[@metadata][request]}Response:%{GREEDYDATA:[@metadata][response]}" } }
        if [@metadata][request] { json { source => "[@metadata][request]" target => "httpRequest" } }
        if [@metadata][response] { json { source => "[@metadata][response]" target => "httpResponse" } }
        # If you never have more than one value for a header then you could also...
        ruby {
            code => '
                reqHdr = event.get("[httpRequest][Headers]")
                if reqHdr.is_a? Hash
                    reqHdr.each { |k, v|
                        event.set("[httpRequest][Headers][#{k}]", v[0])
                    }
                end
                respHdr = event.get("[httpResponse][Headers]")
                if respHdr.is_a? Hash
                    respHdr.each { |k, v|
                        event.set("[httpResponse][Headers][#{k}]", v[0])
                    }
                end
            '
        }
    }

which will produce

"httpResponse" => {
       "Headers" => {
            "test-val" => "1",
          "Set-Cookie" => "HttpOnly",
        "test-Control" => "test",
        "Content-Type" => "text/json"
    },
    "StatusCode" => 200,
          "Body" => ""
},
 "httpRequest" => {
    "Headers" => {
                   "Cookie" => "XXXXX",
              "server_name" => "test123",
           "sec-fetch-site" => "same-origin",
                   "Accept" => "application/json, text/plain, /",
     ...
4

thanks Badger - solution is working..

my next step is to put the json data into database table..
for example extract data of the fields Authorization , Referrer and iv-user and insert into database table. please could you help

5

I do not understand what you mean by that. In any case, it should be a new topic.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.