How to retrieve JSON object from log in logstash confuguration?

Elastic Stack Logstash
1

I am creating an application where I need to retrieve key value pairs from json object in logs with the help of logstash configuraition.

Here is my configuration:

 input{
    file{
    path => "D:\ELK_Info\TestLogs_Updated_tablev4.log"
    start_position => beginning
            codec => multiline {
                                   pattern => "^%{TIMESTAMP_ISO8601}"
                                   negate => true
                                   what => "previous"
            }
    }
    }

filter{
    grok{
       match => {
      "message" => "%{IP:client_ip}%{NOTSPACE:space}%{GREEDYDATA:json_data}"
   }
  }
  #mutate { remove_field => [ "tags"]}
  json { source => "json_data" target => "parsedJson" remove_field=>["json_data"]}
  mutate {
     add_field => {
        "AssetManagerId" => "%{[parsedJson][AssetManagerId]}"
        "Amount" => "%{[parsedJson][Amount]}"
		"AccountId" => "%{[parsedJson][AccountId]}"
		"RequestCode" => "%{[parsedJson][RequestCode]}"
		"TicketNumber" => "%{[parsedJson][TicketNumber]}"
		"Status" => "%{[parsedJson][Status]}"
        "message" => ["%{[parsedJson][message]}"]
      }
    }
}

output {
file{
path => "D:\ELK_Info\logstashOutput.log"
}
}

Here is the log for example:

Sep 28 15:09:50 52.231.153.246 gateway: [6] INFO AppLog - 180 - XXXGatewayAPI.APIHandlers - UpdateDepositTicket called by xyzadmin from 211.211.211.211: {"AssetManagerId":211,"AccountId":211,"AssetId":211,"AssetName":" ","Amount":"211","RequestCode":"211-211-211-211-211","RequestIP":"211.211.211.211","RequestUser":211,"RequestUserName":"211@211.com","OperatorId":211,"Status":"Accepted","FeeAmt":0,"UpdatedByUser":211,"UpdatedByUserName":"211","TicketNumber":211,"DepositInfo":"{"Full Name":"211","language":"kr","Comments":""}","CreatedTimestamp":"2018-09-27T11:02:22Z","LastUpdateTimeStamp":"211-09-211:09:48.203Z","Comments":,"Attachments":null,"type":"deposit"}

With my current configuration, I get key value pair as:

"Status" : "%{[parsedJson][Status]}"

whereas, I need exact value for key 'Status' in json in the place of "%{[parsedJson][Status]}".

What changes I need to make to have required output?

2

I do not understand your question. What do you mean with "I need exact value for key 'Status' in json in the place of '%{[parsedJson][Status]}'."?

The JSON in the log example seems to be not valid. Formatted it looks like this:

    {
    "AssetManagerId": 211,
    "AccountId": 211,
    "AssetId": 211,
    "AssetName": " ",
    "Amount": "211",
    "RequestCode": "211-211-211-211-211",
    "RequestIP": "211.211.211.211",
    "RequestUser": 211,
    "RequestUserName": "211@211.com",
    "OperatorId": 211,
    "Status": "Accepted",
    "FeeAmt": 0,
    "UpdatedByUser": 211,
    "UpdatedByUserName": "211",
    "TicketNumber": 211,
    "DepositInfo": "{" Full Name ":" 211 "," language ":" kr "," Comments ":" "}",
    "CreatedTimestamp": "2018-09-27T11:02:22Z",
    "LastUpdateTimeStamp": "211-09-211:09:48.203Z",
    "Comments": [],
    "Attachments": null,
    "type": "deposit"
}

The stuff in DepositInfo is invalid JSON.

3

Thanks for your correction, redX.
What I mean is, currently in output I am getting values as

"AssetManagerId" => "%{[parsedJson][AssetManagerId]}"

for all fields in json retrieved. But what I need is something like

"AssetManagerId": 211,

I need to get value from the input json. Please check json data. Hope you got my point.

4

Got it.
If I remove the input->file->codec defintion from your config, it works:

config:

input {
    file {
        path => "C:\ProgramFiles\logstash-6.3.0\test.log"
        start_position => beginning
        sincedb_path => "NUL"
    }
}

filter {
    grok {
        match => {
        "message" => "%{IP:client_ip}%{NOTSPACE:space}%{GREEDYDATA:json_data}"
        }
    }
    json {
        source => "json_data" 
        target => "parsedJson" 
        remove_field => ["json_data"]
    }
    mutate {
        add_field => {
            "AssetManagerId" => "%{[parsedJson][AssetManagerId]}"
            "Amount" => "%{[parsedJson][Amount]}"
            "AccountId" => "%{[parsedJson][AccountId]}"
            "RequestCode" => "%{[parsedJson][RequestCode]}"
            "TicketNumber" => "%{[parsedJson][TicketNumber]}"
            "Status" => "%{[parsedJson][Status]}"
            "message" => ["%{[parsedJson][message]}"]
        }
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

And I removed the invalid part from your JSON:

 Sep 28 15:09:50 52.231.153.246 gatewayy: [6] INFO AppLog - 180 - XXXGatewayAPI.APIHandlers - UpdateDepositTicket called by xyzadmin from 211.211.211.211: {"AssetManagerId": 211,"AccountId": 211,"AssetId": 211,"AssetName": " ","Amount": "211","RequestCode": "211-211-211-211-211","RequestIP": "211.211.211.211","RequestUser": 211,"RequestUserName": "211@211.com","OperatorId": 211,"Status": "Accepted","FeeAmt": 0,"UpdatedByUser": 211,"UpdatedByUserName": "211","TicketNumber": 211,"CreatedTimestamp": "2018-09-27T11:02:22Z","LastUpdateTimeStamp": "211-09-211:09:48.203Z","Comments": [],"Attachments": null,"type": "deposit"}

Then I get:

{
	"space" => ":",
	"path" => "C:\\ProgramFiles\\logstash-6.3.0\\test.log",
	"AccountId" => "211",
	"host" => "[...]",
	"Amount" => "211",
	"@version" => "1",
	"TicketNumber" => "211",
	"RequestCode" => "211-211-211-211-211",
	"client_ip" => "211.211.211.211",
	"message" => [
		[0]"[...]",
		[1]"%{[parsedJson][message]}"
	],
	"@timestamp" => 2018 - 10 - 22T16: 24: 35.555Z,
	"Status" => "Accepted",
	"parsedJson" => {
		[...]
	},
	"AssetManagerId" => "211"
}

If your logs are on one line, then you don't need the multiline coded. If you need it, you need to investigate further. But the problem then is with that part only.

1 Like
5

Thanks redX, it worked. Can you please take a look at this question too?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.