JSON parsing depth in logstash

Elastic Stack Logstash
1

Hi All,

Im trying to index the camel logs by using grok pattern and json filter in logstash and then push the same to ES.

Below is the sample log snippet that I'm trying to parse.

07:32:49.805 [Camel (MyCamel) thread #3 - seda://fetchSFDCRetailLogs] INFO  loggerRoute - {"timestamp":"17-2020-08 07-32-49","processName":"EmailValidation","serviceName":"SFDCRETAIL_FE_ADAPTER","targetSystem":"CREDITVIDYA_BE_ADAPTER","messageType":"SourceRequest","username":"SFDCRetail","conversationId":"123345678","transactionId":"ID-sfdcretailfrontendadapter-263-8wsrv-1597305746529-0-4876","headerDetails":{"authorization":"Basic ODY5Y2FjZDk6c2ZkY3JldGFpbHVhdA==","conversationId":"123345678","sourceName":"SFDCRetail","recipient":"","otpRefNo":""},"payload":{
    "firstName": "Jigar",
    "middleName": "Nareshbhai",
    "lastName": "Shah",
    "pfNumber": "",
    "designation": "",
    "companyName": "UNION BANK OF INDIA",
    "city": "Ahmedabad",
    "uniqueId": "",
    "officeStateCode": "",
    "officeAddressLine1": "",
    "email": "THusena.prj@tatacapital.com",
    "clientReference": {
        "transactionId": "",
        "applicantType": "1",
        "losId": "6059185",
        "applicantId": "CUST11056089",
        "loanType": "Personal Loan",
        "leadId": "",
        "webtopNumber": "272PZ0001137",
        "sourceSystem": "sfdc_retail"
    },
    "officeAddressLine3": "",
    "income": "34235",
    "officeAddressLine2": "",
    "imeiNo": "",
    "iPAddress": "",
    "mobileNumber": ""
},"status":""}

Below is the logstash configuration file configured for the same.

input{
	beats{
		port => 5044
	}
}

filter{
	if [kubernetes][namespace] == "dev"{
		grok{
			match => ["message", "%{TIME:timestamp:date} %{GREEDYDATA:Thread} %{WORD:LoggingLevel}  %{WORD:RouteName} - (?<logmessage>(.|\r|\n)*)"]
		}
		json{
			source => "logmessage"
			target => "doc"
		}

		if  "_grokparsefailure" in [tags]{
			drop {}
		}
	}
}
output{
	if [kubernetes][namespace] == "dev"{
		elasticsearch{
				hosts => ["http://ipaddress:port"]
				index => "esblogs-%{+YYYY.MM.dd}"
				user => "elastic"
				password => "password"
		}
	}
}

I'm able to parse the logs using the grok pattern and json filter using the logstash conf file. But this creates all the fields that are part of the above json request.

What I would like to achieve is the json filter should parse logs only for the depth of 1. It should not create fields that are part of "payload" tag in json request.

Can some one please guide me on how to achieve the same.

Thanks and Regards,
Rakesh Chhabria

2

I cannot think of a way to avoid parsing the JSON, but you can use

mutate { remove_field => [ "payload" ] }

to delete it once it has been parsed.

1 Like
3

Hi Badger,

Thanks for your reply, but since payload contains ESB Logs, I cannot remove the field.

Do you have any pointers on how to add the below setting in logstash conf file mentioned above for all the new indices that being created daily.

index.mapping.depth.limit

Below is the link for the same.

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings

Looks like this can solve my issue.

Thanks and Regards,
Rakesh Chhabria

4

You would not do that in logstash, it is an elasticsearch setting. You may be able to set it in an index template, but I am not sure of that.

5

Hi Badger,

Thanks for providing the pointer to remove the payload field and it worked for me.

Since I was logging the entire message as part of "LogMessage" field I considered removing the [doc][payload] field.

Thanks and Regards,
Rakesh Chhabria

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.