Logstash code=>"json" not parse complex json log as expected (please close this topic)

[cheriemilk]

Hi team,

I have below logstash.conf and json event. Most of the fields are parsed correctly, but for complex array types, it looks like that not parsed as expecte. For example, profile criteria and rating criteria are json object arrays and the actual parsed result is below.

But the expected parsed result is below, and the first level fields (searchOption, keywordCriteria, ratingcriteria or profilecriteria etc.) in body{} is dynamic and could be changed
"body.criteria.type": "text"
"body.criteria.id": "country"
"body.criteria.value": "USA" (here should be parsed as array)
...
"body.ratingCriteria.Criterias": "2018 - 12 - 04", "2018 - 12 - 31" (here should be parsed as array, not string)

**1. logstash configuration **

input {
file{
path => "C:/elkstack/elasticsearch-6.5.1/logs/app.log"
start_position => "beginning"
sincedb_path => "null"
codec => "json"
}
}

2. json event f

"body": {
	"Action": "TalentSearch",
	"searchOption": [{
		"id": "allCriteriaOptional",
		"value": ["false"]
	}],
	"keywordCriteria": {
		"searchValue": "",
		"bgTypes": ["languages", "specialAssign", "insideWorkExperience", "memberships", "benefitselection", "outsideWorkExperience", "sysScoreCardDevelopmentObjectivesPortlet", "mobility", "documents", "compensation", "tsv2##mdftalentpoolcriteria", "promotability", "leadExperience", "awards", "googledocs", "varPayEmpHistData", "courses", "funcExperience", "education", "fsaelection", "certificates", "community", "preferredNextMove", "tsv2##jobinfo"]
	},
	"userStatusCriteria": {
		"source": [{
			"key": "includeInactive",
			"value": "true"
		}]
	},
	"profileCriteria": [{
		"id": "country",
		"type": "text",
		"value": ["USA"]
	}],
	"facetedCriteria": [{
		"id": "department",
		"elements": [{
			"value": "Marketing"
		}]
	}, {
		"id": "division",
		"elements": []
	}, {
		"id": "location",
		"elements": []
	}],
	"ratingCriteria": [{
		"id": "sysOverallPotential",
		"type": "7",
		"name": "Potential - 3x3 Rating",
		"item": "null",
		"scaleId": "Potential",
		"scaleMin": "1",
		"scaleMax": "3",
		"criterias": [{
			"id": "tsv2RatingValidFrom",
			"type": "date",
			"value": ["2018 - 12 - 04", "2018 - 12 - 31"]
		}, {
			"id": "tsv2RatingValidTo",
			"type": "date",
			"value": ["2018 - 12 - 31", "2018 - 12 - 31"]
		}, {
			"id": "tsv2RatingFromValue",
			"type": "prepopulate",
			"value": ["1.0"]
		}, {
			"id": "tsv2RatingEndValue",
			"type": "prepopulate",
			"value": ["2.0"]
		}]
	}]
}

}

[Badger]

You JSON is not valid JSON. However, if I add {} around it and puts quotes around the dates in the value arrays inside ratingCriteria.criterias I get

        "ratingCriteria" => [
        [0] {
             "scaleMax" => "3",
              "scaleId" => "Potential",
             "scaleMin" => "1",
            "criterias" => [
                [0] {
                     "type" => "date",
                       "id" => "tsv2RatingValidFrom",
                    "value" => [
                        [0] "2018 - 12 - 04",
                        [1] "2018 - 12 - 31"
                    ]

As far as I can see, everything that should be an array is an array.

[cheriemilk]

Hi Badger,

Thank you ~

1. There's quotes added actually for dates( Sorry , I pasted the wrong json version). What do you mean by add {} around it? like this?? "value": [{"2018 - 12 - 04", "2018 - 12 - 31"}]
2. I already got the result yesterday ,same as what you get. But the extract result looks like not flat enough, and still a json structure for profile criteria, rating criteria and faceted criteria. My expected parse result is

"body.ratingCriteria.scaleMax": "3"
"body.ratingCriteria.scaleId": "Potential"
"body.ratingCriteria.scaleMin": "1"
"body.ratingCriteria.criterias.type": "date"
"body.ratingCriteria.criterias.id": "tsv2RatingValidFrom"
"body.ratingCriteria.criterias.value": "2018 - 12 - 04", "2018 - 12 - 31"

I googled yesterday that ruby filter might be a solution for such complex json and dynamic field scenario , but not familiar with ruby language. Do you have idea about it? or anyother alternatives I can use to meet my parse requirement?

Thanks,
Cherie

Thanks,
Cherie

[Badger]

@cheriemilk

"body": { }

When I say you need {} around that to be valid JSON I mean it needs to be

{ "body": {...} }

I do not understand what you do not like about the way the JSON is parsed.