# Parse Generic Nestaed json without splitting

**URL:** <https://discuss.elastic.co/t/parse-generic-nestaed-json-without-splitting/246435>\
**Category:** Logstash\
**Created:** [August 26, 2020, 10:49am UTC](https://discuss.elastic.co/t/parse-generic-nestaed-json-without-splitting/246435 "2020-08-26T10:49:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhargav\_Patel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhargav_patel1/32/74535_2.png) [@Bhargav\_Patel1](https://discuss.elastic.co/u/Bhargav_Patel1)\
**Post date:** [August 26, 2020, 10:49am UTC](https://discuss.elastic.co/t/parse-generic-nestaed-json-without-splitting/246435/1 "2020-08-26T10:49:43Z")

</div>

Hello,

I have valid json string which i want to parse as json and want to stored in elastic

```auto
{
	"id": "101834545",
	"date": "06-Apr-2016",
	"Gender": "Male",
	"ActiveStatus": "Active",
	"NameDetails": {
		"Name": [{
			"NameType": "Primary Name",
			"NameValue": {
				"FirstName": "xxx",
				"Surname": "x"
			}
		}]
	},
	"Descriptions": {
		"Description": [{
			"Description1": "1"
		}, {
			"Description1": "3",
			"Description2": "10"
		}]
	},
	"RoleDetail": {
		"Roles": [{
			"RoleType": "Primary Occupation",
			"OccTitle": {
				"OccCat": "16",
				"_text": "Deceased"
			}
		}, {
			"RoleType": "Previous Roles",
			"OccTitle": [{
				"SinceMonth": "Mar",
				"SinceYear": "2005",
				"ToYear": "2005",
				"OccCat": "16"
			}, {
				"SinceDay": "22",
				"SinceMonth": "Oct",
				"SinceYear": "1993",
				"ToDay": "15",
				"ToMonth": "Mar",
				"ToYear": "2003",
				"OccCat": "1"
			}, {
				"SinceDay": "08",
				"SinceMonth": "Dec",
				"SinceYear": "1976",
				"ToDay": "14",
				"ToMonth": "Jul",
				"ToYear": "1978",
				"OccCat": "1"
			}, {
				"SinceMonth": "Nov",
				"SinceYear": "2010",
				"ToMonth": "Jan",
				"ToYear": "2011",
				"OccCat": "16"
			}]
		}]
	},
	"DateDetails": {
		"Date": [{
			"DateType": "Date of Birth",
			"DateValue": {
				"Day": "25",
				"Month": "Jan",
				"Year": "88888"
			}
		}, {
			"DateType": "Deceased Date",
			"DateValue": {
				"Day": "05",
				"Month": "Apr",
				"Year": "2011"
			}
		}, {
			"DateType": "Inactive as of (PEP)",
			"DateValue": {
				"Month": "Jan",
				"Year": "2011"
			}
		}]
	},
	"BirthPlace": {
		"Place": {
			"name": "xxxxxxx"
		}
	},
	"CountryDetails": {
		"Country": [{
			"CountryType": "xxxxxxxxx",
			"CountryValue": {
				"Code": "xxxxxxxxx"
			}
		}, {
			"CountryType": "Resident of",
			"CountryValue": {
				"Code": "xxxxxxxxx"
			}
		}, {
			"CountryType": "Jurisdiction",
			"CountryValue": {
				"Code": "xxxxxxxxxxxx"
			}
		}, {
			"CountryType": "Country of Reported Allegation",
			"CountryValue": {
				"Code": "xxxxxxxxxxxxx"
			}
		}]
	},
	"ProfileNotes": "xxxxxxxxxx"
}

```

This is my config file

```auto
##Input section
input {
  file {
         path => "/Users/Downloads/Factiva_PFA_Feed_XML/AML/*.json"
         type => "json"
         codec => "json"
         start_position => "beginning"
    }
}
##Filter section
filter {
   grok {
      match => ["path","/Users/Downloads/Factiva_PFA_Feed_XML/AML/%{DATA:filename}.json"]
    }
}
filter{
    json{
        source => "message"
    }
}
filter {
  mutate {
    gsub => [ "filename", "\d", "",
          "filename", "_","" ]
  }
  mutate {
    lowercase => ["filename"]
  }
}
##Output section
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{filename}-%{+YYYY.MM.dd}"
    codec => "json"
  }
}

```

and getting below error .. since the above json is dynamic any filed can be added anytime so i dont know what key added and what type of value added it would be array or object or simple text

I am getting exceptions

```auto
[2020-08-26T16:03:47,102][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{filename}-2020.08.26", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x1d7077fb>], :response=>{"index"=>{"_index"=>"%{filename}-2020.08.26", "_type"=>"doc", "_id"=>"fjhUKnQBAFysE7Y4rNfV", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [RoleDetail.Roles.OccTitle] tried to parse field [OccTitle] as object, but found a concrete value"}}}}

```

file store data in following manner

```auto
{"id":"32323232323","action":"add","date":"06-Apr-2016","Gender":"Male","ActiveStatus":"Active","Deceased":"Yes","NameDetails":{"Name":[{"NameType":"Primary Name","NameValue":{"FirstName":"xxxx","Surname":"xxxxxx"}},{"NameType":"Spelling Variation","NameValue":[{"FirstName":"xxxx","Surname":"xxxx"},{"FirstName":"xxxxxx","Surname":"xxxxx"},{"FirstName":"xxxxxxx","Surname":"xxxxxx"}]}]},"Descriptions":{"Description":[{"Description1":"1"},{"Description1":"3","Description2":"10"}]},"RoleDetail":{"Roles":[{"RoleType":"Primary Occupation","OccTitle":{"OccCat":"16","_text":"Deceased"}},{"RoleType":"Previous Roles","OccTitle":[{"SinceMonth":"Mar","SinceYear":"2005","ToYear":"2005","OccCat":"16","_text":"xxxxx"},{"SinceDay":"22","SinceMonth":"Oct","SinceYear":"1993","ToDay":"15","ToMonth":"Mar","ToYear":"2003","OccCat":"1","_text":"xxxxxxxx"},{"SinceDay":"08","SinceMonth":"Dec","SinceYear":"1976","ToDay":"14","ToMonth":"Jul","ToYear":"1978","OccCat":"1","_text":"xxxxxxxx"},{"OccCat":"7","_text":"xxxxxxxx"},{"SinceMonth":"Nov","SinceYear":"2010","ToMonth":"Jan","ToYear":"2011","OccCat":"16","_text":"xxxxxxxx"},{"OccCat":"16","_text":"xxxxxxxx"}]}]},"DateDetails":{"Date":[{"DateType":"Date of Birth","DateValue":{"Day":"25","Month":"Jan","Year":"1937"}},{"DateType":"Deceased Date","DateValue":{"Day":"05","Month":"Apr","Year":"2011"}},{"DateType":"Inactive as of (PEP)","DateValue":{"Month":"Jan","Year":"2011"}}]},"BirthPlace":{"Place":{"name":"xxxxxxxxx"}},"CountryDetails":{"Country":[{"CountryType":"Citizenship","CountryValue":{"Code":"xxxxxxx"}},{"CountryType":"Resident of","CountryValue":{"Code":"xxxxxx"}},{"CountryType":"xxxxxx","CountryValue":{"Code":"xxxxxxx"}},{"CountryType":"xxxxxxx","CountryValue":{"Code":"xxxxx"}}]},"ProfileNotes":"xxxxx"}
{"id":"121212121212","action":"add","date":"06-Apr-2016","Gender":"Male","ActiveStatus":"Active","Deceased":"Yes","NameDetails":{"Name":[{"NameType":"Primary Name","NameValue":{"FirstName":"xxxx","Surname":"xxxxxx"}},{"NameType":"Spelling Variation","NameValue":[{"FirstName":"xxxx","Surname":"xxxx"},{"FirstName":"xxxxxx","Surname":"xxxxx"},{"FirstName":"xxxxxxx","Surname":"xxxxxx"}]}]},"Descriptions":{"Description":[{"Description1":"1"},{"Description1":"3","Description2":"10"}]},"RoleDetail":{"Roles":[{"RoleType":"Primary Occupation","OccTitle":{"OccCat":"16","_text":"Deceased"}},{"RoleType":"Previous Roles","OccTitle":[{"SinceMonth":"Mar","SinceYear":"2005","ToYear":"2005","OccCat":"16","_text":"xxxxx"},{"SinceDay":"22","SinceMonth":"Oct","SinceYear":"1993","ToDay":"15","ToMonth":"Mar","ToYear":"2003","OccCat":"1","_text":"xxxxxxxx"},{"SinceDay":"08","SinceMonth":"Dec","SinceYear":"1976","ToDay":"14","ToMonth":"Jul","ToYear":"1978","OccCat":"1","_text":"xxxxxxxx"},{"OccCat":"7","_text":"xxxxxxxx"},{"SinceMonth":"Nov","SinceYear":"2010","ToMonth":"Jan","ToYear":"2011","OccCat":"16","_text":"xxxxxxxx"},{"OccCat":"16","_text":"xxxxxxxx"}]}]},"DateDetails":{"Date":[{"DateType":"Date of Birth","DateValue":{"Day":"25","Month":"Jan","Year":"1937"}},{"DateType":"Deceased Date","DateValue":{"Day":"05","Month":"Apr","Year":"2011"}},{"DateType":"Inactive as of (PEP)","DateValue":{"Month":"Jan","Year":"2011"}}]},"BirthPlace":{"Place":{"name":"xxxxxxxxx"}},"CountryDetails":{"Country":[{"CountryType":"Citizenship","CountryValue":{"Code":"xxxxxxx"}},{"CountryType":"Resident of","CountryValue":{"Code":"xxxxxx"}},{"CountryType":"xxxxxx","CountryValue":{"Code":"xxxxxxx"}},{"CountryType":"xxxxxxx","CountryValue":{"Code":"xxxxx"}}]},"ProfileNotes":"xxxxx"}
{"id":"54545454545","action":"add","date":"06-Apr-2016","Gender":"Male","ActiveStatus":"Active","Deceased":"Yes","NameDetails":{"Name":[{"NameType":"Primary Name","NameValue":{"FirstName":"xxxx","Surname":"xxxxxx"}},{"NameType":"Spelling Variation","NameValue":[{"FirstName":"xxxx","Surname":"xxxx"},{"FirstName":"xxxxxx","Surname":"xxxxx"},{"FirstName":"xxxxxxx","Surname":"xxxxxx"}]}]},"Descriptions":{"Description":[{"Description1":"1"},{"Description1":"3","Description2":"10"}]},"RoleDetail":{"Roles":[{"RoleType":"Primary Occupation","OccTitle":{"OccCat":"16","_text":"Deceased"}},{"RoleType":"Previous Roles","OccTitle":[{"SinceMonth":"Mar","SinceYear":"2005","ToYear":"2005","OccCat":"16","_text":"xxxxx"},{"SinceDay":"22","SinceMonth":"Oct","SinceYear":"1993","ToDay":"15","ToMonth":"Mar","ToYear":"2003","OccCat":"1","_text":"xxxxxxxx"},{"SinceDay":"08","SinceMonth":"Dec","SinceYear":"1976","ToDay":"14","ToMonth":"Jul","ToYear":"1978","OccCat":"1","_text":"xxxxxxxx"},{"OccCat":"7","_text":"xxxxxxxx"},{"SinceMonth":"Nov","SinceYear":"2010","ToMonth":"Jan","ToYear":"2011","OccCat":"16","_text":"xxxxxxxx"},{"OccCat":"16","_text":"xxxxxxxx"}]}]},"DateDetails":{"Date":[{"DateType":"Date of Birth","DateValue":{"Day":"25","Month":"Jan","Year":"1937"}},{"DateType":"Deceased Date","DateValue":{"Day":"05","Month":"Apr","Year":"2011"}},{"DateType":"Inactive as of (PEP)","DateValue":{"Month":"Jan","Year":"2011"}}]},"BirthPlace":{"Place":{"name":"xxxxxxxxx"}},"CountryDetails":{"Country":[{"CountryType":"Citizenship","CountryValue":{"Code":"xxxxxxx"}},{"CountryType":"Resident of","CountryValue":{"Code":"xxxxxx"}},{"CountryType":"xxxxxx","CountryValue":{"Code":"xxxxxxx"}},{"CountryType":"xxxxxxx","CountryValue":{"Code":"xxxxx"}}]},"ProfileNotes":"xxxxx"}

```

Is there any generic way to store data as it is in json ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 26, 2020, 4:28pm UTC](https://discuss.elastic.co/t/parse-generic-nestaed-json-without-splitting/246435/2 "2020-08-26T16:28:22Z")

</div>

Try reading [this](https://discuss.elastic.co/t/error-indexing-using-xml-plugin/223774/2) thread. Or search this forum for either "concrete value" or "mapper\_parsing\_exception". This has been discussed many times.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 23, 2020, 4:28pm UTC](https://discuss.elastic.co/t/parse-generic-nestaed-json-without-splitting/246435/3 "2020-09-23T16:28:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.