Hi,
I am a happy ELK user, but I do face an issue at the moment which I cannot find a solution for, or find any related topics on this specific issue:
I have a very large data pool of JSON arrays which I try to load in ElasticSearch through logstash. The issue I am facing is that I have some fields which are most of the times empty, but sometimes it contains an object with 2 value's. When I use dynamic mapping, it will not load all the output in ES: it will dismiss all documents which contain a object in one of those fields. When I create a seperate property for these fields, it will then only load the data where there is an object in these fields, but will dismiss all documents with empty fields. At the moment I use multiple templates as a workaround, but I would like to have 1 template which works for all of the data.
My Configuration:
Template1:
"properties" : {
"aaaaaaaaaa" : { "type" : "string", "index" : "not_analyzed" },
"xxxxxxxxxxx" : { "type" : "object", "properties" : {
"display_value" : { "type": "string" },
"link" : { "type": "string" }
in this case xxxxxxxxxxx will only go to ES if the xxxxxxxxxxx field contains an object. When the object field is empty ("") it gives the error: "reason"=>"object mapping for [xxxxxxxxxxx] tried to parse field [xxxxxxxxxxx] as object, but found a concrete value"
When I remove this part from the template, it will load all the data which does not contain an object in the xxxxxxxxxxx field, but does not load the data where this field contains an object.
The complete template (without this object specified) is:
Template2:
{
"template": "json_log-",
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"default": {
"numeric_detection": true,
"dynamic_date_formats": [
"dd-MMM-yyyy HH:mm:ss"
],
"date_detection": true,
"_all": {
"enabled": true,
"omit_norms": true
},
"dynamic_templates": [
{
"message_field": {
"match": "message",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "analyzed",
"omit_norms": true
}
}
},
{
"string_fields": {
"match": "",
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "analyzed",
"omit_norms": true,
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}
}
}
}
}
],
"properties" : {
"short_description" : { "type" : "string", "index" : "not_analyzed" }
}
}
}
}
Thanks in Advance!