Object mapping [event_tags] can't be changed from nested to non-nested

[mathias]

Hi,

I have events indexed with logstash
I have a template defined in elasticsearch
Indexing only works sometimes after a new installation of my ELK.
Most of the times ES complains that mapping can't change from nested to non nested.

[2017-01-17T19:52:52,173][DEBUG][o.e.a.b.TransportShardBulkAction] [2s9hNaW] [events-2017.01.17][3] failed to execute bulk item (index) index {[events-2017.01.17][event/system/counter/pbuf][AVmt_T6yvOHW3FxTTZA5], source[{"sequence_number":73,"event_tags":[{"value":"nacl-st-pew5x9-ppu-server-b45nkwxfzwiy","key":"originhost"}],"@timestamp":"2017-01-17T19:52:52.014Z","data":{"counters":[{"name":"n_lost_packets","value":0},{"name":"n_sent_bytes","value":1701361752},{"name":"n_received_packets","value":5166146},{"name":"n_sent_packets","value":10418914},{"name":"n_ce_bytes","value":0},{"name":"n_received_bytes","value":284144254}],"resources":[{"identity":"bearer_path_pm","type":"name"},{"identity":"1","type":"ue_bearer_id"},{"identity":"","type":"pm_id"},{"identity":"1","type":"path_identifier"},{"identity":"2017-01-17 20:52:52.167567","type":"time"}]},"port":41432,"@version":"1","host":"10.68.32.189","producer":"1bda868e172de179c9d7e1b9955655fc059fe810","eventtype":"system/counter/pbuf","type":"event/system/counter/pbuf","timestamp":"2017-01-17T19:52:52.55313611Z"}]}
java.lang.IllegalArgumentException: object mapping [event_tags] can't be changed from nested to non-nested
	at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:430) ~[elasticsearch-5.0.0.jar:5.0.0]

I can not see why my template is not working properly.
Can someone help me?

Before I start indexing I have an no event indexes:

curl -XGET 'http://elasticsearch:9200/_cat/indices?pretty'
green open .kibana                         Ic1CJibdQcKXm94W-UVc3w 1 0     62 362 296.3kb 296.3kb

My template looks like this:
curl -XGET 'http://elasticsearch:9200/_template/e*?pretty'
{
"eventbus" : {
"order" : 0,
"template" : "events-",
"settings" : {
"index" : {
"number_of_replicas" : "0"
}
},
"mappings" : {
"event" : {
"dynamic_templates" : [
{
"notanalyzed" : {
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
},
"match_mapping_type" : "string",
"match" : ""
}
}
],
"properties" : {
"sequence_number" : {
"type" : "long"
},
"event_tags" : {
"type" : "nested"
},
"data" : {
"type" : "object"
},
"producer" : {
"index" : "not_analyzed",
"type" : "keyword"
},
"eventtype" : {
"type" : "keyword"
},
"timestamp" : {
"format" : "date_time",
"type" : "date"
}
}
}
},
"aliases" : { }
}
}

My data looks like this:
{
"sequence_number" => 73,
"event_tags" => [
[0] {
"value" => "nacl-st-pew5x9-ppu-server-b45nkwxfzwiy",
"key" => "originhost"
}
],
"@timestamp" => 2017-01-17T19:52:52.014Z,
"data" => {
"counters" => [
[0] {
"name" => "n_lost_packets",
"value" => 0
},
[1] {
"name" => "n_sent_bytes",
"value" => 1701361752
},
[2] {
"name" => "n_received_packets",
"value" => 5166146
},
[3] {
"name" => "n_sent_packets",
"value" => 10418914
},
[4] {
"name" => "n_ce_bytes",
"value" => 0
},
[5] {
"name" => "n_received_bytes",
"value" => 284144254
}
],
"resources" => [
[0] {
"identity" => "bearer_path_pm",
"type" => "name"
},
[1] {
"identity" => "1",
"type" => "ue_bearer_id"
},
[2] {
"identity" => "",
"type" => "pm_id"
},
[3] {
"identity" => "1",
"type" => "path_identifier"
},
[4] {
"identity" => "2017-01-17 20:52:52.167567",
"type" => "time"
}
]
},
"port" => 41432,
"@version" => "1",
"host" => "10.68.32.189",
"producer" => "1bda868e172de179c9d7e1b9955655fc059fe810",
"eventtype" => "system/counter/pbuf",
"type" => "event/system/counter/pbuf",
"timestamp" => "2017-01-17T19:52:52.55313611Z"
}

My list of indexes after a while. The event index is created but not populated.
curl -XGET 'http://elasticsearch:9200/_cat/indices?pretty'
green open events-2017.01.17 pIfhTieCRwqOS0P9Cvi2Kw 5 0 0 0 650b 650b

Isn't "event_data" nested in the example above?
If so, why there an attempt to change to non nested?

Thanks
Mathias

[mathias]

Hi,

I have removed the event_tags from the application generating the event.
The event now looks like this in Kibana:

{
  "_index": "events-2017.01.18",
  "_type": "event/system/counter/pbuf",
  "_id": "AVmwbnUhpR5aTmxaC2lQ",
  "_score": null,
  "_source": {
    "sequence_number": 581,
    "event_tags": [],
    "@timestamp": "2017-01-18T07:15:44.377Z",
    "data": {
      "counters": [
        {
          "name": "n_lost_packets",
          "value": 0
        },
        {
          "name": "n_sent_bytes",
          "value": 10716880
        },
        {
          "name": "n_received_packets",
          "value": 375984
        },
        {
          "name": "n_sent_packets",
          "value": 194624
        },
        {
          "name": "n_ce_bytes",
          "value": 0
        },
        {
          "name": "n_received_bytes",
          "value": 525896260
        }
      ],
      "resources": [
        {
          "identity": "bearer_path_pm",
          "type": "name"
        },
        {
          "identity": "1",
          "type": "ue_bearer_id"
        },
        {
          "identity": "",
          "type": "pm_id"
        },
        {
          "identity": "0",
          "type": "path_identifier"
        },
        {
          "identity": "2017-01-18 08:15:45.115235",
          "type": "time"
        }
      ]
    },
    "port": 37312,
    "@version": "1",
    "host": "10.68.32.232",
    "producer": "c8692b2c438fd391406ac60eb64cba04b2bdcf10",
    "eventtype": "system/counter/pbuf",
    "type": "event/system/counter/pbuf",
    "timestamp": "2017-01-18T07:15:45.495239019Z"
  },
  "fields": {
    "timestamp": [
      1484723745495
    ],
    "@timestamp": [
      1484723744377
    ]
  },
  "sort": [
    1484723744377
  ]
}

Next to the event_tags I see the following text when I hover over the triangle:
"No cashed mapping for this field. Refresh field list from the Management > Index patterns page."
When going to the Index pattern page I can not find "event_tags" after refresh.
Does this mean there is something wrong with my template.
To me it looks like my template does not match my event. But I do not understand why.

First part of my index looks like this:
curl -XGET 'http://elasticsearch:9200/events-2017.01.18?pretty'
{
"events-2017.01.18" : {
"aliases" : { },
"mappings" : {
"event" : {
"dynamic_templates" : [
{
"notanalyzed" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"index" : "not_analyzed",
"type" : "string"
}
}
}
],
"properties" : {
"data" : {
"type" : "object"
},
"event_tags" : {
"type" : "nested"
},
"eventtype" : {
"type" : "keyword"
},
"producer" : {
"type" : "keyword"
},
"sequence_number" : {
"type" : "long"
},
"timestamp" : {
"type" : "date",
"format" : "date_time"
}
}
},
"event/system/counter/pbuf" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"data" : {
"properties" : {
"counters" : {
"properties" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"value" : {
"type" : "long"
}
}
},
.
.
.

Br Mathias