Could not index event to Elasticsearch - ES 6.3.2 Failed Mapping

Elastic Stack Elasticsearch
1

Hello, I've been working on getting some dates parsed in Elasticsearch and have been occasionally getting errors from elasticsearch/logstash about a failure to index one of my documents. I suspect it has something to do with the dynamic template configuration, as that's what I've most recently been working on and I'll admit to not fully understanding it. Because I'm limited to 7000 characters and I somehow passed that, this is a pastebin link to what I've been using to put the template of the documents.

And here is the actual error I'm getting randomly:
[2018-08-13T13:45:23,096][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.08.13", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x540e8046>], :response=>{"index"=>{"_index"=>"logstash-2018.08.13", "_type"=>"doc", "_id"=>"__ubNGUBs_yBIBeOWtxx", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"expected map for property [fields] on field [MMM dd, yyyy, HH:mm:ss.SSS Z||yyyy-MM-dd'T'HH:mm:ss.SSSZ||yyyy-MM-dd HH:mm:ss||MM-dd-yyyy HH:mm:ss||dd/MMM/yyyy:HH:mm:ss Z||yyyy-MM-dd HH:mm:ss:SSS||yyyy-MM-dd HH:mm:ss,SSS||yyyy-MM-dd] or [fields] but got a class java.lang.String"}}}}

Unfortunately I can't provide an exact document that's been causing this issue since it doesn't actually index them (is there a way to log unindexed docs...?) but this is a sample doc that does work, grabbed directly from Kibana:

{
  "_index": "logstash-2018.08.13",
  "_type": "doc",
  "_id": "lfubNGUBs_yBIBeOZd0G",
  "_version": 1,
  "_score": null,
  "_source": {
    "log.message": " getStateInfoDB(110,PULSE)",
    "log.level": "DEBUG",
    "log.type": "StateDataAccess",
    "offset": 249376066,
    "message": "DEBUG [2018-08-13 13:45:21,146] [StateDataAccess] - : getStateInfoDB(110,PULSE)",
    "tags": [
      "EP-filebeat",
      "beats_input_codec_plain_applied"
    ],
    "server": {
      "name": "EP-host",
      "client": "asyUvd"
    },
    "@timestamp": "2018-08-13T18:45:21.146Z",
    "@version": "1",
    "source": "/opt/tomcat/logs/catalina.out",
    "beat": {
      "name": "example name",
      "version": "6.3.2",
      "hostname": "hostname.local"
    }
  },
  "fields": {
    "@timestamp": [
      "2018-08-13T18:45:21.146Z"
    ]
  },
  "sort": [
    1534185921146
  ]
}

Any advice is appreciated, whether directly about the issue at hand or general template/elastic tips.

2

You could double check that the doc you put to es, and the field named "fields" must be a timestamp type and the format should be "[MMM dd, yyyy, HH:mm:ss.SSS Z...." as the hint above. You got this error because some of the doc has a string type fields.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.