Json_parse_exception on reindex

Elastic Stack Elasticsearch
1

I'm getting json_parse_exception on reindex from a bunch of daily indices to a consolidated monthly one, quite likely due to the same field in different daily indices being automapped to different types.

How do I tell reindex to ignore these errors - drop the erroneous fields, drop the records that contain them, I don't care really, it's just that I do want the other 99.99% of the data?

{
  "took": 6943,
  "timed_out": false,
  "total": 16776617,
  "updated": 6999,
  "created": 0,
  "deleted": 0,
  "batches": 7,
  "version_conflicts": 0,
  "noops": 0,
  "retries": {
    "bulk": 0,
    "search": 0
  },
  "throttled_millis": 0,
  "requests_per_second": -1,
  "throttled_until_millis": 0,
  "failures": [
    {
      "index": "filebeat-2018.01",
      "type": "doc",
      "id": "AWEDf0pgq8giAw7TVsri",
      "cause": {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse [json.provision.assets.asset-instance.virtual]",
        "caused_by": {
          "type": "json_parse_exception",
          "reason": "Current token (VALUE_TRUE) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@5b323cb5; line: 1, column: 596]"
        }
      },
      "status": 400
    }
  ]
}
2

can you compare that document with any other file beat doc, that also has a json.provision.assets.asset-instance.virtual field? Is it possible that the structure from this single document is different? Also, did you compare that both indices, the one you indexed from and the one you indexed to have the same mapping?

3

I haven't done any manual investigation of that particular field - that's exactly the sort of thing I don't want to have to spend time doing, particularly as I have many gigabytes of these logs in a year's worth of daily indices and heaven knows how many instances of such things. I just want to reindex all the records that don't have problems!

4

You can use conflicts: proceed to ignore errors, see https://www.elastic.co/guide/en/elasticsearch/reference/6.4/docs-reindex.html

5

Yes, I tried that, it doesn't work (it apparently ignores version conflict errors but not the errors I'm getting).

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.