Dynamic template in very dynamic document

Elastic Stack Elasticsearch
1

I have SQL queries that would be indexed into ElasticSearch.
The problem I encounter is that the fields in documents are very dynamic, it could be text, object, integer, etc.

Sample of body (Note that this is not the same for others)
As you see, the values on key 0, 1 ,2 is text/integer but this is not the case for another document.
How do I force it to be text? For example, on indexing time, convert the value of key 1 to text

"bindings": {
    "0": "41615",
    "1": 42,
    "2": "2"
}

Created template with Dynamic Template

PUT _template/cwl_template
{
  "order": 0,
  "index_patterns": ["cwl*"],
  "mappings": {
    "dynamic_templates": [
      {
        "bindings": {
          "path_match": "bindings.*",
          "match_pattern": "regex",
          "match": "^.*$",
          "mapping": {
            "type": "text",
            "fields": {
              "keyword": {
                "ignore_above": 256,
                "type": "keyword"
              }
            }
          }
        }
      }
    ]
  }
}

Receiving error on logstash

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cwl-2020.05.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x646f085a>], :response=>{"index"=>{"_index"=>"cwl-2020.05.22", "_type"=>"_doc", "_id"=>"zG0mUnIBuw9XhmjOhULr", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [properties.bindings.5] of different type, current_type [text], merged_type [long]"}}}}
2

My regex is not the best, and I may be misreading the error, but don't you want to match_pattern on properties.bindings.*?

1 Like
3

You're right, I need more sleep.
Been days trying to get data into ES due to the very dynamic documents structure

1 Like
4

Happens to all of us :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.