Mapping use dynamic strict

Elastic Stack Elasticsearch
1

I am trying to test out this feature but not getting what I want

for example

PUT _index_template/sachin_quick_test
{
  "index_patterns": ["sachin_quick_test-*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
    
  "mappings" : {
      "dynamic": "strict",
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "day" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
}
}
}

Template gets created. and I can see it then when I put the data and day="string" it works

but it still does work when I put data=123 and explicitly convert to integer or not convert. it gets saved in elk as integer

Reason I know it works because if I change number_of_shards setting to 2 it does creates two shard that means it is using this template.

is my syntax is wrong for dynamic?

2

I did all sort of different combination but day field still allows text and integer both

3

Can you show the actual mapping after you have posted this?

4

I suspect you are seeing something similar to this

How the field is stored is different than how the field is represented in the source document I suspect you are looking at the source document and expecting it to change the type please read the response above and it may help clarify.

Elastic does not cast / correct the source document but it will convert if it can into the actual stored field /doc_value

So in your case if you set day=123 in the _source it looks like the integer you entered, but if you looked at the actual doc_value it would be a string.

And pretty much anything can be a string so it will almost always index a value as a string

5

@stephenb this is exactly what I have been seeing. But you are saying when I see value 123 without " around it, it is still a string correct?
if that is the case then what I have seen so far is wrong that string should be inside double quote and integer is without it. :slight_smile:

@warkolm here are detail. exactly what is happening

PUT _index_template/sachin_quick_test
{
  "index_patterns": ["sachin_quick_test-*"],
  "template": {
    "settings": {
      "number_of_shards": 1
    },
  "mappings" : {
    "dynamic": "strict",
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "job" : {
          "type" : "long"
        },
        "day" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Then I send two document

message => '{"job": 100, "day":123}'
message => '{"job": 101, "day":"Monday"}'


GET sachin_quick_test-2021/_search
"hits" : [
      {
        "_index" : "sachin_quick_test-2021",
        "_type" : "_doc",
        "_id" : "101",
        "_score" : 1.0,
        "_source" : {
          "job" : 101,
          "day" : "Monday",
          "@timestamp" : "2021-02-22T15:12:20.125Z"
        }
      },
      {
        "_index" : "sachin_quick_test-2021",
        "_type" : "_doc",
        "_id" : "100",
        "_score" : 1.0,
        "_source" : {
          "job" : 100,
          "day" : 123,
          "@timestamp" : "2021-02-22T15:12:51.359Z"
        }
      }
    ]
6

Yes... that is exactly what I am saying :slight_smile:

what you see in your query is the _source document, the Source what you put into elasticsearch, elasticsearch does not "correct" that. This is the subtlety that most people do not "grok" the source document is just that ... the source document, you can actually chose to not store it (not suggesting that unless you have a specific reason) ..... that data that is actually indexed and searched is stored in other data structures withing elasticsearch most notable doc_values.

To see the fields that are actually indexed and stored as fields that are actually searched try this date field will show up as text

GET sachin_quick_test-2021/_search
{
  "docvalue_fields": ["day"]
}

if you try this opposite and try to define a long and then try it is easier to see

DELETE my-index

PUT my-index

PUT my-index
{
  "mappings": {
    "properties": {
      "testId": {
        "type": "long"
      }
    }
  }
}

PUT my-index/_doc/1
{
  "testId": 1
}

PUT my-index/_doc/2
{
  "testId": "2"
}

# This will fail on indexing with a mapping exception
PUT my-index/_doc/3
{
  "testId": "three"
}

GET my-index/_search
{
  "docvalue_fields": ["testId"]
}

Not in the result the testId in the doc_value is actually a long

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-index",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "testId" : 1
        },
        "fields" : {
          "testId" : [
            1
          ]
        }
      },
      {
        "_index" : "my-index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "testId" : "2"
        },
        "fields" : {
          "testId" : [
            2 <--------- long even though the _source was "2"
          ]
        }
      }
    ]
  }
}
2 Likes
7

Ok I change day=long and I was not able to save Monday in to it.

But if I set day=string I am able to save Monday and 123 (even when I use mutate and convert that to integer inside logstash)
this is weird. but now I understand it hence I can use it the way I wanted it.

Basically I want to define my mapping strict for all field that way no one can put unwanted type in to defined and my pattern gets messed up where one field will have multiple type of data.

8

Thank you @stephenb

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.