[Resolved]Interger field defined in index template is treated as string, it looks like the interger mapping doesn't take effect

[cheriemilk]

Hi Team,

I defined 4 integer fields in index template(if1,if2,if3 and if4), and ship serveral events to Elasticsearch by logstash. I found that these 4 interger fields are displayed as 'string' type in Kibana, which is unexpected.
Please help take a look and advice how to correct this issue.

1. Index template

  {
  "template": "kvaudit",
  "index_patterns": ["kvaudit*"],
  "settings": {
  "index": {
       "number_of_shards": "1",
       "codec": "best_compression",
       "number_of_replicas": "0"
          }
      },
"mappings": {
   "doc": {
   "properties": {
   "@version": {
   "type": "keyword" }
}
   }
     },

"beat": { 
	"properties": {
	"version": {
"type": "keyword"
    }
      }
        },

"fields": {
"properties": { "at": {"type": "keyword"},
	              "ktf1": {"type": "keyword"},
                "kf1": {"type": "keyword"},
                "kf2": {"type": "keyword"},
                "kf3": {"type": "keyword"},
                "if1": { "type": "integer"},
                "if2": { "type": "integer"},
                "if3": { "type": "integer"},
                "if4": { "type": "integer"}}}
}

2. logstash.conf

input { file{ path => "C:/elkstack/elasticsearch-7.0.1-windows-x86_64/data/integertest.csv"		
              start_position => "beginning"
              mode => "tail"
			  sincedb_path => "C:/elkstack/elasticsearch-7.0.1-windows-x86_64/sincedb/sincedb.txt" }
	  }

filter { 
          csv { columns => [ "at",
                             "ktf1",
                             "kf1",
                             "kf2",
                             "kf3",
                             "if1",
                             "if2",
                             "if3",
                             "if4"]
              separator => ","
              skip_header => "true"
			   } 


ruby {
      code => "
        hash = event.to_hash
        hash.each do  |k, v|
          if(v != nil && v.kind_of?(String) && v.length > 2 && v[0,1] == '[' && v[v.length-1,1] == ']')
            event.set(k, v[1, v.length-2].split(','))
                end
                end
         "
  }
}



output {
    elasticsearch {
	   action => "index"
	   hosts  => "localhost:9200"
	   index  => "kvaudit"
     manage_template => true
     template => "C:/elkstack/elasticsearch-7.0.1-windows-x86_64/mapping/kvaudit2.json"
     template_name=> "kvaudit2.json"
     template_overwrite => true }

	stdout { codec => rubydebug {metadata => true}}
}

3. the data shipped to ES

at,ktf1,kf1,kf2,kf3,if1,if2,if3,if4
SAVE,performance,perf,,perf_guideline,1,2,3,4
SAVE,"[performance,liveprofile,talentflag]","[obj,comp,sysoverallperformance,sysoverallpotential]","[obj,comp]","[sysoverallperformance,sysoverallpotential]",2,3,4,5
SAVE,"[performance,liveproifle]","[obj,sysoverallperformance,sysoverallpotential]",obj,"[obj,sysoverallperformance,sysoverallpotential]",1,2,3,4

4. But in kibana, the data type for 4 fields if1, if2, if3 and if4 are string, which is not consittent with what I defined in index template.
   
   

Anyone can take a look?

[Ignacio_Vera]

Hi,

Could you share the version you are using? This is an importation information in order to help. In addition, could you please revise the index template you have posted here, something looks off in that template. If would we better if you can share exactly the same command you are executing.

[cheriemilk]

Hi Ignacio_Vera,
I am using elasticsearch-7.0.1-windows-x86_64, logstash-7.0.1 and kibana-7.0.1-windows-x86_64.

[Ignacio_Vera]

Hi,

I think the template you shared with us is wrong. I built a lighter version:

{
  "template": "kvaudit",
  "index_patterns": [
    "kvaudit*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "1",
      "codec": "best_compression",
      "number_of_replicas": "0"
    }
  },
  "mappings": {
    "properties": {
      "at": {
        "type": "keyword"
      },
      "ktf1": {
        "type": "keyword"
      },
      "kf1": {
        "type": "keyword"
      },
      "kf2": {
        "type": "keyword"
      },
      "kf3": {
        "type": "keyword"
      },
      "if1": {
        "type": "integer"
      },
      "if2": {
        "type": "integer"
      },
      "if3": {
        "type": "integer"
      },
      "if4": {
        "type": "integer"
      }
    }
  }
}

And in addition I might need to set to false this option in your Logstash configuration: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm_enabled

[cheriemilk]

Hi Ignacio_Vera,
I applied your template and add this 'ilm_enabled => false' in logstash output plug-in, but the 4 integer fields are still displayed as string .

Please let me know if there're other information you need to know.

Thanks,
Cherie

[Ignacio_Vera]

Can you share the output from the following API end-point:

GET  _template/kvaudit

That should tell us if the template was created.

[cheriemilk]

Hi Ignacio_Vera,

1. returned nothing when sending request GET _template/kvaudit

2. I tried GET _template/kvaudit*, and it returned below template to me, the template looks like inserted as expected. Does the issue happens on mapping? as when I created the index named kvaudit*, it shows me that if1, if2, if3, if4 are all string. Please refer to below screenshot.

   {
   "kvaudit.json" : {
   "order" : 0,
   "index_patterns" : [
   "kvaudit*"
   ],
   "settings" : {
   "index" : {
   "number_of_shards" : "1",
   "codec" : "best_compression",
   "number_of_replicas" : "0"
   }
   },
   "mappings" : {
   "properties" : {
   "at" : {
   "type" : "keyword"
   },
   "if2" : {
   "type" : "integer"
   },
   "if1" : {
   "type" : "integer"
   },
   "ktf1" : {
   "type" : "keyword"
   },
   "kf2" : {
   "type" : "keyword"
   },
   "if4" : {
   "type" : "integer"
   },
   "kf1" : {
   "type" : "keyword"
   },
   "if3" : {
   "type" : "integer"
   },
   "kf3" : {
   "type" : "keyword"
   }
   }
   },
   "aliases" : { }
   }
   }

[Ignacio_Vera]

Have you deleted the index before ingesting the data gain? Once the mapping is created it cannot be changed so you need to delete it so it gets recreated again. I would recommend to delete the templates as well so you start from a clean cluster.

[cheriemilk]

Yes. I deleted both the index template and index and send this request DELETE /_template/kvaudit* to make sure the template are deleted.

[cheriemilk]

Thanks. I do it again with , and now is correct.

[cheriemilk]

Hi Ignacio_Vera,

One question. what's the difference by deleting the index in index management and index pattern?

Thanks,
Cherie

[Ignacio_Vera]

Not an expert in that area but I think index pattern is something specific to Kibana that allows you to group indices so they can be query together. Deleting an index pattern I believe it has no effect on the index in elasticsearch.

With index management you are actually deleting the indices in Elasticsearch.