After changing index template, new index not taking the changes

Elastic Stack Elasticsearch
1

Hi,

I am experiencing a weird problem where I am updating an index template by changing the mapping of a timestamp property from date to date_nano and even when in the definition of template it shows the proper mapping to date_nano, the new index which is created still showing date and with this the ingestion of data is failing.

I tried multiple things in the sandbox by removing the index, the template, etc, but still happening.

Seems some kind of caching issue, how can I solve it?

Thanks,

Zareh

2

One more comment, I adjusted the index template to have an extra property, just for testing and to validate if the new index get created with this property, it did not work, the mapping on the new index created still showing the same mapping as the original index template, before any change.

3

Welcome to our community! :smiley:

Sharing your template would be useful to diagnose this.

4

The original template:

    put _template/com.sap.c4c.abap.wf
    {
      "index_patterns": ["com.sap.c4c.abap.wf*"],
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "index.lifecycle.name": "com.sap.c4c.abap.wf",
        "index.lifecycle.rollover_alias": "com.sap.c4c.abap.wf"
      },
       "mappings": {
        "properties": {
          "beforeimageindicator":{"type" : "keyword"},
          "calculatedfieldindicator":{"type" : "keyword"},
          "client":{"type" : "keyword"},
          "customerid":{"type" : "keyword"},
          "customername":{"type" : "keyword"},
          "dc":{"type" : "keyword"},
          "description":{"type" : "keyword"},
          "fieldoractionname":{"type" : "keyword"},
          "maxconditionsingroup":{"type" : "long"},
          "moname":{"type" : "keyword"},
          "numberofgroups":{"type" : "long"},
          "system":{"type" : "keyword"},
          "tenant":{"type" : "keyword"},
          "tenantrole":{"type" : "keyword"},
          "timestamp":{"type": "date"},
          "timingtypecode":{"type" : "keyword"},
          "totalnoofconditions":{"type" : "long"},
          "typecode":{"type" : "keyword"}
        }
      },
      "version": 1
    }

The new template:

   put _template/com.sap.c4c.abap.wf
    {
      "index_patterns": ["com.sap.c4c.abap.wf*"],
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 1,
        "index.lifecycle.name": "com.sap.c4c.abap.wf",
        "index.lifecycle.rollover_alias": "com.sap.c4c.abap.wf"
      },
       "mappings": {
        "properties": {
          "beforeimageindicator":{"type" : "keyword"},
          "calculatedfieldindicator":{"type" : "keyword"},
          "client":{"type" : "keyword"},
          "customerid":{"type" : "keyword"},
          "customername":{"type" : "keyword"},
          "dc":{"type" : "keyword"},
          "description":{"type" : "keyword"},
          "fieldoractionname":{"type" : "keyword"},
          "maxconditionsingroup":{"type" : "long"},
          "moname":{"type" : "keyword"},
          "numberofgroups":{"type" : "long"},
          "system":{"type" : "keyword"},
          "tenant":{"type" : "keyword"},
          "tenantrole":{"type" : "keyword"},
          "timestamp":{"type": "date_nanos"},
          "timingtypecode":{"type" : "keyword"},
          "totalnoofconditions":{"type" : "long"},
          "typecode":{"type" : "keyword"}
        }
      },
      "version": 2
    }`

Changing the template worked in 1 elastic cluster, but now I am experiencing the same problem in 2 different elastic clusters.

Basically after updating the index template, I am running the rollover API to move to a new index and apply this settings:

   POST /com.sap.c4c.abap.wf/_rollover
    {
      "conditions": {
        "max_age":   "1d",
        "max_size":  "1gb"
      }
    }

But like I said, when I look into the mapping of the new index on hot phase, the mapping is from the old index template, and this is the problem, but when I see the index template, it has the proper mapping.

Thanks,

Zareh

5

What is the output from _cat/templates?v?

6

.ml-anomalies- [.ml-anomalies-] 0 7080099
.ml-meta [.ml-meta] 0 7080099
.watch-history-11 [.watcher-history-11] 2147483647 11
.ml-inference-000002 [.ml-inference-000002] 0 7080099
.transform-notifications-000002 [.transform-notifications-] 0 7080099
.ml-stats [.ml-stats-] 0 7080099
.slm-history [.slm-history-2*] 2147483647 2
.monitoring-kibana [.monitoring-kibana-7-] 0 7000199
.triggered_watches [.triggered_watches] 2147483647 11
com.sap.c4c.abap.sessions [com.sap.c4c.abap.sessions*] 0 2
.watches [.watches*] 2147483647 11
.monitoring-es [.monitoring-es-7-] 0 7000199
.monitoring-logstash [.monitoring-logstash-7-] 0 7000199
.ml-config [.ml-config] 0 7080099
.management-beats [.management-beats] 0 70000
.ml-state [.ml-state*] 0 7080099
.logstash-management [.logstash] 0
ilm-history [ilm-history-2*] 2147483647 2
.transform-internal-005 [.transform-internal-005] 0 7080099
.monitoring-beats [.monitoring-beats-7-] 0 7000199
.ml-notifications-000001 [.ml-notifications-000001] 0 7080099
.kibana-event-log-7.8.0-template [.kibana-event-log-7.8.0-] 0
.monitoring-alerts-7 [.monitoring-alerts-7] 0 7000199
com.sap.c4c.abap.meter [com.sap.c4c.abap.meter*] 0
com.sap.c4c.abap.oberon [com.sap.c4c.abap.oberon*] 0
com.sap.c4c.abap.api_calls [com.sap.c4c.abap.api_calls*] 0
com.sap.c4c.hana.backups [com.sap.c4c.hana.backups*] 0
com.sap.c4c.abap.bo [com.sap.c4c.abap.bo*] 0
com.sap.c4c.abap.wf [com.sap.c4c.abap.wf*] 0
com.sap.c4c.hana.hash [com.sap.c4c.hana.hash*] 0
com.sap.c4c.abap.sessions [com.sap.c4c.abap.sessions*] 0
com.sap.c4c.abap.logon [com.sap.c4c.abap.logon*] 0
com.sap.c4c.oberon.analysis [com.sap.c4c.oberon.analysis*] 0
com.sap.c4c.hana.size.system [com.sap.c4c.hana.size.system*] 0
com.sap.c4c.hana.tables [com.sap.c4c.hana.tables*] 0
com.sap.c4c.harmonization [com.sap.c4c.harmonization*] 0
com.sap.apm.performance.tracking [com.sap.apm.performance.tracking*] 0
com.sap.c4c.abap.bc [com.sap.c4c.abap.bc*] 0
com.sap.c4c.hana.size.tenant [com.sap.c4c.hana.size.tenant*] 0

7

Hi,

I am observing the same issue with a different index template com.sap.c4c.abap.sessions [com.sap.c4c.abap.sessions*], same type of issue, the new mapping defined on the template is not reflected in the index, now it is like this

New template:

PUT _template/com.sap.c4c.abap.sessions
{
  "index_patterns": ["com.sap.c4c.abap.sessions*"],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "com.sap.c4c.abap.sessions",
    "index.lifecycle.rollover_alias": "com.sap.c4c.abap.sessions"
  },
   "mappings": {
    "properties": {
      "act_program":{"type" : "keyword"},
      "application_info":{"type" : "keyword"},
      "application": {"type" : "keyword"},
      "category":{"type": "keyword"},
      "client":{"type" : "keyword"},
      "customerid":{"type" : "keyword"},
      "customername":{"type" : "keyword"},
      "dc":{"type" : "keyword"},
      "em_memory":{"type" : "long"},
      "heap_memory":{"type" : "long"},
      "request_time":{"type": "date_nanos"},
      "security_context_id":{"type" : "keyword"},
      "server_name":{"type" : "keyword"},
      "system":{"type" : "keyword"},
      "tenant":{"type" : "keyword"},
      "tenantrole":{"type" : "keyword"},
      "timestamp":{"type": "date_nanos"},
      "user_name":{"type" : "keyword"},
      "username_encrypted":{"type" : "keyword"},
      "system_description":{"type" : "keyword"}
    }
  },
  "version": 2
}

Mapping reflected on the index:

{
  "mappings": {
    "_doc": {
      "properties": {
        "act_program": {
          "type": "keyword"
        },
        "application": {
          "type": "keyword"
        },
        "application_info": {
          "type": "keyword"
        },
        "category": {
          "type": "keyword"
        },
        "client": {
          "type": "keyword"
        },
        "customerid": {
          "type": "keyword"
        },
        "customername": {
          "type": "keyword"
        },
        "dc": {
          "type": "keyword"
        },
        "em_memory": {
          "type": "long"
        },
        "heap_memory": {
          "type": "long"
        },
        **"request_time": {**
**          "type": "date",**
**          "format": "epoch_second"**
**        },**
        "security_context_id": {
          "type": "keyword"
        },
        "server_name": {
          "type": "keyword"
        },
        "system": {
          "type": "keyword"
        },
        "tenant": {
          "type": "keyword"
        },
        "tenantrole": {
          "type": "keyword"
        },
        "timestamp": {
          "type": "date_nanos"
        },
        "user_name": {
          "type": "keyword"
        }
      }
    }
  }
}

Basically I am changing the request_time from date to date_nanos and same problem.

Zareh

8

Please make sure you continue to format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Looking at the _cat output, it looks like you have multiple mentions of com.sap.c4c.abap.sessions*?

9

Thank you Mark,

At the end I realized that I got caught in the middle of the old and new index template APIs:

Old: /_template/template_1
New: /_index_template/template_1

Seems that I mixed them some way, at the end I was sure to use the new api and everything worked perfectly

Thanks all for your help,

Zareh

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.