Elasticsearch Create Wrong time based Index

Elastic Stack Elasticsearch
1

Using ELK 7.11
we have 11 index out of these we have one index called log-wlb-sysmon.It created its first index on 2021.04.06(log-wlb-sysmon-2021.04.06-000001) , we are using ILM policy for all the indexes
It created the another index on 19th April (log-wlb-sysmon-2021.04.19-000002) because we have set "max_age": "30d", "max_size": "50gb" , after some period of time we faced some mapping problem in this index so we decided to delete our old index(log-wlb-sysmon-2021.04.19-000002) and recreate this index so we did this

PUT log-wlb-sysmon-2021.04.26-000002
{
 "aliases": {
   "log-wlb-sysmon": {
      "is_write_index": true 
   }
 }
}

from then it was working fine both the indices(log-wlb-sysmon-2021.04.06-000001 and (log-wlb-sysmon-2021.04.26-000002)

Now we've notice that it created index log-wlb-sysmon-2021.04.26-000003 on 7th april , it should create index like log-wlb-sysmon-2021.05.07-000003

[2021-05-07T16:30:36,725][INFO ][o.e.c.m.MetadataCreateIndexService] [em2] [log-wlb-sysmon-2021.04.26-000003] creating index, cause [rollover_index], templates [winlogbeat_sysmon], shards [1]/[1]
[2021-05-07T16:30:36,767][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-wlb-sysmon-2021.04.26-000003] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [winlogbeat_sysmon_policy]
[2021-05-07T16:30:36,845][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-wlb-sysmon-2021.04.26-000002] from [{"phase":"hot","action":"rollover","name":"attempt-rollover"}] to [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] in policy [winlogbeat_sysmon_policy]
[2021-05-07T16:30:36,880][INFO ][o.e.c.m.MetadataMappingService] [em2] [log-wlb-sysmon-2021.04.26-000003/QSzhCbMPRQ2Yynils14EFA] update_mapping [_doc]
[2021-05-07T16:30:36,920][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-wlb-sysmon-2021.04.26-000003] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"set_priority","name":"set_priority"}] in policy [winlogbeat_sysmon_policy]
[2021-05-07T16:30:36,989][INFO ][o.e.c.m.MetadataMappingService] [em2] [log-wlb-sysmon-2021.04.26-000003/QSzhCbMPRQ2Yynils14EFA] update_mapping [_doc]
[2021-05-07T16:30:36,991][INFO ][o.e.c.m.MetadataMappingService] [em2] [log-wlb-sysmon-2021.04.26-000003/QSzhCbMPRQ2Yynils14EFA] update_mapping [_doc]
[2021-05-07T16:30:37,032][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-wlb-sysmon-2021.04.26-000002] from [{"phase":"hot","action":"rollover","name":"wait-for-active-shards"}] to [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] in policy [winlogbeat_sysmon_policy]
[2021-05-07T16:30:37,033][INFO ][o.e.x.i.IndexLifecycleTransition] [em2] moving index [log-wlb-sysmon-2021.04.26-000002] from [{"phase":"hot","action":"rollover","name":"update-rollover-lifecycle-date"}] to [{"phase":"hot","action":"rollover","name":"set-indexing-complete"}] in policy [winlogbeat_sysmon_policy]
[2021-05-07T16:30:37,100][INFO ][o.e.c.m.MetadataMappingService] [em2] [log-wlb-sysmon-2021.04.26-000003/QSzhCbMPRQ2Yynils14EFA] update_mapping [_doc]

log-wlb-sysmon index detail

health status index                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   log-wlb-sysmon-2021.04.26-000002 p5ROhVjwT0Gpl3pgwu2bRA   1   1   84829663        79510      100gb           50gb
green  open   log-wlb-sysmon-2021.04.06-000001 IAHprh30T6-rh2hxB67TLA   1   1   84049138        52926      100gb           50gb
green  open   log-wlb-sysmon-2021.04.26-000003 QSzhCbMPRQ2Yynils14EFA   1   1   39651126        67857     46.6gb         23.3gb

logstash pipeline file for this index

 elasticsearch {
      hosts => ["https://xx.xx.xx.xx:9200","https://xx.xx.xx.xx:9200"]
      cacert => '/etc/logstash/certs/ca/ca.crt'
      user => "logstash_user"
      password => "password@xxx"
      document_id => "%{[@metadata][log_hash]}"
      manage_template => false
      ilm_rollover_alias => "log-wlb-sysmon"
      ilm_pattern => "{now/d}-000001"
      ilm_enabled => "true"
      ilm_policy => "winlogbeat_sysmon_policy"
2

That's not how ILM works, the timestamp in the index name is the date that the ILM policy created its first index.

3

I don't know i am right or wrong because i am using the ilm concept from past 1 month,i have other indices also apart from log-wlb-sysmon

health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   log-pb-flow-2021.04.05-000001 _HTx3o3iTEW25cP-anki-A   1   1   37767666            0     39.2gb         19.6gb
green  open   log-pb-tls-2021.05.05-000002  wiWn9yhrQ8yHwbV_zSLmsw   1   1      86550            0    361.2mb        179.5mb
green  open   log-pb-dns-2021.04.05-000001  y51OL9YBTBiZ4bFY-Kdupg   1   1     236493            0      455mb        226.7mb
green  open   log-pb-http-2021.05.05-000002 6PcQu2Q6Q0q5-yqotcGD6Q   1   1      18021            0     42.1mb         21.3mb
green  open   log-pb-tls-2021.04.05-000001  ZGu9rPQiT765hT_L0wun5g   1   1     254845            0    969.4mb        489.4mb
green  open   log-pb-icmp-2021.04.05-000001 NwzA1kw-TFm37IRcKknAGQ   1   1     437700            0    337.2mb        168.5mb
green  open   log-pb-icmp-2021.05.05-000002 liHOxFn6T5GMojxEunTNRA   1   1      84208            0     94.5mb         59.1mb
green  open   log-pb-http-2021.04.05-000001 OjapOmTCQa2k4xxgzA3-4w   1   1     159927            0    192.3mb         96.2mb
green  open   log-pb-flow-2021.05.05-000002 CkJzzRBxQ1m1BDD3Rkqcjg   1   1    9342075            0      9.7gb          4.8gb
green  open   log-pb-dns-2021.05.05-000002  AA7i7xrdSsCfA73gBJncSA   1   1      64963            0    144.6mb           72mb

You can see that month is changing but in log-wlb-sysmon index case it is was not changing, it created the new index but with same date .

4

Thanks for using Elasticsearch and ILM.

Date math on rollover is based on a hidden index setting called index.provided_name.

When you executed

PUT log-wlb-sysmon-2021.04.26-000002
{
 "aliases": {
   "log-wlb-sysmon": {
      "is_write_index": true 
   }
 }
}

the index was created as a regular index that has a date in its name, but date math was not configured.

Unfortunately there is no way to enable date math for an index after it's been created (index.provided_name is a private, internal setting). However you can create a new index with date math configured and then switch the is_write_index alias config from log-wlb-sysmon-2021.04.26-000002 to the new index you created.

eg (please adapt the indices names to match your current state)

// this will create log-wlb-sysmon-2021.04.26-000001
PUT %3Clog-wlb-sysmon-%7Bnow%2Fd%7D-000001%3E 
{
"aliases": {
   "log-wlb-sysmon": {
      "is_write_index": false      // keep it false for now as log-wlb-sysmon-2021.04.26-000002 is still the write_index for this alias
   }
 }
}

// swap the write alias from the previous index to the newly created one
POST _aliases
{
  "actions": [
    {
      "add": {
        "index": "log-wlb-sysmon-2021.04.26-000002",
        "alias": "log-wlb-sysmon",
        "is_write_index": false
      }
    }, {
      "add": {
        "index": "log-wlb-sysmon-2021.05.13-000001",
        "alias": "log-wlb-sysmon",
        "is_write_index": true
      }
    }
  ]
}

You can verify the new index has date math configured by issuing a
GET log-wlb-sysmon-2021.04.26-000001/_settings and looking for the setting

"index": {
   ... 
        "provided_name": "<log-wlb-sysmon-{now/d}-000001>",
   ...
}
5

Hey @andreidan i understand your point but i want to know that in ilm policy we have "max_age": "30d", "max_size": "50gb" for log-wlb-sysmon index , it will create new index after 20 days because in 20 days it has 100gb of data(replica including) my question is will it create index on time based or this log-wlb-sysmon-2021.04.26-000003 if it create index like this
Then what we have to do to change this permanently so that in future we don't have to deal with this

6

log-wlb-sysmon-2021.04.26-000002 doesn't have date math configured at the moment so on the next rollover operation index log-wlb-sysmon-2021.04.26-000003 will be created.

In order for date math to be enabled (and for rollover to update the date in the index name) you'll have to create a new index with date math enabled (with ILM configured for it, presumbaly by an index template? so no extra action should be required for this) and shift the alias with is_write_index to the newly created index as described in the previous comment

7

I will test this

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.