Reindexing Large index

[Aniket_Pant (kaal)]

Earlier we were using ELK stack version 7.11 when we upgrade to 7.13.2 we phase mapping conflict problem

although we didn't define mapping for event_id,thread_id,z_elastic_ecs.event.code

GET log-wlb-2021.07.02-0000010/_mapping
{
  "log-wlb-sysmon-2021.07.02-000010" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
  "event_id" : {
          "type" : "long"
        },
 "thread_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

The template we had for this index

{
  "winlogbeat_sysmon" : {
    "order" : 0,
    "index_patterns" : [
      "log-wlb-sysmon-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "winlogbeat_sysmon_policy",
          "rollover_alias" : "log-wlb-sysmon"
        },
        "refresh_interval" : "1s",
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "properties" : {
        "thread_id" : {
          "type" : "long"
        },
        "geoip" : {
          "type" : "object",
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        },
         "dst_ip_addr" : {
          "type" : "ip"
        }
      }
    },
    "aliases" : { }
  }
}

GET /_cat/indices/log-wlb-sysmon-*?v
green  open   log-wlb-sysmon-2021.06.09-000007 9CmsyVUGT26UwxID3LT8Xw   1   1   82308203        53179      100gb           50gb
green  open   log-wlb-sysmon-2021.05.31-000006 nL70VTUmQ7iuPsJusGHBsw   1   1   85460152        89372     99.9gb           50gb
green  open   log-wlb-sysmon-2021.07.02-000010 FIpdmZwcQdmSsGbwu6vYIA   1   1   60883104        79489     76.2gb         37.9gb
green  open   log-wlb-sysmon-2021.06.25-000009 HXwRnl8QRq6F_BCmztlBHw   1   1   78000899        38081      100gb           50gb
green  open   log-wlb-sysmon-2021.06.17-000008 VMIdvZhhQ7eDrtBrrk2t0w   1   1   81276018        46164      100gb         49.9gb

I don't why we have this conflict problem in kibana.The problem we phase during reindexing

• reindexing the 76gb or 100gb of data(log-wlb-sysmon-2021.07.02-000010) which take atleast 4 hours
• we have to shut down the logstash also so that no data cannot be write in that index\
• downtime is also 4 hours depends upon the size of index(in case of large index)

How i reindex the index is as follows

1. we check no of documents present in that index
2. Then we shutdown the logstash(no data cannot be write)
3. make the temp index having same setting and mapping properties
4. start the reindexing using reindex api

POST _reindex?max_docs=	60883104&wait_for_completion=false
{
  "conflicts": "proceed"
  , "source": {
    "index": "log-wlb-sysmon-2021.07.02-000010"
    },
   "dest": {
    "index": "log-test"
  }
}

The question is this i am doing right ? can i reduce the time of reindexing ? can it be happen without closing the logstash so that new data and old data is writing to that index

[ritchierich (Rich Horace)]

Hey @Aniket_Pant -

You can avoid stopping logstash if you changing the ILM policy to force a rollover to a new index with a new date then you would not have to stop logstash. Once data is being written to the new index you can start reindexing.

Here suggested steps:

1. Force new index by change ILM policy

2. Validate log-wlb-sysmon-2021.07.02-000010 no longer written to

3. Create new index for reindex log-wlb-sysmon-2021.07.02-000010-reindex

4. Valdate reindex successful.

    GET log-wlb-sysmon-2021.07.02-000010/_count
    GET log-wlb-sysmon-2021.07.02-000010-reindex/_count
   

5. Delete log-wlb-sysmon-2021.07.02-000010

6. Change ILM policy back to original setting

Hope this helps. Good luck.

Cheers,
Rich