# Reindexing Large index

**URL:** <https://discuss.elastic.co/t/reindexing-large-index/278313>\
**Category:** Elasticsearch\
**Tags:** reindex\
**Created:** [July 10, 2021, 5:20am UTC](https://discuss.elastic.co/t/reindexing-large-index/278313 "2021-07-10T05:20:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\
**Post date:** [July 10, 2021, 5:20am UTC](https://discuss.elastic.co/t/reindexing-large-index/278313/1 "2021-07-10T05:20:22Z")

</div>

Earlier we were using ELK stack version 7.11 when we upgrade to 7.13.2 we phase mapping conflict problem

 ![Screenshot 2021-07-09 220136](https://us1.discourse-cdn.com/elastic/original/3X/4/9/49f4d16b1e886e3f6abf87f0d5fadd5ac6eb8018.jpeg)  
although we didn't define mapping for `event_id,thread_id,z_elastic_ecs.event.code`

```auto
GET log-wlb-2021.07.02-0000010/_mapping
{
  "log-wlb-sysmon-2021.07.02-000010" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
  "event_id" : {
          "type" : "long"
        },
 "thread_id" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

```

The template we had for this index

```auto
{
  "winlogbeat_sysmon" : {
    "order" : 0,
    "index_patterns" : [
      "log-wlb-sysmon-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "winlogbeat_sysmon_policy",
          "rollover_alias" : "log-wlb-sysmon"
        },
        "refresh_interval" : "1s",
        "number_of_shards" : "1",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : {
      "properties" : {
        "thread_id" : {
          "type" : "long"
        },
        "geoip" : {
          "type" : "object",
          "properties" : {
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            }
          }
        },
         "dst_ip_addr" : {
          "type" : "ip"
        }
      }
    },
    "aliases" : { }
  }
}

```

```auto
GET /_cat/indices/log-wlb-sysmon-*?v
green open log-wlb-sysmon-2021.06.09-000007 9CmsyVUGT26UwxID3LT8Xw 1 1 82308203 53179 100gb 50gb
green open log-wlb-sysmon-2021.05.31-000006 nL70VTUmQ7iuPsJusGHBsw 1 1 85460152 89372 99.9gb 50gb
green open log-wlb-sysmon-2021.07.02-000010 FIpdmZwcQdmSsGbwu6vYIA 1 1 60883104 79489 76.2gb 37.9gb
green open log-wlb-sysmon-2021.06.25-000009 HXwRnl8QRq6F_BCmztlBHw 1 1 78000899 38081 100gb 50gb
green open log-wlb-sysmon-2021.06.17-000008 VMIdvZhhQ7eDrtBrrk2t0w 1 1 81276018 46164 100gb 49.9gb

```

I don't why we have this **conflict** problem in kibana.The problem we phase during reindexing

- reindexing the **76gb or 100gb of data(log-wlb-sysmon-2021.07.02-000010)** which take atleast 4 hours
- we have to shut down the logstash also so that no data cannot be write in that index\
- downtime is also 4 hours depends upon the size of index(in case of large index)

**How i reindex the index is as follows**

1. we check no of documents present in that index
2. Then we shutdown the logstash(no data cannot be write)
3. make the temp index having same setting and mapping properties
4. start the reindexing using reindex api

```auto
POST _reindex?max_docs=	60883104&wait_for_completion=false
{
  "conflicts": "proceed"
  , "source": {
    "index": "log-wlb-sysmon-2021.07.02-000010"
    },
   "dest": {
    "index": "log-test"
  }
}

```

The question is this i am doing right ? can i reduce the time of reindexing ? can it be happen without closing the logstash so that new data and old data is writing to that index

---

<div class="post-metadata">

**Author:** ![ritchierich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ritchierich/32/4329_2.png) [@ritchierich](https://discuss.elastic.co/u/ritchierich)\
**Post date:** [July 12, 2021, 6:40am UTC](https://discuss.elastic.co/t/reindexing-large-index/278313/2 "2021-07-12T06:40:48Z")

</div>

Hey @Aniket_Pant -

You can avoid stopping logstash if you changing the ILM policy to force a rollover to a new index with a new date then you would not have to stop logstash. Once data is being written to the new index you can start reindexing.

**Here suggested steps:**

1. Force new index by change ILM policy

2. Validate `log-wlb-sysmon-2021.07.02-000010` no longer written to

3. Create new index for reindex `log-wlb-sysmon-2021.07.02-000010-reindex`

4. Valdate reindex successful.

5. Delete log-wlb-sysmon-2021.07.02-000010

6. Change ILM policy back to original setting

Hope this helps. Good luck.

Cheers,  
Rich

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 9, 2021, 6:41am UTC](https://discuss.elastic.co/t/reindexing-large-index/278313/3 "2021-08-09T06:41:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.