# Help reindexing to new index **URL:** \ **Category:** Elasticsearch\ **Tags:** reindex\ **Created:** [December 31, 2020, 3:38am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932 "2020-12-31T03:38:56Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 3:38am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/1 "2020-12-31T03:38:56Z")
I have a logstash .conf file whos outputs originally look like this: ```auto if "PAN-OS_config" in [tags] { elasticsearch { index => "config" hosts => ["localhost:9200"] } ``` This created indexes in Elasticsearch such as "config", "traffic", "threat", etc. Since I want to setup log rotation and I need (and I read it's best practice) to use have indexes with %{+YYYY.MM} appended to them. I've changed the .conf and updated the index to: ```auto index => "log-panos-config-%{+YYYY.MM}" ``` However, when I try to reindex the old index to the new, I get the following error: ```auto { "index" : "logs-panos-config-%{+YYYY.MM}", "type" : "_doc", "id" : "CTqktnYBBWs-cule_Kv7", "cause" : { "type" : "illegal_argument_exception", "reason" : "data_stream [logs-panos-config-%{+YYYY.MM}] must be lowercase" }, "status" : 400 } ``` Here is what I'm running to reindex: ```auto curl -u user:pass -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d' { "source": { "index": "panos-config" }, "dest": { "index": "logs-panos-config-%{+YYYY.MM}" } } ' ``` I'm assuming the "must be lowercase" is in reference to the "YYYY.MM" but I'm not sure what to do to get the date appended. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 31, 2020, 4:16am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/2 "2020-12-31T04:16:43Z")
> [@jtillman2020](#): > > Since I want to setup log rotation and I need (and I read it's best practice) to use have indexes with %{+YYYY.MM} appended to them. I've changed the .conf and updated the index to: Why not use [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-start.html)? 🙂 ---
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 4:18am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/3 "2020-12-31T04:18:34Z")
I'm looking around ILM now...getting a major headache lol. So am I trying to take this single index and convert it into a data stream? ---
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 4:23am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/4 "2020-12-31T04:23:00Z")
Not sure if this is correct but I went into "Manage" for the "config" index and added a lifecycle policy I created. It gave me a message about no alias existing but it seemed to add the policy anyways? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 31, 2020, 4:23am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/5 "2020-12-31T04:23:54Z")
The date expression you have there isn't converted in a query automatically. Logstash will do it for you, but you need to be explicit when talking to Elasticsearch. ---
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 4:25am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/6 "2020-12-31T04:25:18Z")
I changed it from the expression to "2020.12" and then received these errors: ```auto { "index" : "logs-panos-config-2020.12", "type" : "_doc", "id" : "CTqktnYBBWs-cule_Kv7", "cause" : { "type" : "illegal_argument_exception", "reason" : "only write ops with an op_type of create are allowed in data streams" }, "status" : 400 } ``` ---
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 4:32am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/7 "2020-12-31T04:32:46Z")
I may be over complicating this...Basically I changed the output index on my logstash .conf file so now new logs are coming to the new index. Now I need a way to get the logs in the old index into the new one and have my visualizations point to the new index instead of the old one. Lol and I have no clue how. ---
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 4:42am UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/8 "2020-12-31T04:42:14Z")
I take that back. New logs are not coming into the new index (which is actually a data stream now). The logstash log is showing the same "only write ops with an op\_type of create are allowed in data streams" error and saying it couldn't index event to Elasticsearch. ---
**Author:** ![jtillman2020](https://avatars.discourse-cdn.com/v4/letter/j/85f322/32.png) [@jtillman2020](https://discuss.elastic.co/u/jtillman2020)\ **Post date:** [December 31, 2020, 4:25pm UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/9 "2020-12-31T16:25:25Z")
I've solved my reindex issue. I was able to successfully change the index, reindex the old into the new, and delete the old. I then changed the UUID for my visualizations to point to the new index UUID. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 31, 2020, 9:48pm UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/10 "2020-12-31T21:48:54Z")
Great to hear! Can you share what solved it, as it might help someone in future? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 28, 2021, 9:49pm UTC](https://discuss.elastic.co/t/help-reindexing-to-new-index/259932/11 "2021-01-28T21:49:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.