# Elasticsearch reindex only appears to take a few documents **URL:** \ **Category:** Elasticsearch\ **Created:** [February 7, 2018, 8:23pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920 "2018-02-07T20:23:31Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 8:23pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/1 "2018-02-07T20:23:31Z")
Hello there, I am trying to reindex some data in index `firewall2` to index `firewall4`. I set up the firewall4 index and its mapping. When I try to run the reindex command, I see the following: ``` curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d' { "source": { "index": "firewall2" }, "dest": { "index": "firewall4" } } ' { "took" : 2, "timed_out" : false, "total" : 0, "updated" : 0, "created" : 0, "batches" : 0, "version_conflicts" : 0, "noops" : 0, "retries" : 0, "throttled_millis" : 0, "requests_per_second" : "unlimited", "throttled_until_millis" : 0, "failures" : [] } ``` There are more than 2 records in the original index; there are approximately 40k. This doesn't display any errors, so I would think things are ok, but when I try to make the index pattern (firewall4-\*) in Kibana, it says it cannot find data matching the index pattern. There was no data originally in the firewall4 index, so I did not care if anything existing would be deleted or not. Here are all the indices: ``` firewall/ firewall2-2018-02-07/ firewall4/ .kibana/ firewall2/ firewall3/ firewall-2018-02-07/ firewall3-2018-02-07/ ``` The "firewall" index had data as well, but I deleted that as part of a test. What I want to do now is reindex the data from firewall2 and firewall3 to firewall4, but I have no luck. Any assistance would be great 😃 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2018, 8:43pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/2 "2018-02-07T20:43:15Z")
What makes you think that the reindex operation is done? Did you check the running tasks? ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 9:02pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/3 "2018-02-07T21:02:21Z")
I am not seeing anything that looks like reindexing, unless this is what it's supposed to look like 😃 ``` curl -XGET localhost:9200/_tasks/?pretty { "nodes" : { "X24HtNOOSrGFp2KQQldJPw" : { "name" : "Scimitar", "transport_address" : "localhost:9300", "host" : "localhost", "ip" : "localhost:9300", "tasks" : { "X24HtNOOSrGFp2KQQldJPw:590" : { "node" : "X24HtNOOSrGFp2KQQldJPw", "id" : 590, "type" : "transport", "action" : "cluster:monitor/tasks/lists", "start_time_in_millis" : 1518036782414, "running_time_in_nanos" : 316439 }, "X24HtNOOSrGFp2KQQldJPw:591" : { "node" : "X24HtNOOSrGFp2KQQldJPw", "id" : 591, "type" : "direct", "action" : "cluster:monitor/tasks/lists[n]", "start_time_in_millis" : 1518036782414, "running_time_in_nanos" : 139752, "parent_task_id" : "X24HtNOOSrGFp2KQQldJPw:590" } } } } } ``` I started the reindex process a couple hours ago, not sure of the exact time. I also wasn't really sure if it started correctly as it seemed strange that the "took" field stated a number between 1 and 12 while the example from [https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html) shows 147. At any rate, I appreciate your time with this 😃 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2018, 9:55pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/4 "2018-02-07T21:55:48Z")
Could you run: ```auto GET firewall2/_search?size=0 GET firewall4/_search?size=0 ``` Also do you see anything in your logs? ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 9:58pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/5 "2018-02-07T21:58:58Z")
``` curl -XGET localhost:9200/firewall2/_search?size=0 {"took":1,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}} curl -XGET localhost:9200/firewall4/_search?size=0 {"took":1,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}} ``` ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 10:04pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/6 "2018-02-07T22:04:03Z")
There is also no entries in elasticsearch.log since I restarted elasticsearch earlier today. There are also no logs in depreciated or index\* logs. ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 10:27pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/7 "2018-02-07T22:27:09Z")
There are some logs that popped up just a few minutes ago. They are all saying basically the same thing, but for different indices; firewall2, 3, 4, etc and some other ftp indexes I created that don't ship the logs in real time. ``` [2018-02-07 17:10:43,512][DEBUG][action.fieldstats] [Scimitar] [firewall4][4], node[X24HtNOOSrGFp2KQQldJPw], [P], v[6], s[STARTED], a[id=PGDwElgYQvelwemlJcPxdQ]: failed to execute [org.elasticsearch.action.fieldstats.FieldStatsRequest@64190be9] RemoteTransportException[[Scimitar][localhost:9300][indices:data/read/field_stats[s]]]; nested: IllegalArgumentException[field [@timestamp] doesn't exist]; Caused by: java.lang.IllegalArgumentException: field [@timestamp] doesn't exist at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:166) at org.elasticsearch.action.fieldstats.TransportFieldStatsTransportAction.shardOperation(TransportFieldStatsTransportAction.java:54) at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:282) at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$ShardTransportHandler.messageReceived(TransportBroadcastAction.java:278) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:378) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2018, 10:33pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/8 "2018-02-07T22:33:25Z")
So if I sum up: You have no data in index `firewall2`. And you ran: ```auto curl -XPOST 'localhost:9200/_reindex?pretty' -H 'Content-Type: application/json' -d' { "source": { "index": "firewall2" }, "dest": { "index": "firewall4" } } ' ``` Which means that you copied "no data" from firewall2 to firewall4. What did you expect actually? ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 10:35pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/9 "2018-02-07T22:35:36Z")
I have data in firewall2. Here is the data from Kibana. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/b/5b4707f247b8c0bf6f17a6f3b9923ca30e63ec64.png) I have no data in firewall4. It was a blank index created plus the mapping. I am trying to practice reindexing data. That is why I am trying to get firewall2 and 3 to go to firewall4. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2018, 10:47pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/10 "2018-02-07T22:47:40Z")
> [@motts](#): > > I have data in firewall2. No you don't: ```auto curl -XGET localhost:9200/firewall2/_search?size=0 ``` gave ```auto {"took":1,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":0.0,"hits":[]}} ``` total is 0... No hits. I'm not sure what your kibana instance is connected to. ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 10:56pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/11 "2018-02-07T22:56:28Z")
Maybe this is me being new to all of this, but it looks like there is data in firewall2 index here ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/b/2b9303260975ff0c69e72b91ee37b46c8ee01b0c.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2018, 10:59pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/12 "2018-02-07T22:59:23Z")
So you have data in `firewall2-2018-02-07` index. But no data in `firewall2` index. ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 7, 2018, 11:07pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/13 "2018-02-07T23:07:54Z")
Ah ok. When I look in the /var/lib/elasticsearch...../indices, I seen the default logstash index. So I figured all the logstash-\* indices were in place because of the logstash index, but I guess this is not the case? Instead I should use the date-specific indices to get the data from and then create a new index called firewall4-(today) to put the data in correct? I was hoping I could use the index prefix as catch all to move all the firewall2-\* and firewall3-\* data to firewall4 hoping that it would create the date suffix when it completed. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 7, 2018, 11:23pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/14 "2018-02-07T23:23:57Z")
Don't look in ``` /var/lib/elasticsearch...../indices ``` It's internal. Just use the API. May be reindex from `firewall2-*`? ---
**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\ **Post date:** [February 8, 2018, 12:00am UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/15 "2018-02-08T00:00:34Z")
Alrighty. I was just thinking along the same lines as creating an index how I just use the name and the rest, like the -date, happens on it's own. Thank you for clearing that up. 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 8, 2018, 12:00am UTC](https://discuss.elastic.co/t/elasticsearch-reindex-only-appears-to-take-a-few-documents/118920/16 "2018-03-08T00:00:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.