Is there a limit on the number of fields specified in the "include" param in Reindex call

Elastic Stack Elasticsearch
1

Hi,
This is the error message I am facing while executing the Reindex call

[2020-03-13T16:08:39,042][WARN ][o.e.i.r.TransportReindexAction] [ExportDatabase] giving up on search because it failed with a non-retryable exception
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:288) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:128) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:249) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:527) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1188) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1172) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:385) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_212]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_212]
Caused by: org.elasticsearch.ElasticsearchException$1: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states.
at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:658) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:126) ~[elasticsearch-6.3.0.jar:6.3.0]
... 26 more
Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton with 33501 states and 34977 transitions would result in more than 10000 states.
at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:746) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.RunAutomaton.(RunAutomaton.java:69) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:39) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.automaton.CharacterRunAutomaton.(CharacterRunAutomaton.java:28) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.elasticsearch.common.xcontent.support.XContentMapValues.filter(XContentMapValues.java:174) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.fetch.subphase.FetchSourceContext.getFilter(FetchSourceContext.java:234) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.lookup.SourceLookup.filter(SourceLookup.java:132) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.fetch.subphase.FetchSourceSubPhase.hitExecute(FetchSourceSubPhase.java:56) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:162) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:393) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:368) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0]
... 9 more

So as per my analysis this is the query which I am executing:

POST _reindex
{
"source":{
"index":"index_source",
"size":1000,
"query":
{
"terms":{
"ID":["7078"],
"boost":1.0
}
},
"_source":
{
"includes":here we have 1216 fields]
}
},
"dest":
{
"index":"index_dest"
}
}

Here if I remove the 1216 fields in the include param or make it less to around 500 then it works

So need help in this, is there any other way we can only reindex for these specific fields or is there something wrong I am doing.

2

You might be able to reindex through an ingest pipeline that deletes the fields you no longer want to keep but I guess it depends on the total field count.

3

Thanks Christain for this prompt reply, but currently I have around 1300 fields in this call, I just want to know that is this a limitation from the elastic side or I am doing something wrong in this? because I can't sent these attributes in batches, need to think of another way then. I will try that through ingest pipeline as well.

4

Hello,
After analysis I found the max terms which I can specify in the include parameter of the Reindex call is 960. More than that it is not taking. If I reduced the no. of fields to 960 then it works. But not sure why it is that way.

5

Hi @Harshraj_Shinde.

max 1000 fields (default)

yes ,there is default limitation for that but you can break that limitation by using below link-

Thanks
HadoopHelp

6

Hello @rameshkr1994,
Thanks for your reply, but already increased the total fields limit on the index level.
Like currently my
Source index:

"Source_index": {
"settings": {
"index": {
"mapping": {
"total_fields": {
"limit": "4636"
}
}

Destination Index:

 "Dest_index": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "4000"
          }
        }

And I am passing 1216 fields in the reindexing call.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.