Elasticsearch 7.2.1
Ingesting an XLSX sheet with 1x title and 1x data row using the excellent tool @GitHub - codingchili/excelastic: Vert.x web and commandline application to import CSV/XLS/XLSX files into ElasticSearch.
- Index created @ingestion time
- Dynamic mapping used (only used in dev phase in order to have an initial mapping to get started with for the final product)
- XLSX sheet contains 347 fields, so expecting around 694 fields excl system generated fields.
Getting the following error message:
[2020-03-02T00:50:51,053][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [node1] failed to put mappings on indices [[[qtitle2/Q0vYr8G7RfGmsu8NxtT3eA]]], type [default]
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [qtitle2] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:605) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:508) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:402) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:335) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:315) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:238) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.2.1.jar:7.2.1]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.2.1.jar:7.2.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_221]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_221]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_221]
[2020-03-02T00:50:51,063][INFO ][o.e.c.r.a.AllocationService] [node1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[qtitle2][0]] ...]).
Sample of my data as csv (which shows the same behaviour by the way). I have a no. of multi-fields as you can see, which are automatically generated from e.g. "q1.title".
q1.title,q2.title,q3.title,q4
What is your name?,How old are you?,Where do you currently live?,Can you read and write?
, the resulting dynamic mapping:
"q1" : {
"properties" : {
"title" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
I increased the max no of fields to 2000 as follows for troubleshooting:
DELETE qtitle2
PUT qtitle2
{
"settings":{
"index.mapping.total_fields.limit": 2000
}
}
Now, ingestion works, however the no of fields reported: 699, actually what was expected as shown above.
Any idea what might go wrong when using default setting?
(Have looked around on the forum, but found nothing addressing this particular issue).