Hi Guys,
I am a newbie to ElasticSearch , I am running into errors when I try to disable dynamic mapping in ElasticSearch settings. I am using ElasticSearch 1.7 version for implementation.
StackTrace :
8151 [main] WARN org.apache.hadoop.mapred.YarnChild - Exception running child : org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [10.74.51.71:9200] returned Not Found(404) - [TypeMissingException[[test_2017051222] type[[vehicle, trying to auto create mapping, but dynamic mapping is disabled]] missing]]; Bailing out..
at org.elasticsearch.hadoop.rest.RestClient.retryFailedEntries(RestClient.java:207)
at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:170)
at org.elasticsearch.hadoop.rest.RestRepository.tryFlush(RestRepository.java:225)
at org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:248)
at org.elasticsearch.hadoop.rest.RestRepository.doWriteToIndex(RestRepository.java:187)
at org.elasticsearch.hadoop.rest.RestRepository.writeToIndex(RestRepository.java:163)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:151)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:566)
at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)
at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:150)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:635)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:390)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
I tried this in two different ways :
First via the Settings :
"settings": {
"number_of_shards": 5,
"number_of_replicas": 1,
"index.query.default_field":"test",
"index.refresh_interval" : "5s",
"index.mapper.dynamic": false ,
"analysis": {
"filter": {
"ngram_filter": {
"type": "ngram",
"min_gram": 2,
"max_gram": 18,
"token_chars": [
"letter",
"digit"
]
}
},
"analyzer": {
"ngram_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"ngram_filter"
]
}
}
}
}
Second via the mapping :
"mappings": {
"testing": {
"dynamic" : "false" ,
"properties": {
"field1": {
"type": "string",
"index": "analyzed"
},
"field2": {
"type": "string",
"index": "not_analyzed"
},
"field3": {
"type": "string",
"index": "not_analyzed"
},
"field4": {
"type": "string",
"index": "analyzed"
},
"field5": {
"type": "string",
"index": "analyzed"
}
}
}
}
I am seeing that dynamic mapping is disabled in settings as well as in mapping on ES endpoint but the job fails. I have a avro json mapping file and es json mapping file where avro json mapping file is the superset while es json mapping file is subset. I do not want all the fields in the superset mapping file to be reflected on ES index instead only dump fields which are in subset mapping file. Am I doing it wrong or is there any other way of doing it? Thanks.