I'm pulling some basic data into ElasticSearch via LogStash from an ER database (data on support tickets). I'm still learning about this so configuring mappings for the first time. I have fields which I'm sure should be used as "keywords" (such as "Product" which contain text data on fields we'll need to search and filter by). However the data only gets pulled in successfully if I use the "text" data type (which I don't believe is what I should be using here).
Here is the template I'm using which I use curl to upload, and also use curl to check it gets applied:
{
"template" : "logstash-asdf*",
"mappings" : {
"asdf" : {
"properties" : {
<<<other fields I'm not listing here since irrelevant>>>
"product_family":{"type":"keyword"}
}
}
}
}
In ElasticSearch I get the following error for every record which LogStash attempts to pass to ElasticSearch (I've removed some fields and data between <<< >>> in the below snippet, I've also trimmed some of the error stack of messages with ...... because I was hitting the character limit to post this):
[[logstash-asdf-2017.03.07][2]] containing [index {[logstash-asdf-2017.03.07][logs][156251819], source[{<<<removed listing of other fields>>>,"product_family":"SampleProductName","@timestamp":"2017-03-07T08:33:05.000Z","@version":"1","ticket_id":156251819,}]}]
org.elasticsearch.index.mapper.MapperParsingException: failed to parse
at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:176) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:69) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:277) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:536) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.shard.IndexShard.prepareIndexOnPrimary(IndexShard.java:513) ~[elasticsearch-5.4.0.jar:5.4.0]
..........
Caused by: java.lang.IllegalStateException: Mixing up field types: class org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType != class org.elasticsearch.index.mapper.KeywordFieldMapper$KeywordFieldType on field product_family
at org.elasticsearch.index.mapper.FieldMapper.updateFieldType(FieldMapper.java:366) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.index.mapper.FieldMapper.updateFieldType(FieldMapper.java:49) ~[elasticsearch-5.4.0.jar:5.4.0]
..........
I can't seem to find many pointers which seem relevant to my problem when I search on this error, and the "keyword" datatype seems relatively straightforward I am not sure if there are some additional properties I need to configure along with this. Any guidance appreciated, happy to share any further details to assist I've just tried to share the basics which I think are relevant here. Thanks for any help in advance.
Edit: Also here is the output when I query the templates to check the mapping is correctly applied (again I've redacted the irrelevant fields which aren't related to this issue):
# curl -XGET http://server:9200/_template/asdf*
{"asdf_template_1":{"order":0,"template":"logstash-asdf*","settings":{},"mappings":{"asdf":{"properties":{<<<removed some fields>>>,"product_family":{"type":"keyword"}}}},"aliases":{}}}
Edit2: Looking at the error messages, it looks like it is complaining about trying to apply the type as "keyword" when it already thinks it is "text" but I am not sure how/why it is getting it as text. Could this be something to do with dynamic type mapping which happens by default and do I need to do something with that? I've looked at all the templates and can't find any conflicts but I'm not familiar enough with how the mapping works to know what to check next.
