We just updated our ES cluster from 2.4 to 5.2.2, and one of the changes in that is the
keyword type fields. A couple of the
string type fields we had included analyzers that were basically pulling out values from XML. This now requires the
fielddata property to be set true to use. We have an idea of a better way to go about sending these values to ES, but it's a much bigger change and a lower priority, so kinda stuck just doing it at the ES layer. Setting the
fielddata property to true is easy enough, but I was trying to get the fields to be
keywords instead. The problem here is that
keyword type fields do not allow analyzers, and the filter/tokenizers are more limited in what can be used in the
normalizer. We had been using a
pattern tokenizer in the analyzer, but that sort of thing does not appear to be supported by normalizers because of needing to operate on single characters and only emitting one token (from what I tried to understand in the docs).
The only thing that seemed like it should work was using a field on the analyzed field that was a keyword, but I don't think I'm understanding how fields work.
For example, let's say there is a
message field that is the full XML blob. With the analyzer, a field called
message.analyzed_field can be made with a type
text and that uses an analyzer to grab whatever it needed from the
message field. This is the field that has
fielddata set to true and works in visualizations. What I thought could be done was to "pass" that value from the analyzed field into a sub field of its own. ie a field
message.analyzed_field.keyword that had a type
keyword and was just the analyzed field, but as a keyword instead of text. This does not appear to be the case, it ends up being the
message field value that is used in the
message.analyzed_field.keyword. My assumption was that this has to do with how things are indexed in Lucene and the nested nature of the fields and their properties isn't what I expected.
I'm sort of at a loss for what the proper way to do this with ES might be. Any suggestions would be greatly helpful, thanks!