Data stopped going to *.keyword field

[DigiAngel]

And on the heels of my last post about this, elasticsearch has again swapped where the data goes. Previously data from logstash would go to the service field. After a few days however, data starts going to the service.keyword field (by design apparently). Now, after stopping the service, upgrading the OS as shown below:

Install: linux-headers-4.4.0-47:amd64 (4.4.0-47.68, automatic), linux-image-4.4.0-47-generic:amd64 (4.4.0-47.68, automatic), linux-image-extra-4.4.0-47-generic:amd64 (4.4.0-47.68, automatic), linux-headers-4.4.0-47-generic:amd64 (4.4.0-47.68, automatic)
Upgrade: libmpx0:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libgcc-5-dev:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), linux-headers-generic:amd64 (4.4.0.45.48, 4.4.0.47.50), linux-libc-dev:amd64 (4.4.0-45.66, 4.4.0-47.68), linux-image-generic:amd64 (4.4.0.45.48, 4.4.0.47.50), cpp-5:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libitm1:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), linux-headers-generic-lts-vivid:amd64 (4.4.0.45.48, 4.4.0.47.50), libcilkrts5:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libasan2:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libquadmath0:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), gcc-5-base:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libstdc++-5-dev:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libtsan0:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libubsan0:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), g++-5:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), gcc-5:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), liblsan0:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libgomp1:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), linux-generic-lts-vivid:amd64 (4.4.0.45.48, 4.4.0.47.50), libatomic1:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), accountsservice:amd64 (0.6.40-2ubuntu11.2, 0.6.40-2ubuntu11.3), libcc1-0:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), libstdc++6:amd64 (5.4.0-6ubuntu1~16.04.2, 5.4.0-6ubuntu1~16.04.4), linux-generic:amd64 (4.4.0.45.48, 4.4.0.47.50), libaccountsservice0:amd64 (0.6.40-2ubuntu11.2, 0.6.40-2ubuntu11.3)

things went fine until at 17:00 on the dot data stopped going to the service.keyword field:

Secondly, now all the data went back to going to the service field:

And lastly, now, even though I have a template that says not to analyze strings, my service field is now being analyzed:

Can anyone shed any light on this behavior before I open a bug on this? Thank you.

[DigiAngel]

Additional info from the logs:

[2016-11-11T17:00:14,271][INFO ][o.e.c.m.MetaDataCreateIndexService] [scanner] [logstash-2016.11.12] creating index, cause [auto(bulk api)], templates [logstash_template, logstash], shards [5]/[1], mappings [_default_, syslog_dst_ip, syslog_src_ip]
[2016-11-11T17:00:16,174][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] create_mapping [connlog]
[2016-11-11T17:00:36,028][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] create_mapping [log]
[2016-11-11T17:00:36,033][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [log]
[2016-11-11T17:00:42,045][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] create_mapping [ssllog]
[2016-11-11T17:00:44,037][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [log]
[2016-11-11T17:00:44,042][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [log]
[2016-11-11T17:00:51,027][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [connlog]
[2016-11-11T17:03:29,097][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [log]
[2016-11-11T17:20:31,517][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [log]
[2016-11-11T17:21:41,541][INFO ][o.e.c.m.MetaDataMappingService] [scanner] [logstash-2016.11.12/ukw327T_Qge7L2wfZCbRrg] update_mapping [log]

right after this on first access is when I see:

[2016-11-11T17:54:39,854][DEBUG][o.e.a.s.TransportSearchAction] [scanner] [logstash-2016.11.11][0], node[-_ADS4ViShCeOSWUW-jYDw], [P], s[STARTED], a[id=r_0DOJL8SV6A2XDCe6hcvg]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[lo
gstash-2016.11.11, logstash-2016.11.12], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_alisases_to_multiple_indices=true, forbid_closed_indices=tr
ue], types=[], routing='null', preference='1478910626176', requestCache=null, scroll=null, source={

org.elasticsearch.transport.RemoteTransportException: [scanner][127.0.0.1:9300][indices:data/read/search[phase/query]] Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:335) ~[elasticsearch-5.0.0.jar:5.0.0]

[DigiAngel]

Github link: https://github.com/elastic/elasticsearch/issues/21553

[DigiAngel]

So after redoing this...yea this indeed is a case of "if only the user would get out of the way and let ES do it's thing". I was trying to fix strings being analyzed, and ES 5 has already done that. Removing the dynamic template portion and just using a template that changed a couple fields to IP addresses is what was needed.