New mapped field not getting added as searchable even after doing "refresh field list"

Elastic Stack Kibana
1

Hello,
I recently upgraded from 5.X to 6.7.2.
Most of my new logging indexes are fresh on the system, not imported.
I also imported some old indexes from old ES cluster using reindex.
So all indexes should be in native 6.7.2 format etc.

I recently added a new field to my logging setup in 6.7.2.
I needed to add "fields.type" because, with 6+ ES and filebeats 6.?+ removal of types
it was breaking logstash filter rules for matching.

Filebeats.yml now looks like this

filebeat.inputs:
 
 - type: log
   paths:
     - /var/log/nginx/access.log
   fields:
    environment: env1
   type: nginx_access

There is also a matching logstash filter entry:

if [type] == 'nginx_access' or [fields][type] == 'nginx_access' {

That is working as I see the entries show up in Kibana as
kibana_fields_type

Please note that a preexisting fields.environment is and has been working since I setup the cluster.

I am using a template that has these mapping.
I updated the template to have the new mapping.
(leaving out some details for brevity from a GET _template)

"index_patterns" : [
  "logs-*"
],

},
"mappings" : {
  "doc" : {
    "dynamic" : false,
    "properties" : {

      "fields.type" : {
        "normalizer" : "lowercase",
        "type" : "keyword"
      },
      "http_upstream_connect_time" : {
        "type" : "float"
      },
      "fields.environment" : {
        "normalizer" : "lowercase",
        "type" : "keyword"
      },

I know logstash is processing and setting the "field.type" and I can see it in Kibana.

I have attempted multiple times to use the "Refresh field list" option in the Kibana UI.

I do have multiple index patterns in Kibana to allow for segmented "views".
I did send some new data with the new fields before setting the template.
However I am getting new data and at least once new index has been created (daily index for logging) with the new fields.

Does anyone have any idea what I might need to do to get Kibana to update and use the new field.type as searchable ?

It does not show up in the list of fields in the index despite multiple field refresh attempts.

I have not seen any issues in Kibana or ES logs. I do see new .kibana indexes being created when I attempt the refresh.

The changes made were in reference to this

Is it possible that type is a reserved name that breaks when used anywhere?

Thanks very much in advance for any help or ideas!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.