Issue with keyword aggregation following update to 6.0

[MrTraan (Nathan Grasset)]

Hello world,

Since I updated Kibana and ES to 6.0, my visualizations that relies on aggregations do not work anymore.
I see a message saying "No results displayed because all values equal 0." instead. If I select a time range that goes before the migrations they work perfectly fine.

If I run a search with the following body:

{
    "aggs" : {
        "intents" : {
            "terms" : {
              "field" : "intent.keyword",
              "size": 10
            }
        }
    }
}

It runs without any problem but it only aggregates documents that were already there before the migration.

I have already tried to refresh my index pattern but it hasn't helped.

I'm quite new to ELK and I feel a bit lost, sorry if I'm missing something obvious!

[weltenwort (Felix Stürmer)]

Hi @MrTraan,

I would need to find out more about your setup to narrow down the possible causes:

• In which visualization is the message being displayed?
• And are the new documents indexed into a new index?
• Is that index matched by the index pattern configure in Kibana?
• Does that index have the same mapping as the old ones?

[MrTraan (Nathan Grasset)]

Hi @weltenwort and thanks for your reply,

• It happens in every visualizations that rely on aggregations by keyword terms. But it works still fine when they use the string format (for example I have graphs with filters like intent:"something-*")
• The new documents are under the same index as the old one
• Yes
• Yes

One thing I am not sure about and don't know how to check, is if the new documents have their keyword field created properly.
I don't really understand how keywords work yet, and I don't know if the keyword value of a string field is stored in elasticsearch or if it is just an abstract notion.

[weltenwort (Felix Stürmer)]

If the documents are indexed into the same index, the type of the intent.keyword field is bound to be the same. Would it be possible for you to show me the mapping of that index (by running GET indexname/_mapping in the Kibana devtools with "indexname" replaced by the proper name)?

[MrTraan (Nathan Grasset)]

Sure:

{
  "xxx": {
    "mappings": {
      "logs": {
        "_all": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "index": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "intent": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "message_content": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "message_type": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "processing_time": {
            "type": "long"
          }
        }
      },
      "doc": {
        "_all": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "text"
          },
          "index": {
            "type": "text"
          },
          "intent": {
            "type": "text"
          },
          "message_content": {
            "type": "text"
          },
          "message_type": {
            "type": "text"
          },
          "processing_time": {
            "type": "long"
          }
        }
      }
    }
  }
}

[weltenwort (Felix Stürmer)]

Of what type are the new documents? It looks like the mapping for the type doc does not contain the intent.keyword subfield.

(As an aside please note that mapping types have been deprecated in 6.0.0. While they will continue to work for indices created before 6.0, but will not be supported from 7.0 onwards. Please see https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html for migration strategies.)

[MrTraan (Nathan Grasset)]

How can I check the type of the new documents?

[MrTraan (Nathan Grasset)]

Oh sorry @weltenwort I figured it out myself!
Indeed the new documents have _type:doc while the old ones have type _type:logs.
I now understand that the issue is that the mapping for type doc does not have keyword subfields.
Am I correct if I say that sending the following request:

PUT my_index
{
	"mappings": {
	  "logs": {
	    "_all": {
	      "enabled": true
	    },
	    "properties": {
	      "@timestamp": {
	        "type": "date"
	      },
	      "@version": {
	        "type": "text",
	        "fields": {
	          "keyword": {
	            "type": "keyword",
	            "ignore_above": 256
	          }
	        }
	      },
	      "index": {
	        "type": "text",
	        "fields": {
	          "keyword": {
	            "type": "keyword",
	            "ignore_above": 256
	          }
	        }
	      },
	      "intent": {
	        "type": "text",
	        "fields": {
	          "keyword": {
	            "type": "keyword",
	            "ignore_above": 256
	          }
	        }
	      },
	      "message_content": {
	        "type": "text",
	        "fields": {
	          "keyword": {
	            "type": "keyword",
	            "ignore_above": 256
	          }
	        }
	      },
	      "message_type": {
	        "type": "text",
	        "fields": {
	          "keyword": {
	            "type": "keyword",
	            "ignore_above": 256
	          }
	        }
	      },
	      "processing_time": {
	        "type": "long"
	      }
	    }
	  },
	  "doc": {
	    "_all": {
	      "enabled": true
	    },
	    "properties": {
	      "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "index": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "intent": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "message_content": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "message_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "processing_time": {
          "type": "long"
        }
	    }
	  }
	}
}

Should fix it?

And do you know why the type of my documents have changed? I didn't do it on purpose

[MrTraan (Nathan Grasset)]

Yay I figured it all out!

If anyone come across this post I had to the following request:

PUT /my_index/mappings/doc
{
  "_all": {
    "enabled": true
  },
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "@version": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "index": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "intent": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "message_content": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "message_type": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "processing_time": {
      "type": "long"
    }
  }
}

And then I did the following request to apply the mapping to my stored documents:

POST /my_index/_update_by_query?pretty&conflicts=proceed&refresh

Thanks a lot @weltenwort for your help!

[weltenwort (Felix Stürmer)]

Thanks for sharing your solution. As to why the document type has changed, it is probably a change in your ingestion pipeline (beats, logstash, ...?). The removal of mapping types I linked to might be the underlying reason.