How to make a field aggregatable in Kibana

[CDR (Colton)]

I am running the latest versions of Kibana, Logstash and Elasticsearch. I am unsure how to make my logMessage field aggregatable. I have searched on the forums for a solid answer but can't seem to find a definitive one. Here is my mappings.

"mappings": {
		"prnformat": {
			"properties":{
				"@timestamp": {
					"type": "date",
					"format": "strict_date_optional_time||epoch_millis"
				},
				"hostName": {
					"type": "string",
					"index": "not_analyzed"
				},
				"processName": {
					"type": "string",
					"index": "not_analyzed"
				},
				"sourcefilename": {
					"type": "string",
					"index": "not_analyzed"
				},
				"processID": {
					"type": "integer"
				},
				"fileName": {
					"type": "string",
					"index": "not_analyzed"
				},
				"lineNumber": {
					"type": "integer"
				},
				"logMessage": {
					"type": "string",
					"analyzer": "simple",
				}

Can someone provide some guidance?

Thanks!

[BigFunger (Jim Unger)]

More than likely, the field you are trying to aggregate on is not populated with data in your index. Here is a link to another thread that asked a similar question.

[CDR (Colton)]

I have data in that index.

[BigFunger (Jim Unger)]

Is there data in that field in your index?

If so, are you using a wildcard in your index pattern?

This can also occur when there is a type inconsistency for a field between two indexes in the same index pattern. You could start by creating a new index pattern in kibana that only examines a single index to see if that field shows up as aggregatable.

[CDR (Colton)]

I did have a wildcard in the index pattern. But when I created an index without the wildcard there was data present and the field was still unaggregatable

[BigFunger (Jim Unger)]

Can you execute this in the Dev Tools console and report the results?

GET <whatever your index is>/_mapping/<whatever your type is>/field/logMessage

[CDR (Colton)]

[BigFunger (Jim Unger)]

Yeah, sorry I missed that earlier. You can't aggregate on a text field type. You need a keyword field type to aggregate.

https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#text

[CDR (Colton)]

I changed it in my json template and rebooted the system and it still comes up as a text type.

[BigFunger (Jim Unger)]

You should be able to add a field to the existing mapping: documentation

PUT prn-/_mapping/prnformat
{
  "properties": {
    "logMessage": {
      "type": "text",
      "fields": {
        "raw": { 
          "type":  "keyword"
        }
      }
    }
  }
}

Then you should be able to aggregate on the raw field. However, keep in mind that this field will only be populated for new documents being indexed.

[CDR (Colton)]

I am sorry for all of these questions. I am not terribly good with Kibana and the json templates. I prefer logstash. I made my json template this:

And the new logMessage.raw never showed up when I created a new index. I put that message in the console and this is what the output came out to be:

[BigFunger (Jim Unger)]

The message returns seems to indicate that you're attempting to change the analyzer. With mappings, you can only add things. Most things can't be changed.

These are the steps that I went through to try and figure out the issue:

# Delete any existing test index
DELETE test_index

# Create a new index mapping that has a text field
PUT test_index
{
  "mappings": {
    "doc": {
      "properties": {
        "city": {
          "type": "text",
          "analyzer": "simple"
        }
      }
    }
  }
}

# Examine the mapping
GET test_index/_mapping

# Index a document that populates that field
PUT test_index/doc/1
{
  "city": "I am a text value, and will get analyzed, but cannot be aggregated"
}

# Examine the document
POST test_index/_search

# Modify the mapping to add the city.raw field
PUT test_index/_mapping/doc 
{
  "properties": {
    "city": {
      "type": "text",
      "fields": {
        "raw": { 
          "type":  "keyword"
        }
      },
      "analyzer": "simple"
    }
  }
}

# Index another document that populates the city field
PUT test_index/doc/2
{
  "city": "I am also a text value, and will get analyzed, but cannot be aggregated, but I also have an unanalyzed aggregatable .raw field that can"
}

# Notice that I can now aggregate on the city.raw field, but only the new document has values.
POST test_index/_search
{
  "size": 0,
  "aggs": {
    "my-term-aggregation": {
      "terms": {
        "field": "city.raw",
        "size": 5,
        "order": {
          "_count": "desc"
        }
      }
    }
  }
}

[BigFunger (Jim Unger)]

I see you using the syntax 'string' in many places. That has been deprecated and replaced with the 'text' and 'keyword' syntax. That may be another source of conflict.