How to make a field aggregatable in Kibana

Elastic Stack Kibana
1

I am running the latest versions of Kibana, Logstash and Elasticsearch. I am unsure how to make my logMessage field aggregatable. I have searched on the forums for a solid answer but can't seem to find a definitive one. Here is my mappings.

"mappings": {
		"prnformat": {
			"properties":{
				"@timestamp": {
					"type": "date",
					"format": "strict_date_optional_time||epoch_millis"
				},
				"hostName": {
					"type": "string",
					"index": "not_analyzed"
				},
				"processName": {
					"type": "string",
					"index": "not_analyzed"
				},
				"sourcefilename": {
					"type": "string",
					"index": "not_analyzed"
				},
				"processID": {
					"type": "integer"
				},
				"fileName": {
					"type": "string",
					"index": "not_analyzed"
				},
				"lineNumber": {
					"type": "integer"
				},
				"logMessage": {
					"type": "string",
					"analyzer": "simple",
				}

Can someone provide some guidance?

Thanks!

2

More than likely, the field you are trying to aggregate on is not populated with data in your index. Here is a link to another thread that asked a similar question.

3

image.png796×68 6.08 KB

I have data in that index.

4

Is there data in that field in your index?

If so, are you using a wildcard in your index pattern?

This can also occur when there is a type inconsistency for a field between two indexes in the same index pattern. You could start by creating a new index pattern in kibana that only examines a single index to see if that field shows up as aggregatable.

5

I did have a wildcard in the index pattern. But when I created an index without the wildcard there was data present and the field was still unaggregatable

6

Can you execute this in the Dev Tools console and report the results?

GET <whatever your index is>/_mapping/<whatever your type is>/field/logMessage

7

image.png1302×363 18.7 KB

8

Yeah, sorry I missed that earlier. You can't aggregate on a text field type. You need a keyword field type to aggregate.

https://www.elastic.co/guide/en/elasticsearch/reference/current/text.html#text

1 Like
9

I changed it in my json template and rebooted the system and it still comes up as a text type.

image.png593×768 13.1 KB

image.png1322×776 36.5 KB

10

You should be able to add a field to the existing mapping: documentation


PUT prn-/_mapping/prnformat
{
  "properties": {
    "logMessage": {
      "type": "text",
      "fields": {
        "raw": { 
          "type":  "keyword"
        }
      }
    }
  }
}

Then you should be able to aggregate on the raw field. However, keep in mind that this field will only be populated for new documents being indexed.

1 Like
11

I am sorry for all of these questions. I am not terribly good with Kibana and the json templates. I prefer logstash. I made my json template this:

And the new logMessage.raw never showed up when I created a new index. I put that message in the console and this is what the output came out to be:

image.png919×310 21.1 KB

12

The message returns seems to indicate that you're attempting to change the analyzer. With mappings, you can only add things. Most things can't be changed.

These are the steps that I went through to try and figure out the issue:

# Delete any existing test index
DELETE test_index

# Create a new index mapping that has a text field
PUT test_index
{
  "mappings": {
    "doc": {
      "properties": {
        "city": {
          "type": "text",
          "analyzer": "simple"
        }
      }
    }
  }
}

# Examine the mapping
GET test_index/_mapping

# Index a document that populates that field
PUT test_index/doc/1
{
  "city": "I am a text value, and will get analyzed, but cannot be aggregated"
}

# Examine the document
POST test_index/_search

# Modify the mapping to add the city.raw field
PUT test_index/_mapping/doc 
{
  "properties": {
    "city": {
      "type": "text",
      "fields": {
        "raw": { 
          "type":  "keyword"
        }
      },
      "analyzer": "simple"
    }
  }
}

# Index another document that populates the city field
PUT test_index/doc/2
{
  "city": "I am also a text value, and will get analyzed, but cannot be aggregated, but I also have an unanalyzed aggregatable .raw field that can"
}

# Notice that I can now aggregate on the city.raw field, but only the new document has values.
POST test_index/_search
{
  "size": 0,
  "aggs": {
    "my-term-aggregation": {
      "terms": {
        "field": "city.raw",
        "size": 5,
        "order": {
          "_count": "desc"
        }
      }
    }
  }
}
13

I see you using the syntax 'string' in many places. That has been deprecated and replaced with the 'text' and 'keyword' syntax. That may be another source of conflict.

1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.