Illegal_argument_exception Text Fields are not Optmised

Elastic Stack Elasticsearch
1

I have started getting this error when trying to run an average and median on a what certainly used to be mapped as a number.

Type: illegal_argument_exception
Reason: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [duration] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

Under Stack Management -> Index Patterns it shows as a number

Screenshot 2025-03-04 at 16.57.42
Screenshot 2025-03-04 at 16.57.422374×400 23 KB

This is what I have in the Visualize settings

Screenshot 2025-03-04 at 16.59.17
Screenshot 2025-03-04 at 16.59.17960×1140 36 KB

To me this doesn't make sense as it should be mapped to a number?

Edited to add:

Looking at GET /index_name/_mappings

        "duration" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },

I'm confused as to why it's mapped to a text, but showing as number in the index management?

2

Guessing - As I've seen this, and in our case it was just "luck" that sometimes the first doc for some indices that got indexed didn't have a number for the specific field. Our field was httpCode, and most (vast majority) of the docs ingested had 200, 301, 404, etc. But some had "404 Not Found" or similar. If they happen to be the first, then dynamic field mapping means you get effectively what you saw. Even if the field is "", i.e. no value, it probably defaults to text field.

If my hunch is correct, you cant really fix it without re-indexing. But you can set the mapping in advance of index creation, lots of ways to do this.

You can also check that field mapping for all the indices where it might apply, see how many have the right/wrong mapping.

Oh, and the index-patterns mappings can mislead here as those are not always auto-refreshed.

1 Like
3

Thank you for the help - that looks like it's exactly what happened.

How safe is re-indexing? Each index is approximately 30GB - could this cause performance issues?

I ran the re-index with two much smaller indexes and noticed there is very slight difference in sizes

health 	status 	index                          uuid                   	pri 	rep 	docs.count docs.deleted store.size 	pri.store.size
green 	open 	logs-small-index-old           kQYj-tUWRtyGGsOnL-SKZg 	2 		1  		112467      0   		95.2mb  	48.2mb
green 	open 	logs-small-index-new           5OqsiNwbRu2Mj2yIM3MKhA 	2 		1   	97000       0   		96.1mb  	47.4mb

Is that potentially showing data loss?