FIELDDATA Data is too large

Elastic Stack Elasticsearch
1

I have a problem with sorting namely, sorting work but only for price field. When I try to sort by start_date, end_date, uid, cat title get the message about exceeding the limit:

Data too large, the date for [ "name of field here"] would be larger than the limit of [19798897459 / 18.4gb]]

I do not know why this is happening code looks correct sample query for elastica looks like this:

Mapping:

"auctions": {
                "_all": { "enabled": false }, 
                "properties": {
                    "cat": { "store": true,  "type": "long" }, 
                    "curr": { "index": "not_analyzed",  "store": true,  "type": "string" }, 
                    "end_date": { "store": true,  "type": "long" }, 
                    "price": { "store": true,  "type": "long" }, 
                    "start_date": { "store": true,  "type": "long" }, 
                    "tcat": { "store": true,  "type": "long" }, 
                    "title": { "store": true,  "type": "string" }, 
                    "uid": { "store": true,  "type": "long" }
                }
            },

/_stats/fielddata?fields=*

{
    "_shards": {
        "total": 10,
        "successful": 5,
        "failed": 0
    },
    "_all": {
        "primaries": {
            "fielddata": {
                "memory_size_in_bytes": 19466671904,
                "evictions": 0,
                "fields": {
                    "_id": {
                        "memory_size_in_bytes": 0
                    },
                    "cat": {
                        "memory_size_in_bytes": 0
                    },
                    "price": {
                        "memory_size_in_bytes": 3235221240
                    },
                    "title": {
                        "memory_size_in_bytes": 16231450664
                    }
                }
            }
        },
        "total": {
            "fielddata": {
                "memory_size_in_bytes": 19466671904,
                "evictions": 0,
                "fields": {
                    "_id": {
                        "memory_size_in_bytes": 0
                    },
                    "cat": {
                        "memory_size_in_bytes": 0
                    },
                    "price": {
                        "memory_size_in_bytes": 3235221240
                    },
                    "title": {
                        "memory_size_in_bytes": 16231450664
                    }
                }
            }
        }
    },
    "indices": {
        "allek": {
            "primaries": {
                "fielddata": {
                    "memory_size_in_bytes": 19466671904,
                    "evictions": 0,
                    "fields": {
                        "_id": {
                            "memory_size_in_bytes": 0
                        },
                        "cat": {
                            "memory_size_in_bytes": 0
                        },
                        "price": {
                            "memory_size_in_bytes": 3235221240
                        },
                        "title": {
                            "memory_size_in_bytes": 16231450664
                        }
                    }
                }
            },
            "total": {
                "fielddata": {
                    "memory_size_in_bytes": 19466671904,
                    "evictions": 0,
                    "fields": {
                        "_id": {
                            "memory_size_in_bytes": 0
                        },
                        "cat": {
                            "memory_size_in_bytes": 0
                        },
                        "price": {
                            "memory_size_in_bytes": 3235221240
                        },
                        "title": {
                            "memory_size_in_bytes": 16231450664
                        }
                    }
                }
            }
        }
    }
}

Any ideas what might cause this exception?

I should add that I've tried these solutions:

FIELDDATA Data is too large

But the effect was even worse, then no query did not work as quickly.

For any help I will be extremely grateful!

2

Hi @Radoslaw_Wojtkiewicz,

which Elasticsearch version do you use?

Daniel

3 
{
  "status" : 200,
  "name" : "Quicksand",
  "cluster_name" : "allek",
  "version" : {
    "number" : "1.7.5",
    "build_hash" : "00f95f4ffca6de89d68b7ccaf80d148f1f70e4d4",
    "build_timestamp" : "2016-02-02T09:55:30Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}
4

Hi @Radoslaw_Wojtkiewicz,

you could enable doc_values to avoid using field data for sorting. But be aware of the restrictions stated in the docs:

Lowers memory usage but only works on non-analyzed strings (index: no or not_analyzed).

So you could use a second field (e.g. title_sort) which is not_analyzed and sort on that instead of the actual title field.

Daniel

5

Thanks for your answer !

So my map should be look like this now?

auctions:

{
  "_all": {
    "enabled": false
  },
  "properties": {
    "cat": {
      "store": true,
      "type": "long",
      "doc_values": true
    },
    "curr": {
      "index": "not_analyzed",
      "store": true,
      "type": "string",
      "doc_values": true
    },
    "end_date": {
      "store": true,
      "type": "long",
      "doc_values": true
    },
    "price": {
      "store": true,
      "type": "long",
      "doc_values": true
    },
    "start_date": {
      "store": true,
      "type": "long",
      "doc_values": true
    },
    "tcat": {
      "store": true,
      "type": "long",
      "doc_values": true
    },
    "title": {
      "store": true,
      "type": "string",
      "fields": {
        "raw": {
          "type": "string",
          "index": "not_analyzed",
          "ignore_above": 256,
          "doc_values": true
        }
      }
    },
    "uid": {
      "store": true,
      "type": "long",
      "doc_values": true
    }
  }
}
6

Yes, that looks about right. Don't forget that you need to reindex your data after you've changed the mapping.

Daniel

7

Okey thanks !