On-going troubles with field expansion

[alastairs (Alastair Smith)]

I'm running Elasticsearch 7.4.0 and Filebeat 7.4.0. It seems that, when I query over a period longer than the current filebeat index has been open, I receive the following message:

elasticsearch field expansion matches too many fields, limit: 1024, got: 1059

I've read these three posts and have not yet managed to resolve this issue. My current approach is to reduce the number of fields we are exporting from our logs (there's >2500 currently defined in the index pattern).

This issue has broken our ability to query in Discover, and all of our Elasticsearch-backed Grafana widgets, and there seems to be very little documentation available on what to do when this issue is encountered. Should I raise the maximum number of fields queried? The default fields collection is already populated in the index template. How is this limit being reached, even, when my query targets a small number of fields specifically?

fields.RequestPath:"foo/bar" AND fields.StatusCode:[200 TO 300} AND fields.Environment:Production AND -fields.SourceContext:AzureBlobStore

[alastairs (Alastair Smith)]

This is the request Kibana sent to Elasticsearch, querying over a time period of the last 24 hours. The response indicated that "1 of 15 shards failed" and that the index for yesterday's logs returned the field expansion error.

{
  "version": true,
  "size": 500,
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "boolean"
      }
    }
  ],
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "30m",
        "time_zone": "Europe/London",
        "min_doc_count": 1
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.created",
      "format": "date_time"
    },
    {
      "field": "event.end",
      "format": "date_time"
    },
    {
      "field": "event.start",
      "format": "date_time"
    },
    {
      "field": "file.ctime",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    },
    {
      "field": "process.start",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.flow.end",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.flow.start",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.timestamp",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.tls.notafter",
      "format": "date_time"
    },
    {
      "field": "suricata.eve.tls.notbefore",
      "format": "date_time"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "fields.RequestPath:\"foo/bar\" AND fields.StatusCode:[200 TO 300} AND fields.Environment:Production AND -fields.SourceContext:AzureBlobStore",
            "analyze_wildcard": true,
            "time_zone": "Europe/London"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2019-10-31T10:57:52.246Z",
              "lte": "2019-11-01T10:57:52.246Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "highlight": {
    "pre_tags": [
      "@kibana-highlighted-field@"
    ],
    "post_tags": [
      "@/kibana-highlighted-field@"
    ],
    "fields": {
      "*": {}
    },
    "fragment_size": 2147483647
  }
}